400 bad request while using Kubernetes Client

[abhinavsingh1196]

Hi, we recently upgraded our fabric8 version from 5.12.1 -> 6.5.1. Since we were earlier using DefaultKubernetesClient, we changed the usage to use KubernetesClientBuilder.

Before:
DefaultKubernetesClient client = new DefaultKubernetesClient(simpleClientContext).inNamespace(namespace)

After:
KubernetesClient client = new KubernetesClientBuilder() .withConfig(config) .withHttpClientFactory(fabric8HttpClientFactory) .build() .adapt(NamespacedKubernetesClient.class) .inNamespace(namespace);

The okhttpclient which we pass to this kubernetes client is still constructed using the exact same logic.
But after making this change, ONLY a small portion of our customers have reported an issue where we're seeing 400 bad request when trying to execute this DSL
client.extensions().deployments().inNamespace(namespace).withLabels(labels).list()

The error message we get is:
KubernetesClientException: Failure executing: GET at: https://<K8s_CLUSTER_URL>/apis/extensions/v1beta1/namespaces/<NAMESPACE>/deployments?labelSelector=labelkey%3Dlabelvalue. Message: Bad Request.

Do you know what could be the issue here? Do you see something wrong in the way we are now creating these kubernetes clients?
How could we debug this further?

[shawkins]

Do you know what could be the issue here?

Do you mean to use the old v1beta1 deployments endpoint:

apis/extensions/v1beta1/namespaces

client.extensions().deployments() is deprecated. Based upon your error message, I'm not sure aware of what older clients may have been doing in this circumstance. The BackwardsCompatibilityInterceptor is no longer enabled by default, but I don't see that it would handle this case.

[abhinavsingh1196]

Just to provide you more info, the function which fails, makes these calls together:

client.extensions().deployments().inNamespace(namespace).withLabels(labels).list()
client.extensions().replicaSets().inNamespace(namespace).withLabels(labels).list()
client.apps().statefulSets().inNamespace(namespace).withLabels(labels).list()
client.extensions().daemonSets().inNamespace(namespace).withLabels(labels).list()

There is also one version call which we make in case all of these fail:
client.getKubernetesVersion()

All of these calls fail with 400 bad request. This is not consistently failing for everyone, since only a small portion of customers have reported this error. We have not been able to reproduce this issue however.
Interestingly, all the customers who reported this are using Rancher to manage their clusters.

[shawkins]

Interestingly, all the customers who reported this are using Rancher to manage their clusters.

It should just be an issue of what kubernetes version they are on.

You may try re-enabling the BackwardsCompatibilityInterceptor, but I don't quite see it handling this case. It would be best to move off of using deprecated endpoints. For example apis/extensions/v1beta1 has not been in use for nearly 4 years - https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/ - "Deployment in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served"

[shawkins]

At least for my local minikube I see a 404, not a 400 when trying to access v1beta1 deployments. The only other thing to check would be if there is anything unusual about the labelKey or value that does not appear to be encoded correctly - that could result in a 400 error as well.

[abhinavsingh1196]

We have observed that a simple version API call is also resulting in 400 bad request.
client.getKubernetesVersion()

Error: GET at: https://<cluster_host>/version. Message: Bad Request.

[shawkins]

If this behavior is repeatable, we'll need to see a trace logging of the request being sent - especially for a simple version request there would have to be something like a header making it invalid as the url is very straight-forward. Also be sure to check the api server logs for additional clues and ensure that there's no proxy that might be causing problems.