SocketTimeoutException when connecting

[ChrisMullard]

Hi,
I have three Kubernetes clusters in AWS - these all constructed from a script so should be effectively identical. The generated K8 configs allow Kubernetes Dashboard connections with no issues (and kubectl commands work fine to all 3 too)

However making an API connection (using the same configs) to the cluster doesn't work for one of them and I've no why. The other two work fine.

Java code:

            ApiClient client = Config.fromConfig(configFile);
            client.getHttpClient().setReadTimeout(60, TimeUnit.SECONDS);
            Configuration.setDefaultApiClient(client);              

            ExtensionsV1beta1Api extApi = new ExtensionsV1beta1Api();
            ExtensionsV1beta1DeploymentList list = extApi.listNamespacedDeployment("monitoring", false, "false", null, null, null, null, null, 10000,
                    null);
            for (ExtensionsV1beta1Deployment item : list.getItems()) {
                logger.info(region + " " + item.getMetadata().getName() + " " + item.getStatus().getAvailableReplicas());
            }

For the cluster it doesn't work for I get a timeout:

io.kubernetes.client.ApiException: java.net.SocketTimeoutException: connect timed out
at io.kubernetes.client.ApiClient.execute(ApiClient.java:801)
at io.kubernetes.client.apis.ExtensionsV1beta1Api.listNamespacedDeploymentWithHttpInfo(ExtensionsV1beta1Api.java:4219)
at io.kubernetes.client.apis.ExtensionsV1beta1Api.listNamespacedDeployment(ExtensionsV1beta1Api.java:4196)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)

Is there any way to get more information about why the API cannot connect?
Thanks

Versions of client and cluster:
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"clean", BuildDate:"2018-09-27T17:05:32Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"windows/amd64"}

Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

The server versions are the same for all three clusters

[yue9944882]

doesn't sounds like an issue from the client.. are you accessing the clusters behind a l4 or l7 load-balancer?

[brendandburns]

Does kubectl get deployments --namespace monitoring work to the cluster that is non-responsive from this library?

I agree with @yue9944882 that this seems to be a problem in your cluster, not this library, but if kubectl works then this may deserve further investigation.

[ChrisMullard]

Yes that works fine in all 3 clusters.
H:>kubectl get deployments --namespace monitoring
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
grafana 1 1 1 1 29d
kube-state-metrics 1 1 1 1 29d
prometheus-adapter 1 1 1 1 29d
prometheus-operator 1 1 1 1 29d

The clusters are running under EKS (AWS)

Is there anyway to get more information from the connection attempt as to why it cannot connect? (like debug info etc)?

[yue9944882]

ApiClient#setDebugging will probably help, but error is suggesting something's wrong/unstable w/ the underlying l4 network.

[brendandburns]

Worst case scenario using wireshark to trace the network should be useful...

[ChrisMullard]

Turns out the issue was with the company firewall and not Kubernetes at all.
Thanks for all assistance.

[venkatesh588]

Hi ChrisMullard

I am also facing same issue, can you please tell me how you resolved the issue

Please find error log file.
error.txt

[ast-sf-tok]

I am also getting the same issue. Can someone help me if this is occurring with k8s cluster then what can be the fix ?