Getting Timeout Error after following Kubernetes example

[beldur]

Hi,

I created a GKE Cluster and followed the Kubernetes instructions:

Now I'm getting the following error (XXX:XXX is my Project and Instance Name):

Open socket for "XXX:XXX" at "/cloudsql/XXX:XXX"
Socket prefix: /cloudsql
Got a connection for "XXX:XXX"
couldn't connect to "XXX:XXX": Post https://www.googleapis.com/sql/v1beta4/projects/XXX/instances/XXX/createEphemeral?alt=json: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp: i/o timeout

Any idea where the timeout comes from?

[Carrotman42]

Interesting. I'm thinking that error happens when the proxy was attempting to receive an oauth token for your service account, although that's a strange failure mode. Is it possible that you are preventing outgoing HTTP requests in your Kubernetes configuration?

@dlorenc : have you ever seen this?

[dlorenc]

I haven't seen this, but did notice another error that I didn't see before. I just sent a PR to add that.

@beldur, can you try again with the ssl-certs volume like I added in #10?

Other than that, I'd try to make sure you have outbound internet access and your DNS configuration is correct. If you SSH to a node, can you ping accounts.google.com?

[beldur]

From a Cluster Node:

user@gke-main-xfx19x6x-node-3fmc:~$ ping accounts.google.com
PING accounts.google.com (74.125.206.84) 56(84) bytes of data.
64 bytes from wk-in-f84.1e100.net (74.125.206.84): icmp_req=1 ttl=55 time=0.609 ms

From a Container (nginx vanilla container) running on a node

/ # ping accounts.google.com
PING accounts.google.com (74.125.206.84): 56 data bytes
64 bytes from 74.125.206.84: seq=0 ttl=54 time=0.664 ms

[beldur]

I added the ssl certs volume and replaced my RC with --force=true --cascade. My RC looks something like this:

...
    containers:
      - name: sql-proxy
        image: b.gcr.io/cloudsql-docker/gce-proxy
        command:
        - /cloud_sql_proxy
        - -dir=/cloudsql
        - -instances=xxx:xxx
        - -credential_file=/sql-creds/creds.json
        volumeMounts:
        - mountPath: /cloudsql
          name: sql-proxy
        - mountPath: /sql-creds/
          name: sql-creds
        - mountPath: /etc/ssl/certs
          name: ssl-certs
...
      volumes:
      - name: sql-proxy
        emptyDir: {}
      - name: sql-creds
        secret:
          secretName: xxx
      - name: ssl-certs
        hostPath:
          path: /etc/ssl/certs

I still get the same timeout error.

[Carrotman42]

My guess is similar to @dlorenc, in that it seems that something is blocking outgoing connections to that address from within the container. Do you happen to set any sort of network flags on that container?

Can you SSH and try to do a wget https://accounts.google.com (from within the node and from a container that has similar settings as the proxy)? Maybe it's just blocking port 443.

[beldur]

I completely recreated my cluster with the new GKE Version 1.2

Now I don't get the timeout error anymore.

[tmaiaroto]

Was there any more info about this? I've seen it again lately, a few times. Under GKE Version 1.14.9-gke.23. Support hasn't been able to see anything conclusive and suggested creating a new cluster.