Can't connect to Google Cloud Sql #6593

xolott · 2018-06-25T14:46:28Z

Describe the bug
Can't connect to Cloud SQL (postgres 9.6) using istio in Google Container Engine (Kubernetes). The service uses the cloudsql proxy sidecar along with istio in the same pod. Tested with a docker container with psql installed to test the connection.

In vanilla kubernetes I can connect to the database:

root@auth-66c4bf88df-bs7q2:/app# psql -h 127.0.0.1 -U auth -d passport
Password for user auth: 
psql (9.6.7, server 9.6.6)
Type "help" for help.

passport=>

But, with istio I got:

root@auth-7f769bc877-fbzp4:/app# psql -h 127.0.0.1 -U auth -d passport
psql: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

In the early tries I got blocked by the container (Cloud Proxy), and trying to connect to googleapis.com and accounts.google.com

$ cat <<EOF | istioctl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: account-google-serviceentry-rule
spec:
  hosts:
  - accounts.google.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
EOF

$ cat <<EOF | istioctl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: googleapis-serviceentry-rule
spec:
  hosts:
  - www.googleapis.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
EOF

Expected behavior
Successful connection to the database

Steps to reproduce the bug

Create a Cloud SQL database instance (postgres 9.6)
Create a service account with Cloud Client role enabled and Cloud Sql account following the GCloud tutorial
Install istio following the Quick Start page without auth enabled
Deploy a service (based on debian stretch with psql installed)
Try to connect to the Database using the credential provided in the previous steps. For example:

$ psql -h 127.0.0.1 -U auth -d passport

Version

$ istioctl version
Version: 0.8.0
GitRevision: 6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84
User: root@48d5ddfd72da
Hub: docker.io/istio
GolangVersion: go1.10.1
BuildStatus: Clean

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.4-gke.2", GitCommit:"eb2e43842aaa21d6f0bb65d6adf5a84bbdc62eaf", GitTreeState:"clean", BuildDate:"2018-06-15T21:48:39Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not?
Installing following the Quick Start page without auth enabled

$ kubectl apply -f install/kubernetes/istio-demo.yaml

Environment
Kubernetes Engine in Google Cloud
Tested with version 1.9.7 and 1.10.4-gke.2

The text was updated successfully, but these errors were encountered:

ahmmadkour · 2018-07-16T13:51:14Z

I got a similar issue where I wasn't able to connect from another container to the database using the cloudsql proxy (I was able to connect from inside cloudsql proxy using localhost, though) it is solved after I found the solution in https://groups.google.com/forum/#!topic/istio-users/-cNoLrE5904

The cloudsql proxy service port name should not be http, the problem solved after I changed it from http to db

veryhumble · 2018-08-21T09:21:11Z

Any update on this? I am experiencing the same problem using the cloudsql-proxy sidecar.

Edit:
I got the connection somehow working. I added a sidecar with the postgres client and tried connecting from there manually. In the logs of the cloudsql-proxy i saw that it's trying to connect to the postgresql db using port 3307. So I added the following ServiceEntry:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloudsql-external
spec:
  hosts:
  - IP_ADDRESS
  addresses:
  - IP_ADDRESS/32
  ports:
  - name: tcp
    number: 3307
    protocol: tcp
  location: MESH_EXTERNAL

It now prompts correctly the password after psql -h 127.0.0.0 -U proxyuser -d test . But its stuck on that password prompt. So not quite resolved yet.

yskopets · 2018-09-12T10:10:58Z

I confirm that solution suggested by @veryhumble works.

It's actually documented at https://istio.io/blog/2018/egress-tcp/

Here is the complete configuration that works for me:

# Cloud SQL Proxy makes requests to `www.googleapis.com`
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: www.googleapis.com
  namespace: istio-example-cloud-sql
spec:
  hosts:
  - www.googleapis.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
# see https://istio.io/blog/2018/egress-tcp/
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloud-sql-instance
  namespace: istio-example-cloud-sql
spec:
  hosts:
  # although blog article says that `hosts` field is ignored for TCP service entries,
  # clients fail to establish connection when `hosts` is omitted, 
  # namely "curl: (56) Recv failure: Connection reset by peer"
  #
  # use `gcloud sql instances list` to find out the IP address of your Google Cloud Instance
  - X.X.X.X
  addresses:
  # use `gcloud sql instances list` to find out the IP address of your Google Cloud Instance
  - X.X.X.X/32 # a block of IPs in CIDR notation
  ports:
  - name: tcp
    number: 3307 # at the moment, Google Cloud SQL always available on port 3307
    protocol: tcp # enable TCP traffic
  location: MESH_EXTERNAL

danielfoehrKn · 2018-10-10T13:42:36Z

Hi,
I am also currently having an issue with the connection to CloudSQL from my istio cluster.

I am using https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory (NOT the CloudSQL Proxy).

The way it worked for me was to configure the sidecas to only intercept mesh-internal traffic as described here: https://istio.io/docs/tasks/traffic-management/egress/#calling-external-services-directly

However I was not able to make a successful connection to the MSQL Instance using ServiceEntries.
I could see that the authentication to the google servers using the ServiceAccount worked.
But when the java application tried to connect to the mysql, the SSL handshake always failed with

"SSL peer shut down incorrectly"

The same behaviour could be observed using a mysql client deployed in another container.

I think that my ServiceEntry configuration is still wrong, but I could not figure out why.
I am not sure about the CIDR Block tbh.
Definitely the sidecar is interfering in the wrong way in my case, because when I disable it, the connection works.

Maybe also worth mentioning: I am using the option "useSSL=false" in the jdbc connection string in the cloud-sql-jdbc library.

Does anybody have an idea where the problem might be? thanks!

This is the ServiceEntry configuration:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: www.googleapis.com
#  namespace: istio-example-cloud-sql
spec:
  hosts:
  - "*.googleapis.com"
  - 169.254.169.254 # metadata.google.internal   -> IP was being logged in istio-proxy with 404
  ports:
  - number: 443
    name: https
    protocol: HTTPS
#  - number: 80
#    name: http
#    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloud-sql-instance
#  namespace: istio-example-cloud-sql
spec:
  hosts:
  # although blog article says that `hosts` field is ignored for TCP service entries,
  # clients fail to establish connection when `hosts` is omitted,
  # namely "curl: (56) Recv failure: Connection reset by peer"
  #
  # use `gcloud sql instances list` to find out the IP address of your Google Cloud Instance
  - XXX.XXX.22.151
  addresses:
  # use `gcloud sql instances list` to find out the IP address of your Google Cloud Instance
  - XXX.XXX.22.0/32 # a block of IPs in CIDR notation
  ports:
  - name: tcp
    number: 3307 # at the moment, Google Cloud SQL always available on port 3307
    protocol: tcp # enable TCP traffic
  location: MESH_EXTERNAL

I also tried unsuccessfully to add a VirtualService (because of the SSL /TLS handshake error) as described here https://istio.io/docs/tasks/traffic-management/egress/#configuring-istio-external-services and here #5288

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: cloudsql-virtualservice-2
spec:
  hosts:
  - XXX.XXX.22.115
  tls:
  - match:
    - port: 443
      sni_hosts:
      - XXX.XXX.22.115
    route:
    - destination:
        host: XXX.XXX.22.115
        port:
          number: 443
      weight: 100

danielfoehrKn · 2018-10-11T17:15:05Z

UPDATE: I got it working with a slightly different configuration using jdbc Socket Factory connecting to CloudSQL Mysql 2.Gen.

Difference to @yskopets @veryhumble configuration:

No "Adresses field" /CIDR block in cloud-sql-instance service entry
Resolution: DNS for the service entry that allows the connection to the CloudSQL Instance

# Pod makes requests to `www.googleapis.com` for Authentication
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: www.googleapis.com
spec:
  hosts:
  - "*.googleapis.com"
  - 169.254.169.254 # metadata.google.internal   -> IP was being logged in istio-proxy with 404
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL

---
# jdbc connection to CloudSQL instance
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloud-sql-instance
#  namespace: istio-example-cloud-sql
spec:
  hosts:
  # use `gcloud sql instances list` to find out the IP address of your Google Cloud Instance
  - 104.155.22.115
  ports:
  - name: tcp
    number: 3307 # at the moment, Google Cloud SQL always available on port 3307
    protocol: tcp # enable TCP traffic
  location: MESH_EXTERNAL
  resolution: DNS

ThusharaAma · 2018-11-01T09:20:58Z

I'm also trying to connecting from GKE to the Cloud SQL using Cloud SQL Proxy as described here.
https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine

I have disabled istio mtls and configured Service Entries But I get the following message

couldn't connect to ":": Post https://www.googleapis.com/sql/v1beta4/projects/my-project/instances/my-db/createEphemeral?alt=json: oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: http: server gave HTTP response to HTTPS client

But the same setup works without enabling istio.

My istio configuration

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: account-google-serviceentry-rule
spec:
  hosts:
  - accounts.google.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS

---
    
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: account-google-serviceentry-rule
spec:
  hosts:
  - accounts.google.com
  tls:
  - match:
    - port: 443
      sni_hosts:
      - accounts.google.com
    route:
    - destination:
        host: accounts.google.com
        port:
          number: 443
      weight: 100

---
    
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: googleapis-serviceentry-rule
spec:
  hosts:
  - www.googleapis.com
  - 169.254.169.254
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS 
  
---
      
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: googleapis-serviceentry-rule
spec:
  hosts:
  - www.googleapis.com
  tls:
  - match:
    - port: 443
      sniHosts:
      - www.googleapis.com
    route:
    - destination:
        host: www.googleapis.com 
  - match:
    - port: 80
      sniHosts:
      - www.googleapis.com
    route:
    - destination:
        host: www.googleapis.com 
        
---

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mysql-serviceentry-rule
spec:
  hosts:
  - X.X.X.X
  ports:
  - number: 3306
    name: tcp
    protocol: TCP

vadimeisenbergibm · 2018-11-12T17:23:57Z

@ThusharaAma your configuration for www.googleapis.com and accounts.google.com looks correct, except for the IP 169.254.169.254 - can you check it works when you remove it? Just use the host www.googleapis.com.

The TCP entry for mysql is incorrect. You should either specify a CIDR block for TCP, see https://preliminary.istio.io/blog/2018/egress-tcp/#mesh-external-service-entry-for-an-external-mysql-instance or if your mysql supports TLS, configure egress traffic for it in the same way as for www.googleapis.com.

You can check if your mysql server supports TLS by running openssl s_client -connect X.X.X.X:3306 -servername X.X.X.X. If a certificate of the server is returned, it supports TLS.

mr-smithers-excellent · 2018-11-17T08:01:53Z

@danielfoehrKn I’m curious if you’re using the JDBC Socket Factory natively or as a part of the Spring Boot GCP project, which uses it under the hood. I’m having similar issues connecting a Java app to Cloud SQL in a GKE cluster with Istio 1.0.3.

ermik · 2019-01-29T01:29:04Z

I'm working with this issue now, and it's still unclear what exactly needs to be in place for egress to work on v1alpha3 — so far I have a Gateway with a wildcard tls server *.googleapis.com; then I use that gateway in my VirtualService that handles *.googleapis.com host entry sending it to a ServiceEntry via tls route (sniHosts: "*.googleapis.com") and that ServiceEntry has "*.googleapis.com" in hosts and is MESH_EXTERNAL. Doesn't work.

vadimeisenbergibm · 2019-01-29T07:13:26Z

@ermik The first question is if you need to direct the egress traffic thru a gateway, or you can just direct the traffic directly from the sidecar.

If you need the gateway, and you want to use wildcard hosts like *.googleapis.com, see this example how to do it:
https://preliminary.istio.io/docs/examples/advanced-gateways/wildcard-egress-hosts/

Kampe · 2019-04-25T05:22:56Z

I'm having this same issue, can confirm am able to hit www.googleapis.com from the running container, but while on the sidecar, I'm unable to get a response and get this error:
/ $ wget accounts.google.com Connecting to accounts.google.com (74.125.142.84:80) Connecting to accounts.google.com (74.125.142.84:443) wget: can't execute 'ssl_helper': No such file or directory wget: error getting response: Connection reset by peer

vadimeisenbergibm · 2019-04-25T14:38:44Z

@Kampe Did you define the ServiceEntry for www.googleapis.com or for accounts.google.com? Can you share your ServiceEntry?

Kampe · 2019-04-30T23:26:18Z

Yup here that is

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-svc-https
spec:
  hosts:
  - www.googleapis.com
  location: MESH_EXTERNAL
  ports:
  - number: 443
    name: https
    protocol: TLS
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS

vadimeisenbergibm · 2019-05-01T06:17:35Z

@Kampe But in #6593 (comment) you show output of connecting to accounts.google.com, not to www.googleapis.com. If you want to connect to accounts.google.com, you have to use this host in the ServiceEntry.

Kampe · 2019-05-02T16:38:04Z

I have one for both at the moment, accounts.google.com & www.googleapis.com as these were the logs of what I was seeing fail via the cloudsql proxy sidecar. I've since fallen back and am using private IP to connect.

ermik · 2019-05-02T20:10:01Z

@Kampe Private IP may be the issue. I think there is a separate set of criteria as to how Google CloudSQL expects requests to be made to a Private IP you provision.

Kampe · 2019-05-02T22:07:04Z

I was using the Cloud SQL proxy, moving to private IP was just my workaround since it was an issue getting any egress comms out of the cluster via that sidecar in particular. I attached and tried a test of my own domain (which I have ServiceEntries for) and was unsuccessful in getting any traffic to egress out. It was almost as like the sidecar was not abiding by any of my policies at all.

ermik · 2019-05-03T12:30:44Z

That is certainly odd, but I think as far as istio is concerned, Private IP for CloudSQL is still considered an Egress. Hopefully team will clarify.

davinkevin · 2019-06-06T15:10:15Z

Any news from some of you with Istio in version 1.1 (I'm on Istio on GKE 1.1.3) and I'm trying to do the same 😅.

Some solutions proposed here doesn't seem to be compatible with ServiceEntry definition in 1.1+

Kampe · 2019-06-06T17:49:29Z

Yes I do have some news that may or may not help you resolve your issue as it's a bit different approach as I'm no longer utilizing the cloud sql proxy.

I've gone ahead and utilized VPC-native (Alias IP) functionality on my GKE clusters to allow connections to Cloud SQL from GKE over private IP. Currently this is not enabled by default for GKE cluster creation, but I believe will become the default end of July 2019.

My service entry from above has been changed to reflect the private IP and port of my Cloud SQL instance and I'm able to connect without issue, and the other virtual services and service entries defined above by others for http comms have been removed as we no longer needed them for authentication since we're not utilizing the cloud sql proxy sidecar.

This is now all I need to facilitate traffic to my Cloud SQL instance

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: postgres-serviceentry-rule
spec:
  hosts:
  - 10.xx.xx.xx
  addresses:
  - 10.xx.xx.xx/32
  ports:
  - name: tcp 
    number: 5432
    protocol: TCP
  location: MESH_EXTERNAL

mr-smithers-excellent · 2019-06-06T19:06:59Z

Just to add on to @Kampe's strategy - I was able to achieve the same connectivity using private IP via a private Cloud DNS entry, which offers some nice abstraction. My config looks like this for a MySQL instance:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: staging-vpc-egress
spec:
  hosts:
  - "*.staging-vpc.example.com"
  ports:
  - number: 3306
    name: tcp
    protocol: TCP
  location: MESH_EXTERNAL

ahmed1smael · 2019-06-27T10:36:58Z

@Kampe I don't think listing IP address in hosts field is possible (istio version 1.1.8), how did you get around that. also do you have the case where you enable SSL on cloudSQL side? did you manage to get that done somehow?

bussyjd · 2019-07-31T20:22:30Z

IP addresses are indeed not possible in newer versions
#13009

Kampe · 2019-08-07T04:32:25Z

The hosts can be whatever you like, think of it like giving the IP a DNS record.

gustavovalverde · 2019-08-26T23:03:44Z

If it helps the ones that are stuck; using TCP connection with cloud_sql_proxy worked for me:

        - name: cloudsql-proxy
          command: [
              "-instances=<instance-name>=tcp:5432",
              "-ip_address_types=PRIVATE",
              "-credential_file=/secrets/cloudsql/credentials.json",
            ]

But removing the TCP option and trying to use Unix Socket, always failed:

        - name: cloudsql-proxy
          command: [
              "/cloud_sql_proxy",
              "-dir=/cloudsql",
              "-instances=<instance-name>",
              "-ip_address_types=PRIVATE",
              "-credential_file=/secrets/cloudsql/credentials.json",
            ]

With this error on the Cloud SQL logs:

Get https://www.googleapis.com/sql/v1beta4/projects/<instance-name>?alt=json&prettyPrint=false: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: dial tcp 74.125.141.95:443: connect: connection refused

Using both; private or public IP and with any ServiceEntry combination given on this post so far didn't work while using Unix Socket

gustavovalverde · 2019-09-23T06:36:25Z

@vadimeisenbergibm I can confirm this works without the istio-proxy as a sidecar. All the proposed workarounds are using ServiceEntry to allow this URLs, but if the outboundTrafficPolicy.mode is set to ALLOW_ANY, why are all this steps needed?

Is there another way to tell the sidecar to allow all egress traffic?

gustavovalverde · 2019-11-03T19:25:07Z

@rshriram this one should be kept open. There's no defined solution for it.

gustavovalverde · 2019-11-06T12:37:29Z

This should be kept open; there's no solution to it.

Jumwah · 2019-11-07T23:45:42Z

This issue shouldn't be closed as there is no solution yet. Can someone (@vadimeisenbergibm ?) reopen?

I'm currently stuck with the https://www.googleapis.com/sql/v1beta4/projects/pc-internal-dev/instances/pc-internal-dev-1?alt=json&prettyPrint=false: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: dial tcp 172.217.212.95:443: connect: connection refused issue.

I'm using Istio 1.2-release on 1.13.11-gke.9

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: www.googleapis.com
spec:
  hosts:
  - www.googleapis.com
  - oauth2.googleapis.com
  - accounts.google.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: cloud-sql-instance
spec:
  hosts:
  - www.googleapis.com
  ports:
  - name: tcp
    number: 3306
    protocol: tcp
  addresses:
  - 104.x.x.x
  endpoints:
  - address: 104.x.x.x
  location: MESH_EXTERNAL
  resolution: STATIC

I'm also using mTLS. I'm not currently using the private IP approach - but if I we're, since I'm using mTLS in Istio, and my CloudSQL is using SSL - I'm not sure the best way to set this up?

v-i-o · 2019-11-21T15:23:44Z

This issue has no valid solution atmo and should be kept open.
I can confirm that cloudsql proxy is not working with Istio on GCP. The error is
oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp 74.125.206.84:443: getsockopt: connection refused
Added service entries didn't helped.
However interesting thing is that just after turning istio on, the very first pods started normally with sql proxy working ok. Next generations were not able to start up.

howardjohn · 2019-11-21T15:43:23Z

sigh

akandybaev · 2019-11-29T09:13:23Z

I'm quite new to Istio, but I think I kind of managed to make it work.

I've created the following service entry in the namespace I'm running my service:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: www.googleapis.com
spec:
  hosts:
  - www.googleapis.com
  - oauth2.googleapis.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL

and then restarted Istio control plane (update: first, I assumed that restarting Pilot was enough, but it's not the case). Then after a few failures pod starts up correctly and after that new pods start up correctly with the first attempt.
I'm still trying to figure out what goes wrong and how exactly this "fix" makes it work but I figured it makes sense to share with you.

donnyv12 · 2019-12-22T19:08:38Z

Seeing this issue as well. I'm using Private IP for Cloud SQL and VPC Alias, but have no luck getting the cloudsql-proxy to connect after installing the istio sidecar

istio-policy-bot · 2020-03-06T05:00:25Z

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2019-11-21. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

L-U-C-K-Y · 2020-05-24T14:37:08Z

Hi all

We are all also facing issues using a Cloud SQL sidecar.

Our pods authenticate with localhost, which uses the sidecar to talk to Cloud SQL. While it was working perfectly fine without istio, we are getting connection refused errors.

Has someone a working config, preferably with mTLS?

Much appreciated!

gustavovalverde · 2020-05-24T15:16:34Z

@LucBas if using the proxy is not 100% required, just connect directly using CloudSQL Private IP (which is basically the same the Proxy does).

L-U-C-K-Y · 2020-05-24T15:24:39Z

Hi @gustavovalverde

Thanks for your answer!

When using the CloudSQL private IP, would we still have to apply the ServiceEntry's from above?

If yes, I would prefer to continue using the CloudSQL proxies for security and performance as Google states it.
Further, the GKE cluster is in the default VPC, not sure if the connection to the Cloud SQL instance works in this case as the docs state:

While not as secure, it is possible to connect from a VPC-native GKE cluster to a Cloud SQL instance on the same VPC using private IP without the proxy.

Dev25 · 2020-05-24T16:43:01Z

Using the cloudsql proxy does provide the benefit of automating the SSL cert setup to your SQL instance vs directly connecting to private IP.

I've been using cloudsql proxy w/ istio 1.4+ for a while now as both a deployment + sidecar approach, one of the problems that i've seen is the race condition during startup that leads to the connection refused errors for the proxy trying to talk to google APIs during startup.

The workaround i've been using until sidecars KEP is available is to simply use shell/curl to wait until the proxy is ready before starting up cloud_sql_proxy (rather then just adding a sleep 10 delay)

Unless you are running with a restricted mesh REGISTRY_ONLY you then don't need to create any ServiceEntry unless you want additional observability context.

  command: ["/bin/sh", "-c"]
  args:
  - |
    while ! curl -s -f http://127.0.0.1:15020/healthz/ready; do sleep 5; done
    exec /cloud_sql_proxy -dir=/cloudsql -term_timeout=180s
  env:
  - name: INSTANCES
    value: PROJECT:REGION:INSTANCE=tcp:0.0.0.0:3306

Note: Recent 1.16+ proxy releases have moved to distroless/no shell so you either need to build your custom image until GoogleCloudPlatform/cloud-sql-proxy#371 or use a older image.

FROM alpine:3.11.6

ENV FILE="https://storage.googleapis.com/cloudsql-proxy/v1.17/cloud_sql_proxy.linux.amd64"
RUN apk add --no-cache curl && \
    curl -SL --output /cloud_sql_proxy "$FILE" && \
    chmod +x /cloud_sql_proxy

CMD ["/cloud_sql_proxy"]

gustavovalverde · 2020-05-24T20:29:16Z

@LucBas I don't use the Service Entry, there's no need to using the Private IP. There's additional security with the Proxy, which you can also configure in Cloud SQL; but I can assure you there's no additional performance.

If you'd like additional performance, you'd be better using Pgbouncer instead of the Proxy.

A VPC-native GKE cluster is a cluster with ip-alias enabled.

Eloverflow · 2021-02-24T19:02:24Z

Using the cloudsql proxy does provide the benefit of automating the SSL cert setup to your SQL instance vs directly connecting to private IP.

I've been using cloudsql proxy w/ istio 1.4+ for a while now as both a deployment + sidecar approach, one of the problems that i've seen is the race condition during startup that leads to the connection refused errors for the proxy trying to talk to google APIs during startup.

The workaround i've been using until sidecars KEP is available is to simply use shell/curl to wait until the proxy is ready before starting up cloud_sql_proxy (rather then just adding a sleep 10 delay)

Unless you are running with a restricted mesh REGISTRY_ONLY you then don't need to create any ServiceEntry unless you want additional observability context.
  command: ["/bin/sh", "-c"]
  args:
  - |
    while ! curl -s -f http://127.0.0.1:15020/healthz/ready; do sleep 5; done
    exec /cloud_sql_proxy -dir=/cloudsql -term_timeout=180s
  env:
  - name: INSTANCES
    value: PROJECT:REGION:INSTANCE=tcp:0.0.0.0:3306
Note: Recent 1.16+ proxy releases have moved to distroless/no shell so you either need to build your custom image until GoogleCloudPlatform/cloudsql-proxy#371 or use a older image.
FROM alpine:3.11.6

ENV FILE="https://storage.googleapis.com/cloudsql-proxy/v1.17/cloud_sql_proxy.linux.amd64"
RUN apk add --no-cache curl && \
    curl -SL --output /cloud_sql_proxy "$FILE" && \
    chmod +x /cloud_sql_proxy

CMD ["/cloud_sql_proxy"]

Thanks @Dev25 - That was definitely my issue

aaabramov · 2021-09-19T12:48:26Z

Leaving a checklist that probably will save few hours for someone like me.

Check that you have containerPort defined:

ports:
- containerPort: 5432

Check that service port is defined, targets to correct container port, and starts with tcp:

type: ClusterIP # check
spec:
  ports:
  - name: tcp
    protocol: TCP
    port: 5432       # check
    targetPort: 5432 # check

Check binding address and port for cloudsql binary:

containers:
- command:
  - /cloud_sql_proxy
  - -instances=PROJECT:COMPUTE_ZONE:INSTANCE=tcp:0.0.0.0:5432 # check this: tcp:0.0.0.0: won't produce any error.
  - -credential_file=/secrets/cloudsql/credentials.json
  - -term_timeout=30s
  - -use_http_health_check  # Since 1.25.0
  - -health_check_port=8090 # See: https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/main/examples/k8s-health-check/proxy_with_http_health_check.yaml

loeffel-io · 2024-02-04T15:50:01Z

this worked for my private instance:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: %{service}-googleapis
spec:
  hosts:
    - sqladmin.googleapis.com
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: %{service}-postgres
spec:
  hosts:
    - sqladmin.googleapis.com
  addresses:
    - %{dbPrivateIp}
  ports:
    - name: tcp
      number: 3307
      protocol: TCP
  location: MESH_EXTERNAL

While the Cloud SQL Auth Proxy can listen on any port, it creates outgoing or egress connections to your Cloud SQL instance only on port 3307. Because Cloud SQL Auth Proxy calls APIs through the domain name sqladmin.googleapis.com, which does not have a fixed IP address, all egress TCP connections on port 443 must be allowed. If your client machine has an outbound firewall policy, make sure it allows outgoing connections to port 3307 on your Cloud SQL instance's IP.

I have no idea why but for my staging project it was required to add also this service entry:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: %{service}-googleapis-internal
spec:
  hosts:
    - metadata.google.internal
  addresses:
    - 169.254.169.254
  ports:
    - number: 80
      name: http
      protocol: HTTP
  resolution: STATIC
  location: MESH_EXTERNAL
  endpoints:
    - address: 169.254.169.254

rshriram added the area/networking label Jul 2, 2018

bradenwright mentioned this issue Feb 13, 2019

2nd Service Entry causes both dns entries to route to 1st ServiceEntry #11741

Closed

gustavovalverde mentioned this issue Aug 26, 2019

Can't use default postgresql Unix Socket directory GoogleCloudPlatform/cloud-sql-proxy#301

Closed

jh-sz mentioned this issue Oct 18, 2019

Connecting to AWS RDS in Istio Service Mesh #5580

Closed

istio-policy-bot added the lifecycle/to-be-closed label Oct 30, 2019

geeknoid removed the lifecycle/to-be-closed label Nov 6, 2019

istio-policy-bot closed this as completed Nov 6, 2019

istio-policy-bot added the lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. label Nov 6, 2019

howardjohn reopened this Nov 21, 2019

istio-policy-bot closed this as completed Nov 21, 2019

howardjohn reopened this Nov 21, 2019

istio-policy-bot closed this as completed Nov 21, 2019

howardjohn reopened this Nov 21, 2019

howardjohn mentioned this issue Nov 21, 2019

istio policy bot repeatedly closes things that have been reopened #19120

Closed

istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Feb 20, 2020

istio-policy-bot closed this as completed Mar 6, 2020

jwillker mentioned this issue Jul 8, 2020

Error initializing cloudprober google/cloudprober#432

Closed

Can't connect to Google Cloud Sql #6593

Can't connect to Google Cloud Sql #6593

Comments

xolott commented Jun 25, 2018

ahmmadkour commented Jul 16, 2018 • edited Loading

veryhumble commented Aug 21, 2018 • edited Loading

yskopets commented Sep 12, 2018

danielfoehrKn commented Oct 10, 2018 • edited Loading

danielfoehrKn commented Oct 11, 2018 • edited Loading

ThusharaAma commented Nov 1, 2018

vadimeisenbergibm commented Nov 12, 2018

mr-smithers-excellent commented Nov 17, 2018

ermik commented Jan 29, 2019 • edited Loading

vadimeisenbergibm commented Jan 29, 2019

Kampe commented Apr 25, 2019

vadimeisenbergibm commented Apr 25, 2019

Kampe commented Apr 30, 2019 • edited Loading

vadimeisenbergibm commented May 1, 2019

Kampe commented May 2, 2019 • edited Loading

ermik commented May 2, 2019

Kampe commented May 2, 2019 • edited Loading

ermik commented May 3, 2019

davinkevin commented Jun 6, 2019

Kampe commented Jun 6, 2019 • edited Loading

mr-smithers-excellent commented Jun 6, 2019

ahmed1smael commented Jun 27, 2019

bussyjd commented Jul 31, 2019

Kampe commented Aug 7, 2019 • edited Loading

gustavovalverde commented Aug 26, 2019 • edited Loading

gustavovalverde commented Sep 23, 2019

gustavovalverde commented Nov 3, 2019

gustavovalverde commented Nov 6, 2019

Jumwah commented Nov 7, 2019

v-i-o commented Nov 21, 2019

howardjohn commented Nov 21, 2019

akandybaev commented Nov 29, 2019 • edited Loading

donnyv12 commented Dec 22, 2019

istio-policy-bot commented Mar 6, 2020

L-U-C-K-Y commented May 24, 2020

gustavovalverde commented May 24, 2020

L-U-C-K-Y commented May 24, 2020

Dev25 commented May 24, 2020

gustavovalverde commented May 24, 2020

Eloverflow commented Feb 24, 2021

aaabramov commented Sep 19, 2021 • edited Loading

loeffel-io commented Feb 4, 2024 • edited Loading

ahmmadkour commented Jul 16, 2018 •

edited

Loading

veryhumble commented Aug 21, 2018 •

edited

Loading

danielfoehrKn commented Oct 10, 2018 •

edited

Loading

danielfoehrKn commented Oct 11, 2018 •

edited

Loading

ermik commented Jan 29, 2019 •

edited

Loading

Kampe commented Apr 30, 2019 •

edited

Loading

Kampe commented May 2, 2019 •

edited

Loading

Kampe commented May 2, 2019 •

edited

Loading

Kampe commented Jun 6, 2019 •

edited

Loading

Kampe commented Aug 7, 2019 •

edited

Loading

gustavovalverde commented Aug 26, 2019 •

edited

Loading

akandybaev commented Nov 29, 2019 •

edited

Loading

aaabramov commented Sep 19, 2021 •

edited

Loading

loeffel-io commented Feb 4, 2024 •

edited

Loading