no longer works on Cloud Run on GKE

[samos123]

Steps to reproduce:

gcloud container clusters create cluster-1 \
  --addons=HttpLoadBalancing,CloudRun \
  --machine-type=n1-standard-2 \
  --enable-stackdriver-kubernetes \
  --region=us-central1 \
  --node-locations=us-central1-a

gcloud config set run/cluster cluster-1
gcloud config set run/cluster_location us-central1

gcloud run deploy hello --image gcr.io/cloudrun/hello --platform gke

export GW_IP=$(kubectl get svc -n gke-system istio-ingress | grep istio-ingress | awk '{ print $4}')

curl -s -H "Host: hello.default.example.com" $GW_IP

That curl returns the following:

                        <p>

                                        This container <strong>does not</strong> seem to be running
                                        on Cloud Run.

                        </p>
                   

In previous versions of this image ~6 months ago, the exact same steps did not return This container does not seem to be running on cloud run. So something must have changed on the Cloud Run side or in this image.

[ahmetb]

My suspicion is that our Region detection causes this.
On a VM, this returns 404:

curl -H "metadata-flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/region

as there's /zone on VMs.

[samos123]

I'm not sure why it worked before though, but looking at the code it must be something related to metadata. I will investigate further later tonight and report back.

[ahmetb]

I mean my comment alone explains at least why it doesn't work. Other variables that trigger this situation seem like they work fine on a VM environment.

[samos123]

Yeah you were spot on. I verified by doing another test this time with GKE_METADATA and workload identity enabled. These commands were run from the application pod:

/ # wget --header "metadata-flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/region
Connecting to metadata.google.internal (169.254.169.254:80)
wget: server returned error: HTTP/1.1 404 Not Found
/ # wget --header "metadata-flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/zone
Connecting to metadata.google.internal (169.254.169.254:80)
saving to 'zone'
zone                 100% |************************************************************************************************|    41  0:00:00 ETA
'zone' saved

I'm going to try create a quick fork that doesn't get the region and see if the rest still works as is. Thanks so much for the pointers saved me a lot of time debuggin this.

[samos123]

This samos123/cloud-run-hello@5584ed7 fixed the issue for Cloud Run on GKE as hacky workaround. I will provide a cleaner fix and submit PR later this week.