ComputeCredential.IsRunningOnComputeEngine doesn't work

[cilliemalan]

The method IsRunningOnComputeEngineNoCache checks that the metadata server returns the header Metadata-Flavor: Google. However, that endpoint will return an error if that same header is not also supplied in the request.

Notice in the following, curl-ing the endpoint does not return the header it's checking for:

# create a pod in my gke cluster
$ kubectl run -it --generator=run-pod/v1 --image google/cloud-sdk --namespace default workload-identity-test

# inside gce:
root@workload-identity-test:/# curl -v http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /computeMetadata/v1/instance/service-accounts/default/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Mon, 10 Jun 2019 13:54:13 GMT
< Content-Length: 94
<
GKE Metadata Server encountered an error: Missing required header "Metadata-Flavor": "Google"
* Curl_http_done: called premature == 0
* Connection #0 to host 169.254.169.254 left intact
root@workload-identity-test:/#

Elsewhere in ComputeCredential it does add the header, but not for IsRunningOnComputeEngine.

I'll create a PR in a few moments

[cilliemalan]

This means that one cannot use the default compute service account or Workload Identity

[jskeet]

I'm very surprised it's not working - it certainly has in the past. I do wonder whether this is a GKE-specific problem.

Note that in your example, you're showing a curl command that tries to fetch a token - whereas the code in IsRunningOnComputeEngine just fetches http://169.254.169.254. I've just tried running curl -v http://169.254.169.254 in a Cloud Shell and it's been absolutely fine... although fetching the access token works without the header from Cloud Shell, too.

Could you try your test in GKE but just fetching http://169.254.169.254 ? I have no issue with adding the header in the test, but I'd like to know more about what's going on.

[cilliemalan]

The / endpoint does the same:

root@workload-identity-test:~# curl -v http://169.254.169.254/
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET / HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Mon, 10 Jun 2019 18:33:52 GMT
< Content-Length: 94
<
GKE Metadata Server encountered an error: Missing required header "Metadata-Flavor": "Google"
* Curl_http_done: called premature == 0
* Connection #0 to host 169.254.169.254 left intact
root@workload-identity-test:~#

I get the feeling it might be a Workload Identity specific issue. I'm just guessing but maybe a normal GCE instance has an access token in the well-known location, whereas a k8s pod with Workflow Identity doesn't have one (at least not a in the JSON format). Or GOOGLE_APPLICATION_CREDENTIALS

[cilliemalan]

I tried the same request from a normal GCE instance and it returns 200. From k8s the metadata server behaves differently it seems.

From GCE:

cilliejr@instance-1:~/.config/gcloud$ curl -v http://169.254.169.254/
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET / HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Metadata-Flavor: Google
< Content-Type: application/text
< Date: Mon, 10 Jun 2019 18:45:36 GMT
< Server: Metadata Server for VM
< Content-Length: 22
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< 
0.1/
computeMetadata/
* Curl_http_done: called premature == 0
* Connection #0 to host 169.254.169.254 left intact
cilliejr@instance-1:~/.config/gcloud$