"Failure from metadata server" when using GKE and Workload Identity

[bradam12]

We were getting the "Failure from metadata server" when trying to create a signed URL in our existing application. This is an app already running in prod. We're adding an additional feature that requires signed URLs.

I created a barebones test.js file and also used the included sample at https://github.com/googleapis/nodejs-storage/blob/main/samples/generateV4ReadSignedUrl.js as reproducible cases. Versions 2.x, 3.x, 4.x all work properly, and 5.0.0 and up exhibit the error.

Environment details

• OS: buster
• Node.js version: 16.20.2
• npm version: 8.19.4
• @google-cloud/storage version: 5.0.0 and up

Steps to reproduce

1. Run node:16 pod in GKE (we're currently on v1.24.15-gke.1700), with established workload identity, KSA and GSA linked. GSA has Storage Object Admin and Service Account Token Creator.
2. Exec into pod
3. Write the sample file from https://github.com/googleapis/nodejs-storage/blob/main/samples/generateV4ReadSignedUrl.js
4. Install @google-cloud/storage@5
5. Run node test.js bucketname filename
6. Receive error: Failure from metadata server.
7. Repeat for @google-cloud/storage@6 and @google-cloud/storage@7, receive same error

[ddelgrosso1]

Hi @bradam12 is this only happening within GKE? Has it been successful in other environments for you?

[bradam12]

Local is fine. I can spin up a GCP VM and try similarly.

…

On Fri, Oct 27, 2023, 9:33 AM Denis DelGrosso ***@***.***> wrote: Hi @bradam12 <https://github.com/bradam12> is this only happening within GKE? Has it been successful in other environments for you? — Reply to this email directly, view it on GitHub <#2346 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZHDEGD3WCNTYGDCOVRM3YBOZZNAVCNFSM6AAAAAA6SBBAW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBSHEZDQMZVG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ddelgrosso1]

@bradam12 came across this documentation: https://cloud.google.com/knowledge/kb/unable-to-reach-gke-metadata-server-on-workload-identity-enabled-cluster-000004705 is this inline with your problem?

[bradam12]

Unfortunately does not apply. I'm able to curl to the metadata server to get a token correctly.

[bradam12]

I'm no longer working in that env. Closing.