Workload Identity with AWS & IMDSv2 use expired token

[whs]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Environment details

• OS: Linux
• Node.js version: 16.20.2 (I know it's EOL..)
• npm version: 8.19.4
• google-auth-library version: 9.4.0

Steps to reproduce

1. Setup federation with AWS according to https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds . The JSON file should have imdsv2_session_token_url.
2. Observe that API calls to Google (we're testing with service account signJwt) are working normally
3. Wait about an hour
4. Observe that API calls are now failing

The error object we receive is:

GaxiosError
    at Gaxios._request (/app/node_modules/gaxios/build/src/gaxios.js:142:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async AwsClient.getAwsRoleName (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:217:26)
    at async AwsRequestSigner.getCredentials (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:110:34)
    at async AwsRequestSigner.getRequestOptions (/app/node_modules/google-auth-library/build/src/auth/awsrequestsigner.js:66:40)
    at async AwsClient.retrieveSubjectToken (/app/node_modules/google-auth-library/build/src/auth/awsclient.js:125:25)
    at async AwsClient.refreshAccessTokenAsync (/app/node_modules/google-auth-library/build/src/auth/baseexternalclient.js:288:30)
    at async AwsClient.getAccessToken (/app/node_modules/google-auth-library/build/src/auth/baseexternalclient.js:165:13)
    [scrubbed] {
  config: {
    url: 'http://169.254.169.254/latest/meta-data/iam/security-credentials',
    method: 'GET',
    responseType: 'text',
    headers: {
      'x-aws-ec2-metadata-token': '<private>',
      'User-Agent': 'google-api-nodejs-client/9.6.1',
      'x-goog-api-client': 'gl-node/16.20.2'
    },
    paramsSerializer: [Function: paramsSerializer],
    validateStatus: [Function: validateStatus],
    errorRedactor: [Function: defaultErrorRedactor]
  },
  response: {
    config: {
      url: 'http://169.254.169.254/latest/meta-data/iam/security-credentials',
      method: 'GET',
      responseType: 'text',
      headers: [Object],
      paramsSerializer: [Function: paramsSerializer],
      validateStatus: [Function: validateStatus],
      errorRedactor: [Function: defaultErrorRedactor]
    },
    data: '',
    headers: {
      connection: 'close',
      'content-length': '0',
      'content-type': 'text/plain',
      date: 'Wed, 07 Feb 2024 05:38:23 GMT',
      server: 'EC2ws'
    },
    status: 401,
    statusText: 'Unauthorized',
    request: {
      responseURL: 'http://169.254.169.254/latest/meta-data/iam/security-credentials'
    }
  },
  error: undefined,
  status: 401,
  [Symbol(gaxios-gaxios-error)]: '6.2.0'
}

I think this is because in

google-auth-library-nodejs/src/auth/awsclient.ts

Lines 151 to 161 in 3b19e9c

	if (!this.awsRequestSigner) {
	const metadataHeaders: Headers = {};
	// Only retrieve the IMDSv2 session token if both the security credentials and region are
	// not retrievable through the environment.
	// The credential config contains all the URLs by default but clients may be running this
	// where the metadata server is not available and returning the credentials through the environment.
	// Removing this check may break them.
	if (this.shouldUseMetadataServer() && this.imdsV2SessionTokenUrl) {
	metadataHeaders['x-aws-ec2-metadata-token'] =
	await this.getImdsV2SessionToken();
	}

the IMDSv2 Session Token API is only called once when this.awsRequestSigner is null, then is reused for all subsequent requests.

However in getImdsV2SessionToken the x-aws-ec2-metadata-token-ttl-seconds is only configured to be 300s:

google-auth-library-nodejs/src/auth/awsclient.ts

Line 242 in 3b19e9c

headers: {'x-aws-ec2-metadata-token-ttl-seconds': '300'},

so after the application run for some times the GCP token expires and refreshing of the token fails because it use expired IMDSv2 token.

Semi-related but I checked how the Go SDK has implemented and it seems to be 1:1 match to this behavior and may also have similar problem https://cs.opensource.google/go/x/oauth2/+/refs/tags/v0.17.0:google/internal/externalaccount/aws.go;l=304