Skip to content

by default fromTokenFile() and client-sts uses us-east-1 instead of native region #7452

@vytas-maciulskis

Description

@vytas-maciulskis

Checkboxes for prior research

Describe the bug

Hey,

I have the backend services, which are running in the EKS eu-west-2 region, are using a web tokens to reach out to the AWS Secret service. During 20th of October 2025 outage in us-east-1, services were unable to retrieve tokens to access AWS Secrets. It was weird, as it shouldn't be reaching eu-east-1, as everything resides in eu-west-2. After digging deeper, I have found out that @aws-sdk/client-sts, in conjunction with fromTokenFile(), does not honour AWS region, as it should (at least per documentation).
For example
Environment variables available on the EKS pod:

export AWS_DEFAULT_REGION='eu-west-2'
export AWS_REGION='eu-west-2'
export AWS_ROLE_ARN='arn:aws:iam::xxxxxxxxxxxx:role/basic/ew2-dev-ms-service-role'
export AWS_ROLE_SESSION_NAME='dev-ms-service'
export AWS_SECRETS='ew2-dev/ms-service'
export AWS_SECRET_NAME='ew2-dev/ms-service'
export AWS_STS_REGIONAL_ENDPOINTS='regional'
export AWS_WEB_IDENTITY_TOKEN_FILE='/var/run/secrets/eks.amazonaws.com/serviceaccount/token'

Code, which is used to get the secrets from AWS Secrets - working version with clientConfig: { region: "eu-west-2" }, in fromTokenFile()

const {
  SecretsManagerClient,
  GetSecretValueCommand,
} = require("@aws-sdk/client-secrets-manager");
const { fromTokenFile } = require("@aws-sdk/credential-provider-web-identity");

async function getSecret(secretName) {

  const logger = {
    debug: (...args) => console.debug("AWS SDK DEBUG:", ...args),
    info: (...args) => console.info("AWS SDK INFO:", ...args),
    warn: (...args) => console.warn("AWS SDK WARN:", ...args),
    error: (...args) => console.error("AWS SDK ERROR:", ...args),
  };

    const credentials = fromTokenFile({
      clientConfig: { region: "eu-west-2" },
      logger: logger,
    });

    const client = new SecretsManagerClient({
      region: "eu-west-2",
      credentials,
      logger: logger,
    });

    try {
      const command = new GetSecretValueCommand({ SecretId: secretName });
      const response = await client.send(command);
      return response;
    } catch (err) {
        console.error("Error:", err);
    }
}

logger just for debug information from client-sts

Debug messages from AWS SDK:

AWS SDK DEBUG: @smithy/property-provider -> Not found in config files w/ profile [default]: NODE_AUTH_SCHEME_PREFERENCE_CONFIG_KEY
AWS SDK DEBUG: @aws-sdk/credential-provider-web-identity - fromTokenFile
AWS SDK DEBUG: @aws-sdk/credential-provider-web-identity - fromWebToken
AWS SDK DEBUG: @aws-sdk/client-sts::resolveRegion accepting first of: eu-west-2 (provider) undefined (parent client) us-east-1 (STS default)
AWS SDK DEBUG: @smithy/property-provider -> Not found in ENV: ENV_USE_FIPS_ENDPOINT
AWS SDK DEBUG: @smithy/property-provider -> Not found in config files w/ profile [default]: CONFIG_USE_FIPS_ENDPOINT
AWS SDK DEBUG: @smithy/property-provider -> Not found in ENV: ENV_USE_DUALSTACK_ENDPOINT

We can see, that
@aws-sdk/client-sts::resolveRegion accepting first of: eu-west-2 (provider) undefined (parent client) us-east-1 (STS default) uses eu-west-2 region. And in the tracing, I see, that a call was made to https://sts.eu-west-2.amazonaws.com

Example when fromTokenFile() is without clientConfig: { region: "eu-west-2" },

const {
  SecretsManagerClient,
  GetSecretValueCommand,
} = require("@aws-sdk/client-secrets-manager");
const { fromTokenFile } = require("@aws-sdk/credential-provider-web-identity");

async function getSecret(secretName) {

  const logger = {
    debug: (...args) => console.debug("AWS SDK DEBUG:", ...args),
    info: (...args) => console.info("AWS SDK INFO:", ...args),
    warn: (...args) => console.warn("AWS SDK WARN:", ...args),
    error: (...args) => console.error("AWS SDK ERROR:", ...args),
  };

    const credentials = fromTokenFile({
      logger: logger,
    });

    const client = new SecretsManagerClient({
      region: "eu-west-2",
      credentials,
      logger: logger,
    });

    try {
      const command = new GetSecretValueCommand({ SecretId: secretName });
      const response = await client.send(command);
      return response;
    } catch (err) {
        console.error("Error:", err);
    }
}

Debug loglines:

AWS SDK DEBUG: @smithy/property-provider -> Not found in ENV: NODE_AUTH_SCHEME_PREFERENCE_ENV_KEY
AWS SDK DEBUG: @smithy/property-provider -> Not found in config files w/ profile [default]: NODE_AUTH_SCHEME_PREFERENCE_CONFIG_KEY
AWS SDK DEBUG: @aws-sdk/credential-provider-web-identity - fromTokenFile
AWS SDK DEBUG: @aws-sdk/credential-provider-web-identity - fromWebToken
AWS SDK DEBUG: @aws-sdk/client-sts::resolveRegion accepting first of: undefined (provider) undefined (parent client) us-east-1 (STS default)
AWS SDK DEBUG: @smithy/property-provider -> Not found in ENV: ENV_USE_FIPS_ENDPOINT
AWS SDK DEBUG: @smithy/property-provider -> Not found in config files w/ profile [default]: CONFIG_USE_FIPS_ENDPOINT
AWS SDK DEBUG: @smithy/property-provider -> Not found in ENV: ENV_USE_DUALSTACK_ENDPOINT
AWS SDK DEBUG: @smithy/property-provider -> Not found in config files w/ profile [default]: CONFIG_USE_DUALSTACK_ENDPOINT

Here we can see that
AWS SDK DEBUG: @aws-sdk/client-sts::resolveRegion accepting first of: undefined (provider) undefined (parent client) us-east-1 (STS default)
resolveRegion got undefined, as region was not specified and the call went to https://sts.us-east-1.amazonaws.com

So from this, I suspect that out-of-the-box fromTokenFile() is not region aware. But as per https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html table - it should be.

For now we have placed AWS_ENDPOINT_URL_STS env var to override this default behaviour.

Regression Issue

  • Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/credential-provider-web-identity@3.914.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.16.0

Reproduction Steps

generate "fake token" file, just to make SDK happy and pre-populate the environment variables required to go to web token scenario

export AWS_ROLE_ARN=arn:aws:iam::123456789012:role/MyTestRole
export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/fake-token
export AWS_REGION=eu-west-2
echo 'fake.jwt.token' > /tmp/fake-token

logHttpHost() part generated by LLM - as AWS SDK do not print out the STS endpoint, which was used in the call.

This script where fromTokenFile() without region in the constructor

  const https = require('https');
  const http = require('http');

  const {
    SecretsManagerClient,
    GetSecretValueCommand,
  } = require("@aws-sdk/client-secrets-manager");
  const { fromTokenFile } = require("@aws-sdk/credential-provider-web-identity");

  /**
   * Patches the global HTTP and HTTPS agents to log the host and port
   * for every outgoing request made through the default agents.
   */
  function logHttpHost() {
    console.log("--- Patching HTTP/HTTPS agents to log all outgoing hosts ---");

    const logHostWrapper = (req) => {
      console.log(`📡 OUTGOING REQUEST: ${req.method} ${req.host}:${req.port || (req.defaultPort === 443 ? 443 : 80)}`);
    };

    const originalHttpsRequest = https.Agent.prototype.createConnection;
    https.Agent.prototype.createConnection = function(options, cb) {
      return originalHttpsRequest.call(this, options, cb);
    };

    const originalHttpsRequestFunc = https.request;
    https.request = (...args) => {
      const req = originalHttpsRequestFunc.apply(https, args);
      req.on('socket', (socket) => {
        console.log(`📡 HTTPS Request to Host: ${req.host}`);
      });
      return req;
    };

    const originalHttpRequestFunc = http.request;
    http.request = (...args) => {
        const req = originalHttpRequestFunc.apply(http, args);
        req.on('socket', (socket) => {
             console.log(`📡 HTTP Request to Host: ${req.host}`);
        });
        return req;
    };

    if (http.globalAgent) http.globalAgent.on('request', logHostWrapper);
    if (https.globalAgent) https.globalAgent.on('request', logHostWrapper);

    console.log("------------------------------------------------------------");
  }

  async function getSecret() {

    logHttpHost();

      const client = new SecretsManagerClient({
        region: "eu-west-2",
        credentials: fromTokenFile(),
      });

      try {
        const command = new GetSecretValueCommand({ SecretId: 'not-existing' });
        const response = await client.send(command);
        console.log("✅ Secret:", response.SecretString);
      } catch (err) {
        console.error("Error:", err);
      }
  }

  getSecret();

you will see 📡 HTTPS Request to Host: sts.us-east-1.amazonaws.com

Same script as above, but fromTokenFile() has region

  const https = require('https');
  const http = require('http');

  const {
    SecretsManagerClient,
    GetSecretValueCommand,
  } = require("@aws-sdk/client-secrets-manager");
  const { fromTokenFile } = require("@aws-sdk/credential-provider-web-identity");

  /**
   * Patches the global HTTP and HTTPS agents to log the host and port
   * for every outgoing request made through the default agents.
   */
  function logHttpHost() {
    console.log("--- Patching HTTP/HTTPS agents to log all outgoing hosts ---");

    const logHostWrapper = (req) => {
      console.log(`📡 OUTGOING REQUEST: ${req.method} ${req.host}:${req.port || (req.defaultPort === 443 ? 443 : 80)}`);
    };

    const originalHttpsRequest = https.Agent.prototype.createConnection;
    https.Agent.prototype.createConnection = function(options, cb) {
      return originalHttpsRequest.call(this, options, cb);
    };

    const originalHttpsRequestFunc = https.request;
    https.request = (...args) => {
      const req = originalHttpsRequestFunc.apply(https, args);
      req.on('socket', (socket) => {
        console.log(`📡 HTTPS Request to Host: ${req.host}`);
      });
      return req;
    };

    const originalHttpRequestFunc = http.request;
    http.request = (...args) => {
        const req = originalHttpRequestFunc.apply(http, args);
        req.on('socket', (socket) => {
             console.log(`📡 HTTP Request to Host: ${req.host}`);
        });
        return req;
    };

    if (http.globalAgent) http.globalAgent.on('request', logHostWrapper);
    if (https.globalAgent) https.globalAgent.on('request', logHostWrapper);

    console.log("------------------------------------------------------------");
  }

  async function getSecret() {

    logHttpHost();

      const client = new SecretsManagerClient({
        region: "eu-west-2",
        credentials: fromTokenFile({clientConfig:{region: "eu-west-2"}}),
      });

      try {
        const command = new GetSecretValueCommand({ SecretId: 'not-existing' });
        const response = await client.send(command);
        console.log("✅ Secret:", response.SecretString);
      } catch (err) {
        console.error("Error:", err);
      }
  }

  getSecret();

Here you will see
📡 HTTPS Request to Host: sts.eu-west-2.amazonaws.com

Observed Behavior

fromTokenFile() - always does call to sts.us-east-1.amazonaws.com, despite AWS_REGION environment variable.

Expected Behavior

With the aws-sdk-js-v3 , STSClient should figure out the region in which it is running and reach out to the regional STS endpoint. At least the documentation states that.

Possible Solution

Add functionality that fromTokenFile() would check AWS_REGION environment variable or update documentation, that in some cases, it is required to specify region.

Additional Information/Context

I assume there could be lots of developers who think that they are using regional STS, while in reality going to us-east-1
Our applications use v3.775 of the SDK, but the same behaviour occurs in the latest version as well.

Disclaimer: I'm sorry if I misread the documentation, and this is normal designed behaviour.

Metadata

Metadata

Assignees

Labels

bugThis issue is a bug.p2This is a standard priority issueresponse-requestedWaiting on additional info and feedback. Will move to \"closing-soon\" in 7 days.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions