S3 login fails in AWS GovCloud since 9.1.0: sts:GetCallerIdentity is sent to the commercial STS endpoint

[ylangisc]

Summary

Since 9.1.0, S3Session.login() validates credentials with sts:GetCallerIdentity
instead of an S3 request. The STS client is built with a hard-coded commercial endpoint
and a null signing region, so the call always goes to the aws partition. IAM keys
issued in AWS GovCloud (partition aws-us-gov) are unknown there and the login fails
with 403 InvalidClientTokenId — "The security token included in the request is
invalid." — even though the credentials are valid.

Commercial regions are unaffected because sts.amazonaws.com happens to be the correct
partition for them.

Environment

• Reported on Mountain Duck 5.3.1.29526 (bundles Cyberduck core 9.5.2), macOS 15.7.7
• Bookmark hostname s3.us-gov-west-1.amazonaws.com, long-term IAM access key
  (no session token), non-anonymous
• Same key pair works with the AWS CLI, with third-party clients, and with older
  Mountain Duck releases based on Cyberduck ≤ 9.0.x

Steps to reproduce

1. Create an S3 bookmark for AWS GovCloud, e.g. hostname s3-us-gov-west-1.amazonaws.com
   (the bundled S3 GovCloud (US-West) profile), no bucket in the hostname
2. Enter a valid GovCloud IAM access key / secret key
3. Connect

Expected: connection succeeds.
Actual: Login failed – The security token included in the request is invalid.,
followed by a repeating credential prompt.

Log

S3Session - Connect with session tokens TemporaryAccessTokens{accessKeyId='********',
    secretAccessKey='********', sessionToken='', expiryInMilliseconds=-1}
com.amazonaws.request - Sending Request: POST https://sts.amazonaws.com/ /
com.amazonaws.auth.AWS4Signer - AWS4 String to Sign: 'AWS4-HMAC-SHA256
    20260727T123958Z
    20260727/us-east-1/sts/aws4_request
com.amazonaws.request - Received error response:
    AWSSecurityTokenServiceException: The security token included in the request is
    invalid. (Service: AWSSecurityTokenService; Status Code: 403;
    Error Code: InvalidClientTokenId)
STSExceptionMappingService - Map failure ...
KeychainLoginService - Login failed for Session{host=Host{...,
    hostname='s3.us-gov-west-1.amazonaws.com', ...}}

Stack trace:

ExpiredTokenException: Login failed – The security token included in the request is invalid.
  at ch.cyberduck.core.sts.STSExceptionMappingService.map(STSExceptionMappingService.java:42)
  at ch.cyberduck.core.sts.STSAuthorizationService.getCallerIdentity(STSAuthorizationService.java:101)
  at ch.cyberduck.core.s3.S3Session.login(S3Session.java:363)
  at ch.cyberduck.core.KeychainLoginService.authenticate(KeychainLoginService.java:181)
Caused by: AWSSecurityTokenServiceException ... Error Code: InvalidClientTokenId
  at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeGetCallerIdentity(...)
  at ch.cyberduck.core.sts.STSAuthorizationService.getCallerIdentity(STSAuthorizationService.java:95)

(access key id and local IP redacted)

Root cause

1. Every AWS S3 login goes through STS. S3Session.login():

if(isAwsHostname(host.getHostname(), false)
        && StringUtils.isEmpty(RequestEntityRestStorageService.findBucketInHostname(host))
        && !credentials.isAnonymousLogin()) {
    new STSAuthorizationService(host, trust, key, prompt).getCallerIdentity(credentials);
    return;
}

s3-us-gov-west-1.amazonaws.com matches isAwsHostname, so GovCloud always takes this path.

2. The STS client is pinned to the commercial partition. Decompiled constructor of
STSAuthorizationService (9.5.2):

17: aload_1                     // host
18: Host.getProtocol()
21: Protocol.getSTSEndpoint()   → serviceEndpoint
26: aconst_null                 → signingRegion
27: EndpointConfiguration.<init>(String, String)
30: withEndpointConfiguration(...)

• serviceEndpoint comes from S3Protocol.getSTSEndpoint(), which returns the constant
  https://sts.amazonaws.com/. Still hard-coded in 9.5.3-SNAPSHOT.
• signingRegion is literally null — Host.getRegion() is never consulted. That is why
  the request is signed for us-east-1 regardless of the bookmark.

Only Profile.getSTSEndpoint() can override the endpoint, via the profile key
STS Endpoint:

final String value = this.value("STS Endpoint");
return StringUtils.isBlank(value) ? parent.getSTSEndpoint() : value;

Neither S3 GovCloud (US-West).cyberduckprofile nor S3 GovCloud (US-East).cyberduckprofile
in iterate-ch/profiles sets that key, so the bundled GovCloud profiles cannot work
either. Setting Region in the profile has no effect on this code path.

Regression

f44a2d370 (2024-10-31) — "Test credentials using sts:GetCallerIdentity. Skip querying
root for location." — replaced the previous S3-based login validation:

-        final Location.Name location = new S3LocationFeature(S3Session.this, regions).getLocation(home);
+        if(S3Session.isAwsHostname(host.getHostname(), false)) {
+            if(StringUtils.isEmpty(RequestEntityRestStorageService.findBucketInHostname(host))) {
+                final AWSSecurityTokenServiceClientBuilder builder =
+                        AWSSecurityTokenServiceClientBuilder.standard()

The old path issued an S3 request against the configured endpoint and was therefore
partition-correct by construction.

release-9-0-3 (2024-09-18)	does not contain f44a2d370
release-9-1-0 (2024-12-05)	first release containing it

Note that f44a2d370 still used AWSSecurityTokenServiceClientBuilder.standard() without
an explicit endpoint, so the SDK's default region provider chain could in principle apply.
672f2e1c0 (2025-09-22, "Extract common class") routed the login check through
STSAuthorizationService, which pins endpoint and signing region as shown above.

Suggested fix

Deriving both values from the bookmark would address the general case:

• getSTSEndpoint() should resolve the regional STS endpoint from the bookmark region or
  hostname (sts.us-gov-west-1.amazonaws.com for aws-us-gov) rather than returning a
  constant
• EndpointConfiguration should receive a real signingRegion instead of null

Adding STS Endpoint to the two GovCloud profiles would help existing installations
without a client update, but on its own leaves signingRegion = null.

Alternatively, reconsider whether the login check needs STS at all when the credentials
are long-term IAM keys with no session token.

Workaround

Putting the bucket into the hostname (<bucket>.s3-us-gov-west-1.amazonaws.com) makes
findBucketInHostname non-empty, which skips the STS branch entirely.

Not verified

I have no GovCloud account. It is untested whether setting STS Endpoint alone is
sufficient, i.e. whether the SDK infers the signing region from the endpoint hostname when
signingRegion is null, or whether it then fails with a credential scope mismatch.

[dkocher]

Regional endpoints for STS in

• Override default STS endpoints with regional. profiles#195