SNS publish issue with CredentialsProviderError

[tdschwebach]

We're writing code in TypeScript v4.3.2, running NodeJs v14 and using "@aws-sdk/client-sns": "3.20.0".

We're able to publish messages for a while, sometimes days, sometimes hours and then we start receiving the following error message out of the blue:

error: {"tryNextLink":false,"name":"CredentialsProviderError","$metadata":{"attempts":1,"totalRetryDelay":0}}
message: Could not load credentials from any providers
metadata: { [-]
attempts: 1
totalRetryDelay: 0
}
name: CredentialsProviderError
stack: CredentialsProviderError: Could not load credentials from any providers
at providers (/app/node_modules/@aws-sdk/credential-provider-node/dist/cjs/index.js:58:19)
at /app/node_modules/@aws-sdk/property-provider/dist/cjs/chain.js:20:28
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async SignatureV4.signRequest (/app/node_modules/@aws-sdk/signature-v4/dist/cjs/SignatureV4.js:84:29)
at async /app/node_modules/@aws-sdk/middleware-signing/dist/cjs/middleware.js:14:22
at async StandardRetryStrategy.retry (/app/node_modules/@aws-sdk/middleware-retry/dist/cjs/StandardRetryStrategy.js:51:46)
at async /app/node_modules/@aws-sdk/middleware-logger/dist/cjs/loggerMiddleware.js:6:22
}

We create our SNSClient as follows and it's created once, when the service starts up.
this.snsClient = new SNSClient({ region: 'us-west-2' });

To publish we have a method that we call with this code:
return this.snsClient
.send(new PublishCommand(params))
.then(response => {
this.logger.log(Sent mid: ${response.MessageId});
return true;
})
.catch(error => {
this.logger.error(error);
return false;
});

We're running inside K8s and restarting the pod resolves the issue although it would be nice to make this issue go away altogether, which is the goal of thread. We're only publishing a few hundred messages a day.

[AllanZhengYP]

Hi @tdschwebach Could it caused by the flacky credential provider used in the container? If you use K8s credential solution like kube2iam, sometimes the credential process maybe down and cause the SDK to fail continuously. I have a fix will possibly the issue you mentioned already: #2680

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

[AllanZhengYP]

We notice some customers are getting intermittent failures while fetching credentials from STS. This issue became more prevalent in US-EAST-1 region using Amazon EKS managed AWS Identity and Access Management (IAM) OpenID Connect (OIDC) external identity provider (IdP).

If you have the similar issue, please refer to the comment below for guidance of debugging:
#2680 (comment)