Updating AWSSDK to v4 causes `Invalid authentication payload` exceptions from AWS MSK brokers

[janv97]

Describe the bug

After upgrading AWSSDK.SecurityToken to version 4.x, we began encountering sporadic errors from the Kafka brokers with the message: Invalid authentication payload. These errors occur intermittently, typically every 5 to 30 minutes, a few times per day.

We have since reverted to AWSSDK version 3.x, which resolves the issue for now.

I’m currently in contact with AWS Support, and they recommended that I post the issue here as well. Additionally, I couldn’t find any relevant information in the migration docs.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

There are no authentication failures logged, nor any documentation explaining what changes are required to avoid these exceptions.

Current Behavior

Exception:

/bootstrap: SASL authentication error: [53fb19f7-525a-47e4-a7cd-9b7bed03310c]: Invalid authentication payload (after 101ms in state AUTH_REQ)

Reproduction Steps

Upgrade

    <PackageReference Include="AWS.MSK.Auth" Version="1.0.0" />
    <PackageReference Include="AWSSDK.Glue" Version="3.7.408.2" />

To

    <PackageReference Include="AWS.MSK.Auth" Version="1.1.1" />
    <PackageReference Include="AWSSDK.Glue" Version="4.0.16.2" />
    <PackageReference Include="AWSSDK.SecurityToken" Version="4.0.2.2" />

Possible Solution

No response

Additional Information/Context

OAuthCallback method:

    public void OAuthCallback(IClient client, string principalName)
    {
        try
        {
            // Warning: These are blocking calls. We cannot use async/await here due to Confluent.Kafka limitations.
            (string token, long expiryMs) = _mskTokenGenerator.GenerateAuthTokenAsync(Amazon.RegionEndpoint.EUWest1).Result;

            client.OAuthBearerSetToken(token, expiryMs, principalName);
        }
#pragma warning disable CA1031 // Do not catch general exception types
        catch (Exception ex)
#pragma warning restore CA1031
        {
            client.OAuthBearerSetTokenFailure(ex.ToString());
        }
    }

Consumer builder:

return new ConsumerBuilder<string, TMessageValue>(consumerConfig)
    .SetOAuthBearerTokenRefreshHandler((client, _) => oAuthService.OAuthCallback(client, principalName))
    .SetKeyDeserializer(Deserializers.Utf8)
    .SetValueDeserializer(valueDeserializer)
    .Build();

AWS .NET SDK and/or Package version used

    <PackageReference Include="AWS.MSK.Auth" Version="1.1.1" />
    <PackageReference Include="AWSSDK.Glue" Version="4.0.16.2" />
    <PackageReference Include="AWSSDK.SecurityToken" Version="4.0.2.2" />

Targeted .NET Platform

.NET 8.0

Operating System and version

Linux

[dscpinheiro]

This looks related to #3873 and #3962.

To confirm, can you enable debug logging (prior to invoking the token generator) and check if the SDK is returning invalid credentials? (by invalid I mean expired)

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetricsFormat = LogMetricsFormatOption.JSON;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogMetrics = true;

[janv97]

@dscpinheiro How can we check if the returned credentials are expired? We’ve been seeing some "Session too short" messages, which seem to indicate that the credentials might have expired.

[dscpinheiro]

If you enable logging one of the messages the SDK will show is similar to this:

RefreshingAWSCredentials 37|2025-09-30T14:14:07.357Z|INFO|Determined refreshing credentials should update. Expiration time: 2025-09-30T14:14:07.0 000000Z, Current time: 2025-09-30T14:14:07.3572964Z
AssumeRoleAWSCredentials 42|2025-09-30T14:14:07.692Z|DEBUG|New credentials created for assume role that expire at 2025-09-30T14:29:07.0000000Z

[bdekonin]

Hi @dscpinheiro, I am a collegue of Jan and I am replying on his behalf.
We can not find any logs similar to this, when I search for the keyword credentials these are the only results.

info: AWSSDK[0] Found credentials using the AWS SDK's default credential search

I also see these logs, but I am unaware if these come from the same issue

ERROR|rdkafka#producer-6| [thrd:app]: Failed to acquire SASL OAUTHBEARER token: Confluent.Kafka.KafkaException: Must supply an unexpired token: now=1758369872149ms, exp=1758369872000ms

   at XXX.Kafka.Services.OAuthService.OAuthCallback(IClient client, String principalName, KafkaCluster kafkaCluster)

[dscpinheiro]

We ended up reverting the background refresh change that was introduced in V4, and today's release (https://github.com/aws/aws-sdk-net/releases/tag/4.0.103.0) will have the same behavior as V3.

You'll need to update the service packages too, but let us know if you see the problem again.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.