DefaultCredentialProviderChain timing out when attempting to describe a DynamoDB table

[cavanhussein]

Describe the bug

Hi there,
We are attempting to integrate with the Kotlin AWS SDK for one of our micro services. In our AWS Production environment, there are instances that unexpectedly fail to connect to DynamoDB using the DefaultCredentialProviderChain. Our micro service implements Spring's AbstractHealthIndicator to describe one of our DynamoDB tables during our periodic health check.

Below you can find our Bean configuration for our DynamoDBClient:

import aws.sdk.kotlin.runtime.auth.credentials.DefaultChainCredentialsProvider
import aws.sdk.kotlin.services.dynamodb.DynamoDbClient
import aws.smithy.kotlin.runtime.ExperimentalApi
import aws.smithy.kotlin.runtime.http.engine.crt.CrtHttpEngine
import aws.smithy.kotlin.runtime.net.url.Url
import aws.smithy.kotlin.runtime.telemetry.otel.OpenTelemetryProvider
import io.opentelemetry.api.OpenTelemetry
import org.springframework.boot.context.properties.EnableConfigurationProperties
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration

@Configuration
@EnableConfigurationProperties(DynamoDBConfig::class)
open class DynamoDBConfiguration {
    @OptIn(ExperimentalApi::class)
    @Bean(destroyMethod = "close")
    open fun dynamoDbClient(
        dynamoDBConfig: DynamoDBConfig,
        openTelemetry: OpenTelemetry
    ): DynamoDbClient {
        val crtHttpEngine: CrtHttpEngine = CrtHttpEngine.invoke {
            maxConcurrency = dynamoDBConfig.maxConcurrency.toUInt()
            maxConnections = dynamoDBConfig.maxConnections.toUInt()
        }
        return DynamoDbClient {
            region = dynamoDBConfig.region
            credentialsProvider = DefaultChainCredentialsProvider(region = dynamoDBConfig.region, httpClient = crtHttpEngine)
            httpClient = crtHttpEngine
            if (!dynamoDBConfig.endpointOverride.isNullOrBlank()) {
                endpointUrl = Url.parse(dynamoDBConfig.endpointOverride!!)
            }
            telemetryProvider = OpenTelemetryProvider(openTelemetry)
        }
    }
}

and our DBHealthCheck Bean configuration looks like:

@Component
class DbHealthCheck(
    private val dynamoDbClient: DynamoDbClient,
    private val dynamoDBConfig: DynamoDBConfig,
    private val featureFlagService: FeatureFlagService,
    private val metadataCacheDispatcher: CoroutineDispatcher,
) : AbstractHealthIndicator() {

    private val logger = KotlinLogging.logger {}

    override fun doHealthCheck(builder: Health.Builder) {
        when {
            !featureFlagService.hasFeature(FeatureKey.CHECK_DB_CONNECTION) -> {
                logger.info { "health check disabled" }
            }

            runBlocking(metadataCacheDispatcher) {
                dynamoDbClient.describeTable(DescribeTableRequest {
                    tableName =
                        dynamoDBConfig.tables!!.metadataCache // nullability ignored intentionally since we want this to fail miserably if tables are null
                })
            }.table?.tableName != dynamoDBConfig.tables!!.metadataCache -> {
                logger.error { "db connection down" }
                builder.down().withDetails(mapOf("db" to mapOf(dynamoDBConfig.tables!!.metadataCache to "DOWN")))
            }

            else -> {
                logger.info { "db connection up" }
                builder.up().withDetails(mapOf("db" to mapOf(dynamoDBConfig.tables!!.metadataCache to "UP")))
            }
        }
    }
}

Finally our timeout error we're seeing looks like:

{"timestamp":"2025-08-05 20:30:32.058","level":"TRACE","thread":"metadata-cache-dispatcher-190","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.sdk.kotlin.runtime.auth.credentials.SystemPropertyCredentialsProvider","message":"Attempting to load credentials from system properties aws.accessKeyId/aws.secretAccessKey/aws.sessionToken","context":"default"}
{"timestamp":"2025-08-05 20:30:32.058","level":"DEBUG","thread":"metadata-cache-dispatcher-190","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from SystemPropertyCredentialsProvider: Missing value for system property `aws.accessKeyId`","context":"default"}
{"timestamp":"2025-08-05 20:30:32.058","level":"TRACE","thread":"metadata-cache-dispatcher-190","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from EnvironmentCredentialsProvider","context":"default"}
{"timestamp":"2025-08-05 20:30:32.058","level":"TRACE","thread":"metadata-cache-dispatcher-190","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.sdk.kotlin.runtime.auth.credentials.EnvironmentCredentialsProvider","message":"Attempting to load credentials from env vars AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN","context":"default"}
{"timestamp":"2025-08-05 20:30:32.058","level":"DEBUG","thread":"metadata-cache-dispatcher-190","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from EnvironmentCredentialsProvider: Missing value for environment variable `AWS_ACCESS_KEY_ID`","context":"default"}
{"timestamp":"2025-08-05 20:30:32.058","level":"TRACE","thread":"metadata-cache-dispatcher-190","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@72fc1632","context":"default"}
{"timestamp":"2025-08-05 20:30:32.058","level":"DEBUG","thread":"metadata-cache-dispatcher-190","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@72fc1632: Required field `roleArn` could not be automatically inferred for StsWebIdentityCredentialsProvider. Either explicitly pass a value, set the environment variable `AWS_ROLE_ARN`, or set the JVM system property `aws.roleArn`","context":"default"}
{"timestamp":"2025-08-05 20:30:32.058","level":"TRACE","thread":"metadata-cache-dispatcher-190","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from ProfileCredentialsProvider","context":"default"}
{"timestamp":"2025-08-05 20:30:32.059","level":"DEBUG","thread":"metadata-cache-dispatcher-48","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.sdk.kotlin.runtime.auth.credentials.ProfileCredentialsProvider","message":"Loading credentials from profile `default`","context":"default"}
{"timestamp":"2025-08-05 20:30:32.059","level":"DEBUG","thread":"metadata-cache-dispatcher-48","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from ProfileCredentialsProvider: could not find source profile default","context":"default"}
{"timestamp":"2025-08-05 20:30:32.059","level":"TRACE","thread":"metadata-cache-dispatcher-48","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from EcsCredentialsProvider","context":"default"}
{"timestamp":"2025-08-05 20:30:32.059","level":"DEBUG","thread":"metadata-cache-dispatcher-48","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from EcsCredentialsProvider: Container credentials URI not set","context":"default"}
{"timestamp":"2025-08-05 20:30:32.059","level":"TRACE","thread":"metadata-cache-dispatcher-48","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"Attempting to resolve identity from ImdsCredentialsProvider","context":"default"}
{"timestamp":"2025-08-05 20:31:02.078","level":"DEBUG","thread":"metadata-cache-dispatcher-193","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from ImdsCredentialsProvider: Failed to load instance profile name","context":"default"}
{"timestamp":"2025-08-05 20:31:02.079","level":"DEBUG","thread":"metadata-cache-dispatcher-193","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.awsprotocol.ClockSkewInterceptor","message":"service did not return \"Date\" header, skipping skew calculation","context":"default"}
{"timestamp":"2025-08-05 20:31:02.079","level":"DEBUG","thread":"metadata-cache-dispatcher-193","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware","message":"request failed with non-retryable error","context":"default"}
{"timestamp":"2025-08-05 20:31:02.079","level":"TRACE","thread":"metadata-cache-dispatcher-193","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"86cc37be-9fac-4641-bede-62b38c35a7e3"},"logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation failed","context":"default","exception":"aws.smithy.kotlin.runtime.identity.IdentityProviderException: No identity could be resolved from the chain: SystemPropertyCredentialsProvider -> EnvironmentCredentialsProvider -> aws.sdk.kotlin.runtime.auth.credentials.StsWebIdentityProvider@72fc1632 -> ProfileCredentialsProvider -> EcsCredentialsProvider -> ImdsCredentialsProvider\n\tat aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve$2$chainException$1.invoke(IdentityProviderChain.kt:37)\n\tat aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve$2$chainException$1.invoke(IdentityProviderChain.kt:37)\n\tat kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:86)\n\tat aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve$suspendImpl$$inlined$withSpan$default$1.invokeSuspend(CoroutineContextTraceExt.kt:89)\n\tat aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve$suspendImpl$$inlined$withSpan$default$1.invoke(CoroutineContextTraceExt.kt)\n\tat aws.smithy.kotlin.runtime.identity.IdentityProviderChain$resolve$suspendImpl$$inlined$withSpan$default$1.invoke(CoroutineContextTraceExt.kt)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:66)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)\n\tat kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165)\n\tat kotlinx.coroutines.BuildersKt.withContext(Unknown Source)\n\tat aws.smithy.kotlin.runtime.identity.IdentityProviderChain.resolve$suspendImpl(IdentityProviderChain.kt:94)\n\tat aws.smithy.kotlin.runtime.identity.IdentityProviderChain.resolve(IdentityProviderChain.kt)\n\tat aws.smithy.kotlin.runtime.auth.awscredentials.CredentialsProviderChain.resolve(CredentialsProviderChain.kt:23)\n\tat aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider$resolve$3.invokeSuspend(CachedCredentialsProvider.kt:63)\n\tat aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider$resolve$3.invoke(CachedCredentialsProvider.kt)\n\tat aws.smithy.kotlin.runtime.auth.awscredentials.CachedCredentialsProvider$resolve$3.invoke(CachedCredentialsProvider.kt)\n\tat aws.smithy.kotlin.runtime.util.CachedValue.getOrLoad(CachedValue.kt:80)\n\tat aws.smithy.kotlin.runtime.util.CachedValue$getOrLoad$1.invokeSuspend(CachedValue.kt)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat st.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(Instrumente{"timestamp":"2025-08-05 20:31:32.091","level":"DEBUG","thread":"metadata-cache-dispatcher-196","mdc":{"rpc":"DynamoDB.DescribeTable","sdkInvocationId":"1a7a395f-13f1-4f85-81fc-ea11a080eead"},"logger":"aws.smithy.kotlin.runtime.identity.IdentityProviderChain","message":"unable to resolve identity from ImdsCredentialsProvider: Failed to load instance profile name","context":"default"}

Any help on this issue would be greatly appreciated :)

Regression Issue

• Select this option if this issue appears to be a regression.

Expected behavior

Here is what a successful connection log trace looks like:

{"timestamp":"2025-08-13 13:42:46.967","logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation started","context":"default"}
{"timestamp":"2025-08-13 13:42:46.968","logger":"aws.smithy.kotlin.runtime.http.operation.AuthHandler","message":"resolved endpoint: Endpoint(uri=https://dynamodb.us-east-2.amazonaws.com    , headers=null, attributes=aws.smithy.kotlin.runtime.collections.EmptyAttributes@2bff6cc)","context":"default"}
{"timestamp":"2025-08-13 13:42:46.969","logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"String to sign: AWS4-HMAC-SHA256 20250813T134246Z 20250813/us-east-2/dynamodb/aws4_request ***", "context":"default"}                                                                                                                                                                        {"timestamp":"2025-08-13 13:42:46.969","logger":"aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl","message":"Calculated signature: 274760ff31beebc627410b201fa5fba4633b6c67    5637cc0f8101bc365f9b47a8","context":"default"}
{"timestamp":"2025-08-13 13:42:47.014","logger":"aws.smithy.kotlin.runtime.http.engine.crt.CrtHttpEngine","message":"Acquired connection 132011462603216","context":"default"}
{"timestamp":"2025-08-13 13:42:47.280","logger":"aws.smithy.kotlin.runtime.http.operation.OperationHandler","message":"operation completed successfully","context":"default"}
{"timestamp":"2025-08-13 13:42:47.281","logger":"aws.smithy.kotlin.runtime.http.engine.crt.CrtHttpEngine","message":"completing handler; cause=null","context":"default"}
{"timestamp":"2025-08-13 13:42:47.281","logger":"aws.smithy.kotlin.runtime.http.engine.crt.SdkStreamResponseHandler","message":"Closing connection 132011462603216","context":"default"}
{"timestamp":"2025-08-13 13:42:47.281","level":"INFO","thread":"http-nio-8180-exec-9","mdc":{"loggingId":"c0c3103b-3da7-4352-a906-8cadb52013d0"},"logger":"st.oauth.health.DbHealthCheck",    "message":"db connection up","context":"default"}

Current behavior

See description.

Steps to Reproduce

Unfortunately, reproducing is pretty sporadic, but the easiest way we've found is by having a new EC2 instance standup and start our grails application.

What is interesting is that we can restart our service using systemctl restart and the application can then successfully connect to Dynamo.

Possible Solution

No response

Context

No response

AWS SDK for Kotlin version

AWS Kotlin SDK (version 1.4.120), AWS CRT HTTP engine (1.4.22)

Platform (JVM/JS/Native)

JVM

Operating system and version

Ubuntu 24.04.3

[lauzadis]

Hey @cavanhussein, thanks for the report. You mentioned EC2 so I'm assuming you're trying to use the ImdsCredentialsProvider from the chain.

In v1.4.65 we released changes to the ImdsCredentialsProvider to use a new API, but the project never manifested on EC2's backend side. The credentials provider is configured to fall back to the legacy API, and it's effectively doing this every time, since the new API does not exist. This fallback may be causing the errors you're seeing

We are going to be releasing a change tomorrow to revert our ImdsCredentialsProvider changes. They should have been reverted in v1.5.0 but the revert was unintentionally dropped. You have a few options to look at fixing your issue:

1. Wait until tomorrow's release, then update to v1.5.18
2. Downgrade to v1.4.64, before the changes to ImdsCredentialsProvider were merged
3. Configure credentials from a different source, such as environment variables, ~/.aws/config file, StsAssumeRoleCredentialsProvider, or any other provider we offer

[lauzadis]

The ImdsCredentialsProvider changes were released today. Could you try upgrading to v1.5.18 (or downgrading to v1.4.64) and let me know if it fixes your problem? Thanks!

[cavanhussein]

Hi @lauzadis,

Thank you so much for the quick reply!! I updated my services dependencies to use the v1.5.18 version of the Kotlin SDK. I will work on testing this next week, and will let you know how it goes!

Thanks again,
Cavan

[github-actions]

It looks like this issue has not been active for more than 5 days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[cavanhussein]

Hi @lauzadis ,

My team and I are deploying our service once again today to production to confirm this update solved our issue. I will let you all know our findings by EoD tomorrow :)