ImdsCredentialsProvider fails to retrieve credentials in AgentCode Runtime

[AlbertoSH]

Describe the bug

When running in AgentCore Runtime, the Kotlin SDK (JVM platform) is unable to get AWS credentials from the Default Credentials Chain. Based on a bit of research, I'd say it's a bug in the ImdsCredentialsProvider being unable to work with MicroVM Metadata Service. The equivalent code in the Java SDK (version 2.39.5) works fine

Regression Issue

• Select this option if this issue appears to be a regression.

Expected behavior

AgentCore Runtime uses MicroVM Metadata Service (MMDS) instead of IMDSv2 (https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/security-credentials-management.html), but since the Java SDK works with the IMDS provider, the Kotlin one should behave accordingly

Current behavior

When using the ImdsCredentialsProvider() for getting AWS credentials, it fails to retrieve the IMDS token:

  | 2025-11-29T12:07:04.580Z | aws.smithy.kotlin.runtime.auth.awscredentials.CredentialsProviderException: failed to load instance profile
  | 2025-11-29T12:07:04.580Z | at aws.sdk.kotlin.runtime.auth.credentials.ImdsCredentialsProvider.resolve(ImdsCredentialsProvider.kt:81)
  | 2025-11-29T12:07:04.580Z | at aws.sdk.kotlin.runtime.auth.credentials.ImdsCredentialsProvider$resolve$1.invokeSuspend(ImdsCredentialsProvider.kt)
  | 2025-11-29T12:07:04.581Z | at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
  | 2025-11-29T12:07:04.581Z | at kotlinx.coroutines.UndispatchedCoroutine.afterResume(CoroutineContext.kt:266)
  | 2025-11-29T12:07:04.581Z | at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:101)
...
  | 2025-11-29T12:07:04.583Z | Caused by: aws.sdk.kotlin.runtime.config.imds.EC2MetadataError: Failed to retrieve IMDS token
  | 2025-11-29T12:07:04.583Z | at aws.sdk.kotlin.runtime.config.imds.TokenMiddleware.getToken(TokenMiddleware.kt:80)
  | 2025-11-29T12:07:04.583Z | at aws.sdk.kotlin.runtime.config.imds.TokenMiddleware.access$getToken(TokenMiddleware.kt:32)
  | 2025-11-29T12:07:04.583Z | at aws.sdk.kotlin.runtime.config.imds.TokenMiddleware$getToken$1.invokeSuspend(TokenMiddleware.kt)
  | 2025-11-29T12:07:04.583Z | at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
...

Steps to Reproduce

• Generate a Docker image that runs a JVM application with the Kotlin SDK for AWS. Push it to ECR
• The following code can be used to trigger the error. It can be directly put in the main method

    println("Kotlin SDK failing")
    runBlocking {
        try {
            val kotlinClient = BedrockRuntimeClient {
                credentialsProvider = ImdsCredentialsProvider()
            }
            val message = Message {
                content = listOf(ContentBlock.Text("Just say hi"))
                role = ConversationRole.User
            }
            val response = kotlinClient.converse {
                modelId = "eu.amazon.nova-micro-v1:0"
                messages = listOf(message)
            }
            response.output?.asMessage()?.content?.forEach { println(it) } ?: println("No output message")
            kotlinClient.close()
        } catch (e: Exception) {
            e.printStackTrace()
        }
    }

You can use the following code for checking the Java SDK (beware of imports, you may want to put them in different files or add fully qualified imports)

    println("Java SDK working")
    try {
        val javaClient = BedrockRuntimeClient.builder()
            .credentialsProvider(InstanceProfileCredentialsProvider.create())
            .build()
        val message = Message.builder()
            .content(ContentBlock.fromText("Just say hi"))
            .role(ConversationRole.USER)
            .build()
        val request = ConverseRequest.builder()
            .modelId("eu.amazon.nova-micro-v1:0")
            .messages(message)
            .build()
        val result = javaClient.converse(request)
        result.output().message().content().forEach {
            println(it.text())
        }
        javaClient.close()
    } catch (e: Exception) {
        e.printStackTrace()
    }

• Create an AgentCore Runtime agent. Indicate the previously pushed image as the agent source. Specify the authentication to be JWT based. Steps for creating a user and getting its token are described in https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-oauth.html#setup-cognito
• Invoke the agent with cURL and the user token. Steps described in https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-oauth.html#invoke-agent
• Check CloudWatch logs

Possible Solution

No response

Context

Running agents built with Kotlin in AgentCore Runtime

AWS SDK for Kotlin version

1.5.92

Platform (JVM/JS/Native)

JVM

Operating system and version

AgentCore Runtime

[ianbotsf]

Hi @AlbertoSH. Unfortunately, the AWS SDK for Kotlin supports only IMDS v2 and we have no plans currently to add support for legacy IMDS. Separately, I'm communicating with the Bedrock AgentCore Runtime team to determine if IMDS v2 support can be added, which would eliminate the incompatibility with the SDK. I'll post an update on this issue when I have one.

Unless/until AgentCore Runtime supports IMDS v2, your best options to proceed are to either use the Java SDK or write your own credentials provider that works with IMDS v1. Example code:

class ImdsV1CredentialsProvider(private val profileName: String? = null) : CloseableCredentialsProvider {
    private val httpClient = OkHttpClient()
    private val json = Json { ignoreUnknownKeys = true }

    override suspend fun resolve(attributes: Attributes): Credentials {
        val profileName = this.profileName ?: imdsGet("")
        val payload = imdsGet(profileName)
        val response = json.decodeFromString<ImdsCredentialsResponse>(payload)

        return Credentials(
            accessKeyId = response.accessKeyId,
            secretAccessKey = response.secretAccessKey,
            sessionToken = response.sessionToken,
            expiration = Instant.fromIso8601(response.expiration),
        )
    }

    private suspend fun imdsGet(path: String): String {
        // Wrap OkHttp callback into a coroutine
        val response = suspendCoroutine { continuation ->
            val responseCallback = object : Callback {
                override fun onFailure(call: Call, e: okio.IOException) = continuation.resumeWithException(e)
                override fun onResponse(call: Call, response: Response) = continuation.resume(response)
            }

            val request = Request.Builder()
                .url("http://169.254.169.254/latest/meta-data/iam/security-credentials/$path")
                .build()

            httpClient.newCall(request).enqueue(responseCallback)
        }

        return response
            .takeIf { it.isSuccessful }
            ?.body
            ?.string()
            ?.also { println("IMDS v1 response: $it") }
            ?: error("IMDS v1 call returned status code ${response.code}")
    }

    override fun close() {
        // Clean up HTTP client resources
        httpClient.connectionPool.evictAll()
        httpClient.dispatcher.executorService.shutdown()
    }
}

@Serializable
private data class ImdsCredentialsResponse(
    @SerialName("AccessKeyId") val accessKeyId: String,
    @SerialName("SecretAccessKey") val secretAccessKey: String,
    @SerialName("Token") val sessionToken: String,
    @SerialName("Expiration") val expiration: String,
)

Then you can:

val imdsV1CredentialsProvider = ImdsV1CredentialsProvider().cached() // use default caching strategy
val kotlinClient = BedrockRuntimeClient {
    credentialsProvider = imdsV1CredentialsProvider
}

// ...use kotlinClient and then close to avoid resource leaks:

kotlinClient.close()
imdsV1CredentialsProvider.close()

Are either of those workarounds suitable for your use case?

[AlbertoSH]

Hey @ianbotsf, thanks for the early response! It totally makes sense not to support IMDSv1 in a new SDK as it's deprecated. Going that direction you ensure the problem doesn't get bigger :)

My previous workaround was, well, to invoke the Java SDK 🙃

class CredentialsProviderFix(
    private val javaSDK: InstanceProfileCredentialsProvider = InstanceProfileCredentialsProvider.create(),
) : CredentialsProvider {

    override suspend fun resolve(attributes: Attributes): Credentials {
            val javaSDKCreds = withContext(Dispatchers.IO) {
                javaSDK.resolveCredentials() as AwsSessionCredentials
            }
            return mapToKotlinSdkCredentials(javaSDKCreds)
    }

}

It works, but requires the auth module from the java SDK, which slightly increases the final jar size.

Now, checking your approach, and doing some tests on my own, I can say that I feel a bit stupid for not having tried to implement it on my own 😅

I don't have the exact details on how IMDSv1 works, nor MicroVM Metadata Service, but taking your code as the starting point I discovered that a GET request to http://169.254.169.254/latest/meta-data/iam/security-credentials/ gives the following output:

{
    "execution_role": "{\"Code\":\"Success\",\"LastUpdated\":\"2025-12-04T13:49:25.642263173Z\",\"Type\":\"AWS-HMAC\",\"AccessKeyId\":\"ASIA...\",\"SecretAccessKey\":\"ko5...\",\"Token\":\"IQ...\",\"Expiration\":\"2025-12-04T14:44:10Z\"}"
}

So adjusting a bit the code, I got to the final solution (using Ktor as the HTTP client as I already have it for other stuff):

class MMDSCredentialsProvider() : CloseableCredentialsProvider {

    private val json = Json { ignoreUnknownKeys = true }
    private val httpClient = HttpClient {

        install(ContentNegotiation) {
            json(json)
        }

        defaultRequest {
            url {
                protocol = URLProtocol.HTTP
                host = "169.254.169.254"
                path("/latest/meta-data/iam/security-credentials/")
            }
        }
    }

    override suspend fun resolve(attributes: Attributes): Credentials {
        if (System.getenv("AWS_EXECUTION_ENV") == "AWS_BedrockAgentCore_Runtime") {

            val credentials = getCredentials()

            return Credentials(
                accessKeyId = credentials.accessKeyId,
                secretAccessKey = credentials.secretAccessKey,
                sessionToken = credentials.sessionToken,
                expiration = Instant.fromIso8601(credentials.expiration),
            )
        } else {
            error("MMDSCredentialsProvider can only be used when running in AgentCore Runtime")
        }
    }

    private suspend fun getCredentials(): MMDSCredentialsResponse {
        val response = httpClient.get {}

        return if (response.status.isSuccess()) {
            val asJson = response.body<JsonObject>()

            val executionRole = asJson["execution_role"]
                ?.jsonPrimitive
                ?.content
                ?: error("MMDS response does not contain execution_role field")

            json.decodeFromString<MMDSCredentialsResponse>(executionRole)
        } else {
            error("MMDS call returned status code ${response.status.value}")
        }
    }

    override fun close() {
        httpClient.close()
    }
}

@Serializable
private data class MMDSCredentialsResponse(
    @SerialName("AccessKeyId") val accessKeyId: String,
    @SerialName("SecretAccessKey") val secretAccessKey: String,
    @SerialName("Token") val sessionToken: String,
    @SerialName("Expiration") val expiration: String,
)

There are some minor things here and there but, as a summary, I have a perfectly valid workaround for my needs. Thanks! 😄

Now, the question that comes to my mind is: should the Kotlin SDK support MMDS for running agents in AgentCore Runtime?
If you ask me, I say yes. If AgentCore Runtime, which is quite a new service, is intended to use MicroVM Metadata Service for recovering AWS credentials at runtime, the SDK should support it.

By adding the environment variable check System.getenv("AWS_EXECUTION_ENV") == "AWS_BedrockAgentCore_Runtime", the credentials provider should fail if invoked in any other scenario. So it should not be a problem. But I'm just a user, I'll leave the final decision to you 🙂

[ianbotsf]

Nice, glad you were able to adjust the example to work for MMDS.

On the topic of whether the SDK should support MMDS, that's certainly a possibility. We'll first see what the service team says about simply upgrading to an IMDS v2-compatible API. That would avoid using deprecated IMDS versions and avoid code changes in the newer SDKs. 😁

[AlbertoSH]

Yep, I agree
I assume the service team had their reasons for building their own metadata service, but that's something that will need some internal alignment.

In any case, that's something that can be worked in the following weeks/months. For the framework I'm using, there are some conversations about providing a custom credentials provider for solving the issue in the meantime. So there will be workarounds for the short term

Thanks again for your support! 😄

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.