Unable to assume ec2 IAM role through local kubernetes

[jpmcb]

Describe the bug

I'm working on kubernetes components that interact with AWS services through ec2 instances that have assumed IAM roles.

I am unable to get aws-sdk-rust calls to resolve correctly due to missing credential errors.

Similar issues:
#606
#425

---

Proof of concept rust code:

// [dependencies]
// aws-config = "0.48.0"
// aws-sdk-sts = "0.18.0"
// tokio = { version = "1.20.1", features = ["full"] }

#[tokio::main]
async fn main() {
    let conf = aws_config::from_env().region("us-west-2").load().await;

    let client = aws_sdk_sts::Client::new(&conf);
    let res = client.get_caller_identity().send().await;
    println!("caller_identity = {:#?}", res)
}

When the above program is run on my local hosts, I get the expected results. All seems good here!

$ cargo run
   Compiling sts-poc v0.1.0 (/home/fedora/workspace/ssm-poc/ssm-poc)
    Finished dev [unoptimized + debuginfo] target(s) in 4.91s
     Running `target/debug/sts-poc`
caller_identity = Ok(
    GetCallerIdentityOutput {
        user_id: Some(
            <redacted>,
        ),
        account: Some(
            <redacted>,
        ),
        arn: Some(
            "arn:aws:sts::<redacted>:assumed-role/<redacted>",
        ),
    },
)

Expected Behavior

When my host with the assumed role runs a container, I'd expect aws-sdk-rust to be able to also assume that host's role.

Current Behavior

However, when run through a local kubernetes cluster (kind), I hit the ConstructionFailure(MissingCredentials) error:

Reproduction Steps

Along with the above rust proof of concept program, use the following dockerfile and pod yaml:

---

Dockerfile:

FROM rust
WORKDIR sts-poc
COPY ./ /sts-poc
RUN cargo build --release
CMD ["./target/release/sts-poc"]

Kubernetes pod yaml named run-aws-sdk.yaml

apiVersion: v1
kind: Pod
metadata:
  name: sts-poc
  namespace: default
spec:
  containers:
  - image: sts-poc:demo
    name: sts-proof-of-concept

---

1. Create a kind cluster

$ kind create cluster --name demo

2. Create the container image using docker. Tag it as sts-poc:demo

$ docker build . -t sts-poc:demo

3. Load the image into the kind cluster:

$ kind load docker-image --name demo sts-poc:demo

4. Apply the pod yaml:

$ kubectl apply -f run-aws-poc.yaml

5. Check the logs from the pod. Notice the error.

$ kubectl logs sts-poc
caller_identity = Err(
    ConstructionFailure(
        MissingCredentials,
    ),
)

Possible Solution

Are there things in my environment I'm missing? Other issues on this topic made it seem like this should "just work" ™️

Version

$ cargo tree | rg aws
├── aws-config v0.48.0
│   ├── aws-http v0.48.0
│   │   ├── aws-smithy-http v0.48.0
│   │   │   ├── aws-smithy-types v0.48.0
│   │   ├── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-types v0.48.0
│   │   │   ├── aws-smithy-async v0.48.0
│   │   │   ├── aws-smithy-client v0.48.0
│   │   │   │   ├── aws-smithy-async v0.48.0 (*)
│   │   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.48.0
│   │   │   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   │   ├── aws-smithy-types v0.48.0 (*)
│   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   ├── aws-smithy-types v0.48.0 (*)
│   ├── aws-sdk-sso v0.18.0
│   │   ├── aws-endpoint v0.48.0
│   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   ├── aws-smithy-types v0.48.0 (*)
│   │   │   ├── aws-types v0.48.0 (*)
│   │   ├── aws-http v0.48.0 (*)
│   │   ├── aws-sig-auth v0.48.0
│   │   │   ├── aws-sigv4 v0.48.0
│   │   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   ├── aws-smithy-http v0.48.0 (*)
│   │   │   ├── aws-types v0.48.0 (*)
│   │   ├── aws-smithy-async v0.48.0 (*)
│   │   ├── aws-smithy-client v0.48.0 (*)
│   │   ├── aws-smithy-http v0.48.0 (*)
│   │   ├── aws-smithy-http-tower v0.48.0 (*)
│   │   ├── aws-smithy-json v0.48.0
│   │   │   └── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-types v0.48.0 (*)
│   ├── aws-sdk-sts v0.18.0
│   │   ├── aws-endpoint v0.48.0 (*)
│   │   ├── aws-http v0.48.0 (*)
│   │   ├── aws-sig-auth v0.48.0 (*)
│   │   ├── aws-smithy-async v0.48.0 (*)
│   │   ├── aws-smithy-client v0.48.0 (*)
│   │   ├── aws-smithy-http v0.48.0 (*)
│   │   ├── aws-smithy-http-tower v0.48.0 (*)
│   │   ├── aws-smithy-query v0.48.0
│   │   │   ├── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-smithy-types v0.48.0 (*)
│   │   ├── aws-smithy-xml v0.48.0
│   │   ├── aws-types v0.48.0 (*)
│   ├── aws-smithy-async v0.48.0 (*)
│   ├── aws-smithy-client v0.48.0 (*)
│   ├── aws-smithy-http v0.48.0 (*)
│   ├── aws-smithy-http-tower v0.48.0 (*)
│   ├── aws-smithy-json v0.48.0 (*)
│   ├── aws-smithy-types v0.48.0 (*)
│   ├── aws-types v0.48.0 (*)
├── aws-sdk-sts v0.18.0 (*)

Environment details (OS name and version, etc.)

Fedora 34, kernel 5.11.12, aarch64

Docker build logs

$ docker build . -t sts-poc:demo
Sending build context to Docker daemon 1.329GB
Step 1/5 : FROM rust
---> ab88ceb21a7d
Step 2/5 : WORKDIR sts-poc
---> Using cache
---> ca4a7ec4d0dc
Step 3/5 : COPY ./ /sts-poc
---> d6b97d1263e8
Step 4/5 : RUN cargo build --release
---> Running in c8f64db3cb91
Updating crates.io index
Downloading crates ...
Downloaded base64 v0.13.0
Downloaded libc v0.2.132
Downloaded aws-types v0.48.0
Downloaded socket2 v0.4.7
Downloaded quote v1.0.21
Downloaded rustc_version v0.4.0
Downloaded spin v0.5.2
Downloaded want v0.3.0
Downloaded fnv v1.0.7
Downloaded aws-smithy-query v0.48.0
Downloaded aws-smithy-json v0.48.0
Downloaded aws-smithy-async v0.48.0
Downloaded bytes-utils v0.1.3
Downloaded aws-smithy-http-tower v0.48.0
Downloaded tokio-util v0.7.3
Downloaded futures-core v0.3.24
Downloaded aws-sig-auth v0.48.0
Downloaded itoa v1.0.3
Downloaded pin-utils v0.1.0
Downloaded pin-project-lite v0.2.9
Downloaded http v0.2.8
Downloaded tower-layer v0.3.1
Downloaded aws-sigv4 v0.48.0
Downloaded tower v0.4.13
Downloaded matches v0.1.9
Downloaded aws-smithy-xml v0.48.0
Downloaded aws-smithy-types v0.48.0
Downloaded aws-sdk-sts v0.18.0
Downloaded http-body v0.4.5
Downloaded urlencoding v2.1.2
Downloaded tower-service v0.3.2
Downloaded tokio-stream v0.1.9
Downloaded pin-project v1.0.12
Downloaded tokio-macros v1.8.0
Downloaded ryu v1.0.11
Downloaded httparse v1.8.0
Downloaded tracing v0.1.36
Downloaded hyper v0.14.20
Downloaded futures-util v0.3.24
Downloaded tokio v1.21.0
Downloaded aws-config v0.48.0
Downloaded futures-channel v0.3.24
Downloaded h2 v0.3.14
Downloaded webpki v0.21.4
Downloaded aws-smithy-client v0.48.0
Downloaded zeroize v1.5.7
Downloaded xmlparser v0.13.3
Downloaded ring v0.16.20
Downloaded regex-syntax v0.6.27
Downloaded mio v0.8.4
Downloaded lock_api v0.4.8
Downloaded ct-logs v0.8.0
Downloaded tokio-rustls v0.22.0
Downloaded semver v1.0.13
Downloaded rustls-native-certs v0.5.0
Downloaded hyper-rustls v0.22.1
Downloaded futures-macro v0.3.24
Downloaded either v1.8.0
Downloaded aws-http v0.48.0
Downloaded aws-smithy-http v0.48.0
Downloaded hex v0.4.3
Downloaded regex v1.6.0
Downloaded aho-corasick v0.7.19
Downloaded aws-endpoint v0.48.0
Downloaded fastrand v1.8.0
Downloaded autocfg v1.1.0
Downloaded untrusted v0.7.1
Downloaded unicode-ident v1.0.3
Downloaded try-lock v0.2.3
Downloaded tracing-core v0.1.29
Downloaded tracing-attributes v0.1.22
Downloaded time v0.3.14
Downloaded syn v1.0.99
Downloaded smallvec v1.9.0
Downloaded slab v0.4.7
Downloaded signal-hook-registry v1.4.0
Downloaded sct v0.6.1
Downloaded scopeguard v1.1.0
Downloaded rustls v0.19.1
Downloaded percent-encoding v2.1.0
Downloaded parking_lot_core v0.9.3
Downloaded parking_lot v0.12.1
Downloaded openssl-probe v0.1.5
Downloaded once_cell v1.14.0
Downloaded num_threads v0.1.6
Downloaded num_cpus v1.13.1
Downloaded num-integer v0.1.45
Downloaded memchr v2.5.0
Downloaded log v0.4.17
Downloaded lazy_static v1.4.0
Downloaded indexmap v1.9.1
Downloaded httpdate v1.0.2
Downloaded hashbrown v0.12.3
Downloaded futures-task v0.3.24
Downloaded futures-sink v0.3.24
Downloaded form_urlencoded v1.0.1
Downloaded cfg-if v1.0.0
Downloaded cc v1.0.73
Downloaded bytes v1.2.1
Downloaded proc-macro2 v1.0.43
Downloaded pin-project-internal v1.0.12
Downloaded num-traits v0.2.15
Downloaded aws-sdk-sso v0.18.0
Compiling autocfg v1.1.0
Compiling libc v0.2.132
Compiling proc-macro2 v1.0.43
Compiling unicode-ident v1.0.3
Compiling quote v1.0.21
Compiling syn v1.0.99
Compiling cfg-if v1.0.0
Compiling once_cell v1.14.0
Compiling log v0.4.17
Compiling memchr v2.5.0
Compiling parking_lot_core v0.9.3
Compiling bytes v1.2.1
Compiling pin-project-lite v0.2.9
Compiling scopeguard v1.1.0
Compiling futures-core v0.3.24
Compiling smallvec v1.9.0
Compiling futures-task v0.3.24
Compiling cc v1.0.73
Compiling itoa v1.0.3
Compiling futures-util v0.3.24
Compiling untrusted v0.7.1
Compiling pin-utils v0.1.0
Compiling spin v0.5.2
Compiling fnv v1.0.7
Compiling num_threads v0.1.6
Compiling hashbrown v0.12.3
Compiling futures-channel v0.3.24
Compiling httparse v1.8.0
Compiling futures-sink v0.3.24
Compiling try-lock v0.2.3
Compiling ryu v1.0.11
Compiling tower-service v0.3.2
Compiling httpdate v1.0.2
Compiling base64 v0.13.0
Compiling either v1.8.0
Compiling semver v1.0.13
Compiling percent-encoding v2.1.0
Compiling tower-layer v0.3.1
Compiling openssl-probe v0.1.5
Compiling lazy_static v1.4.0
Compiling fastrand v1.8.0
Compiling regex-syntax v0.6.27
Compiling zeroize v1.5.7
Compiling matches v0.1.9
Compiling hex v0.4.3
Compiling xmlparser v0.13.3
Compiling urlencoding v2.1.2
Compiling tracing-core v0.1.29
Compiling form_urlencoded v1.0.1
Compiling http v0.2.8
Compiling bytes-utils v0.1.3
Compiling lock_api v0.4.8
Compiling tokio v1.21.0
Compiling slab v0.4.7
Compiling num-traits v0.2.15
Compiling indexmap v1.9.1
Compiling num-integer v0.1.45
Compiling aws-smithy-xml v0.48.0
Compiling want v0.3.0
Compiling rustc_version v0.4.0
Compiling aho-corasick v0.7.19
Compiling ring v0.16.20
Compiling aws-types v0.48.0
Compiling socket2 v0.4.7
Compiling mio v0.8.4
Compiling num_cpus v1.13.1
Compiling signal-hook-registry v1.4.0
Compiling time v0.3.14
Compiling http-body v0.4.5
Compiling parking_lot v0.12.1
Compiling regex v1.6.0
Compiling aws-smithy-types v0.48.0
Compiling aws-smithy-query v0.48.0
Compiling aws-smithy-json v0.48.0
Compiling sct v0.6.1
Compiling webpki v0.21.4
Compiling ct-logs v0.8.0
Compiling rustls v0.19.1
Compiling tokio-macros v1.8.0
Compiling tracing-attributes v0.1.22
Compiling futures-macro v0.3.24
Compiling pin-project-internal v1.0.12
Compiling rustls-native-certs v0.5.0
Compiling pin-project v1.0.12
Compiling tracing v0.1.36
Compiling tokio-util v0.7.3
Compiling tokio-stream v0.1.9
Compiling tower v0.4.13
Compiling tokio-rustls v0.22.0
Compiling aws-smithy-async v0.48.0
Compiling h2 v0.3.14
Compiling hyper v0.14.20
Compiling aws-smithy-http v0.48.0
Compiling hyper-rustls v0.22.1
Compiling aws-smithy-http-tower v0.48.0
Compiling aws-sigv4 v0.48.0
Compiling aws-smithy-client v0.48.0
Compiling aws-endpoint v0.48.0
Compiling aws-sig-auth v0.48.0
Compiling aws-http v0.48.0
Compiling aws-sdk-sts v0.18.0
Compiling aws-sdk-sso v0.18.0
Compiling aws-config v0.48.0
Compiling sts-poc v0.1.0 (/sts-poc)
Finished release [optimized] target(s) in 1m 20s
Removing intermediate container c8f64db3cb91
---> 7d8956f9af13
Step 5/5 : CMD ["./target/release/sts-poc"]
---> Running in 1d2c777399f4
Removing intermediate container 1d2c777399f4
---> c5eb809459cf
Successfully built c5eb809459cf
Successfully tagged sts-poc:demo

[jpmcb]

As another data point, this works as expected when using the aws CLI container:

Container yaml:

apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  containers:
  - image: amazon/aws-cli
    command:
      - "aws"
      - "sts"
      - "get-caller-identity"
    name: aws-cli

---

Steps to reproduce

1. Create new kind cluster

$ kind create cluster --name demo

2. Apply the above yaml

$ kubectl apply -f aws-cli.yaml

3. Check logs

$ kubect logs pods/aws-cli
{
    "UserId": "<redacted>",
    "Account": "<redacted>",
    "Arn": "arn:aws:sts::<redacted>:assumed-role/<redacted>"
}

[jmklix]

Can you include detailed logs from when you hit the ConstructionFailure(MissingCredentials) error?

[jpmcb]

Can you include detailed logs from when you hit the ConstructionFailure(MissingCredentials) error?

The only logs that are returned by the pod are:

caller_identity = Err(
    ConstructionFailure(
        MissingCredentials,
    ),
)

I've updated the text of the top level post to be more clear as well as added docker build logs

[mlevkov-ias]

@jpmcb were you able to resolve this at all? I'm having the same issue. From what I understand, the SDK uses IMDSv2 for the EC2 role, not IMDSv1.

[jpmcb]

@mlevkov-ias - I was not able to resolve this: I'm still seeing the same thing.

[jpmcb]

This doesn't appear to be related to IMDSv2 or the hop limit. This is what my instance looks like:

{
    "InstanceId": "i-<redacted>",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "required",
        "HttpPutResponseHopLimit": 2,
        "HttpEndpoint": "enabled",
        "HttpProtocolIpv6": "disabled",
        "InstanceMetadataTags": "disabled"
    }
}

I get the same results:

❯ k logs pods/sts-poc
caller_identity = Err(
    ConstructionFailure(
        MissingCredentials,
    ),
)

[gbhand]

I've observed this same issue with plain Docker containers on an EC2 instance using the same proof-of-concept Rust code and Dockerfile as @jpmcb:

$ docker run --rm -it sts-poc:demo
caller_identity = Err(
    DispatchFailure(
        DispatchFailure {
            source: ConnectorError {
                kind: Other(
                    None,
                ),
                source: CredentialsNotLoaded(
                    CredentialsNotLoaded {
                        source: "no providers in chain provided credentials",
                    },
                ),
                connection: Unknown,
            },
        },
    ),
)

However I've found a simple workaround by setting the Docker network mode to host:

$ docker run --rm -it --network=host sts-poc:demo
caller_identity = Ok(
    GetCallerIdentityOutput {
        user_id: Some(
            <redacted>,
        ),
        account: Some(
            <redacted>,
        ),
        arn: Some(
            "arn:aws:sts::<redacted>:assumed-role/<redacted>",
        ),
        _request_id: Some(
            <redacted>,
        ),
    },
)

The AWS CLI appears to retrieve the credentials correctly when run from within the container even without host networking enabled. My hunch is some faulty fetching of the EC2 instance metadata for nonstandard network interfaces on the SDK side.

[rcoh]

if you did this by setting the docker network mode to host, then this sounds like the hop-limit issue—you've removed a hop by setting the docker network mode to host.

[jmklix]

Can you trying increasing the hop limit to 3: "HttpPutResponseHopLimit": 3
And see if that also fixed the problem?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[gbhand]

For anyone finding this thread later I can confirm that increasing HttpPutResponseHopLimit resolves the issue. In my use case (plain Docker) increasing the value to 2 allows the role to be assumed. I would assume increasing the value to 3 would resolve the issue in Kubernetes.

Tested on both the initial issue's version

[dependencies]
aws-config = "0.48.0"
aws-sdk-sts = "0.18.0"
tokio = { version = "1.20.1", features = ["full"] }

and the latest AWS SDK release

[dependencies]
aws-config = "1.1.7"
aws-sdk-sts = "1.15.0"
tokio = { version = "1.20.1", features = ["full"] }

also tested in the latest release using the new aws_config::defaults(); config loader with the same result.

See AWS API docs for guidance to increase the hop limit.