Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConstructionFailure(MissingCredentials) when using IAM Roles attached to an instance #606

Closed
albx79 opened this issue Aug 17, 2022 · 13 comments
Assignees
Labels
bug This issue is a bug.

Comments

@albx79
Copy link

albx79 commented Aug 17, 2022

Describe the bug

We are running an SQS client as a Kubernetes pod on EC2.

When doing any SQS request, we get the following error:

Error: ConstructionFailure(MissingCredentials)

Expected Behavior

According to the documentation in https://docs.aws.amazon.com/sdk-for-rust/latest/dg/credentials.html, the request should work because "IAM Roles attached to an instance" is supported.

Current Behavior

We tested the EC2 policy using the AWS command line, and we were able to query the SQS queue without API keys.

We also have a number of services that access S3, and they also work correctly without API keys.

Reproduction Steps

I'm just building the client

        let mut conf = aws_config::load_from_env().await;
        let mut builder = aws_sdk_sqs::config::Builder::from(&conf);
        let client = aws_sdk_sqs::Client::from_conf(builder.build());

and sending the request

let get_url_resp = client.get_queue_url().queue_name("foo_queue").send().await?;

Possible Solution

No response

Additional Information/Context

No response

Version

├── aws-config v0.47.0
│   ├── aws-http v0.47.0
│   │   ├── aws-smithy-http v0.47.0
│   │   │   ├── aws-smithy-eventstream v0.47.0
│   │   │   │   ├── aws-smithy-types v0.47.0
│   │   │   ├── aws-smithy-types v0.47.0 (*)
│   │   ├── aws-smithy-types v0.47.0 (*)
│   │   ├── aws-types v0.47.0
│   │   │   ├── aws-smithy-async v0.47.0
│   │   │   ├── aws-smithy-client v0.47.0
│   │   │   │   ├── aws-smithy-async v0.47.0 (*)
│   │   │   │   ├── aws-smithy-http v0.47.0 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.47.0
│   │   │   │   │   ├── aws-smithy-http v0.47.0 (*)
│   │   │   │   ├── aws-smithy-types v0.47.0 (*)
│   │   │   ├── aws-smithy-http v0.47.0 (*)
│   │   │   ├── aws-smithy-types v0.47.0 (*)
│   ├── aws-sdk-sso v0.17.0
│   │   ├── aws-endpoint v0.47.0
│   │   │   ├── aws-smithy-http v0.47.0 (*)
│   │   │   ├── aws-types v0.47.0 (*)
│   │   ├── aws-http v0.47.0 (*)
│   │   ├── aws-sig-auth v0.47.0
│   │   │   ├── aws-sigv4 v0.47.0
│   │   │   │   ├── aws-smithy-eventstream v0.47.0 (*)
│   │   │   │   ├── aws-smithy-http v0.47.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.47.0 (*)
│   │   │   ├── aws-smithy-http v0.47.0 (*)
│   │   │   ├── aws-types v0.47.0 (*)
│   │   ├── aws-smithy-async v0.47.0 (*)
│   │   ├── aws-smithy-client v0.47.0 (*)
│   │   ├── aws-smithy-http v0.47.0 (*)
│   │   ├── aws-smithy-http-tower v0.47.0 (*)
│   │   ├── aws-smithy-json v0.47.0
│   │   │   └── aws-smithy-types v0.47.0 (*)
│   │   ├── aws-smithy-types v0.47.0 (*)
│   │   ├── aws-types v0.47.0 (*)
│   ├── aws-sdk-sts v0.17.0
│   │   ├── aws-endpoint v0.47.0 (*)
│   │   ├── aws-http v0.47.0 (*)
│   │   ├── aws-sig-auth v0.47.0 (*)
│   │   ├── aws-smithy-async v0.47.0 (*)
│   │   ├── aws-smithy-client v0.47.0 (*)
│   │   ├── aws-smithy-http v0.47.0 (*)
│   │   ├── aws-smithy-http-tower v0.47.0 (*)
│   │   ├── aws-smithy-query v0.47.0
│   │   │   ├── aws-smithy-types v0.47.0 (*)
│   │   ├── aws-smithy-types v0.47.0 (*)
│   │   ├── aws-smithy-xml v0.47.0
│   │   ├── aws-types v0.47.0 (*)
│   ├── aws-smithy-async v0.47.0 (*)
│   ├── aws-smithy-client v0.47.0 (*)
│   ├── aws-smithy-http v0.47.0 (*)
│   ├── aws-smithy-http-tower v0.47.0 (*)
│   ├── aws-smithy-json v0.47.0 (*)
│   ├── aws-smithy-types v0.47.0 (*)
│   ├── aws-types v0.47.0 (*)
├── aws-sdk-s3 v0.17.0
│   ├── aws-endpoint v0.47.0 (*)
│   ├── aws-http v0.47.0 (*)
│   ├── aws-sig-auth v0.47.0 (*)
│   ├── aws-sigv4 v0.47.0 (*)
│   ├── aws-smithy-async v0.47.0 (*)
│   ├── aws-smithy-checksums v0.47.0
│   │   ├── aws-smithy-http v0.47.0 (*)
│   │   ├── aws-smithy-types v0.47.0 (*)
│   ├── aws-smithy-client v0.47.0 (*)
│   ├── aws-smithy-eventstream v0.47.0 (*)
│   ├── aws-smithy-http v0.47.0 (*)
│   ├── aws-smithy-http-tower v0.47.0 (*)
│   ├── aws-smithy-types v0.47.0 (*)
│   ├── aws-smithy-xml v0.47.0 (*)
│   ├── aws-types v0.47.0 (*)
├── aws-sdk-sqs v0.17.0
│   ├── aws-endpoint v0.47.0 (*)
│   ├── aws-http v0.47.0 (*)
│   ├── aws-sig-auth v0.47.0 (*)
│   ├── aws-smithy-async v0.47.0 (*)
│   ├── aws-smithy-client v0.47.0 (*)
│   ├── aws-smithy-http v0.47.0 (*)
│   ├── aws-smithy-http-tower v0.47.0 (*)
│   ├── aws-smithy-query v0.47.0 (*)
│   ├── aws-smithy-types v0.47.0 (*)
│   ├── aws-smithy-xml v0.47.0 (*)
│   ├── aws-types v0.47.0 (*)
├── aws-smithy-http v0.47.0 (*)
├── aws-types v0.47.0 (*)
├── aws_lambda_events v0.6.3


### Environment details (OS name and version, etc.)

Debian instance on EC2

### Logs

_No response_
@albx79 albx79 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 17, 2022
@Velfi Velfi removed the needs-triage This issue or PR still needs to be triaged. label Aug 17, 2022
@Velfi
Copy link
Contributor

Velfi commented Aug 17, 2022

I believe this issue is the same one as #425
We don't yet support "anonymous" (from the SDK's point of view at least) requests. We posted a workaround in that past issue thread, here's an updated version for your use case. Would you mind trying it out to see if it works for you?

The required dependencies:

[dependencies]
aws-sdk-sqs = "0.17.0"
aws-smithy-client = "0.47.0"
aws-sig-auth = "0.47.0"
aws-smithy-http = "0.47.0"
aws-config = "0.47.0"
tokio = { version = "1.20.1", features = ["full"] }

The code:

use aws_sdk_sqs::operation::GetQueueUrl;
use aws_sdk_sqs::Credentials;
use aws_sig_auth::signer::OperationSigningConfig;
use aws_sig_auth::signer::SigningRequirements;
use aws_smithy_http::operation::Operation;

#[tokio::main]
async fn main() {
    let conf = aws_config::from_env()
        .credentials_provider(Credentials::new("stub", "stub", None, None, "faked"))
        .region("us-east-1")
        .load()
        .await;
    let conf = aws_sdk_sqs::Config::new(&conf);
    let client = aws_smithy_client::Builder::dyn_https()
        .middleware(aws_sdk_sqs::middleware::DefaultMiddleware::new())
        .build();

    let mut operation = GetQueueUrl::builder()
        .queue_name("foo_queue")
        .build()
        .unwrap()
        .make_operation(&conf)
        .await
        .unwrap();
    make_unsigned(&mut operation);
    let resp = client
        .call(operation)
        .await
        .expect("request should succeed");

    println!("{}", resp.queue_url().unwrap_or_default());
}

// this function will work on any S3 operation
fn make_unsigned<I, R>(operation: &mut Operation<I, R>) {
    let mut props = operation.properties_mut();
    let mut signing_config = props
        .get_mut::<OperationSigningConfig>()
        .expect("has signing_config");
    signing_config.signing_requirements = SigningRequirements::Disabled;
}

🤞

@Velfi Velfi added the response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days. label Aug 17, 2022
@Velfi Velfi changed the title ConstructionFailure(MissingCredentials) when using IAM Roles attached to an instance ConstructionFailure(MissingCredentials) when using IAM Roles attached to an instance Aug 17, 2022
@Velfi
Copy link
Contributor

Velfi commented Aug 17, 2022

(It could also be that our AssumeRoleProvider is just broken in this case, but I wanted to rule out the other thing first)

@Velfi Velfi self-assigned this Aug 17, 2022
@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days. label Aug 17, 2022
@albx79
Copy link
Author

albx79 commented Aug 18, 2022

@Velfi I can confirm that your workaround works.

Thanks for the quick reply, I'll close this issue.

@albx79 albx79 closed this as completed Aug 18, 2022
@albx79
Copy link
Author

albx79 commented Aug 19, 2022

Sorry, I need to reopen this because I realized that, while the workaround prevents the client-side error and allows me to make the HTTP call to receive-messages, I'm now getting an Access Denied error from AWS.

I'll put some log lines here:

2022-08-19T16:08:45.515078Z TRACE receive{self=Queue { client: Client { connector: DynConnector, middleware: DefaultMiddleware, retry_policy: Standard { config: Config { initial_retry_tokens: 500, retry_cost: 5, no_retry_increment: 1, timeout_retry_cost: 10, max_attempts: 3, initial_backoff: 1s, max_backoff: 20s, base: 0x560edfac8b80 }, shared_state: CrossRequestRetryState { quota_available: Mutex { data: 500, poisoned: false, .. } } }, timeout_config: Config { api: Api { call: Unset, call_attempt: Unset }, http: Http { connect: Unset, write: Unset, read: Unset, tls_negotiation: Unset }, tcp: Tcp { connect: Unset, write: Unset, read: Unset } }, sleep_impl: Unset }, url: "https://sqs.eu-west-1.amazonaws.com/087849981844/excalibur-company-materializer-qrp-events", conf: Config }}:send_operation{operation="ReceiveMessage" service="sqs"}: aws_smithy_http_tower::dispatch: request=Request { method: POST, uri: https://sqs.eu-west-1.amazonaws.com/, version: HTTP/1.1, headers: {"content-type": "application/x-www-form-urlencoded", "content-length": "150", "user-agent": "aws-sdk-rust/0.47.0 os/linux lang/rust/1.61.0", "x-amz-user-agent": "aws-sdk-rust/0.47.0 api/sqs/0.17.0 os/linux lang/rust/1.61.0"}, body: SdkBody { inner: Once(Some(b"Action=ReceiveMessage&Version=2012-11-05&QueueUrl=https%3A%2F%2Fsqs.eu-west-1.amazonaws.com%2F087849981844%2Fexcalibur-company-materializer-qrp-events")), retryable: true } }

2022-08-19T16:08:45.541640Z TRACE receive{self=Queue { client: Client { connector: DynConnector, middleware: DefaultMiddleware, retry_policy: Standard { config: Config { initial_retry_tokens: 500, retry_cost: 5, no_retry_increment: 1, timeout_retry_cost: 10, max_attempts: 3, initial_backoff: 1s, max_backoff: 20s, base: 0x560edfac8b80 }, shared_state: CrossRequestRetryState { quota_available: Mutex { data: 500, poisoned: false, .. } } }, timeout_config: Config { api: Api { call: Unset, call_attempt: Unset }, http: Http { connect: Unset, write: Unset, read: Unset, tls_negotiation: Unset }, tcp: Tcp { connect: Unset, write: Unset, read: Unset } }, sleep_impl: Unset }, url: "https://sqs.eu-west-1.amazonaws.com/087849981844/excalibur-company-materializer-qrp-events", conf: Config }}:send_operation{operation="ReceiveMessage" service="sqs"}:load_response: aws_smithy_http::middleware: http_response=Response { status: 403, version: HTTP/1.1, headers: {"x-amzn-requestid": "9c900997-c548-5bc8-be2c-52ee7285eaeb", "connection": "close", "date": "Fri, 19 Aug 2022 16:08:45 GMT", "content-type": "text/xml", "content-length": "319"}, body: b"<?xml version=\"1.0\"?><ErrorResponse xmlns=\"http://queue.amazonaws.com/doc/2012-11-05/\"><Error><Type>Sender</Type><Code>AccessDenied</Code><Message>Access to the resource https://sqs.eu-west-1.amazonaws.com/ is denied.</Message><Detail/></Error><RequestId>9c900997-c548-5bc8-be2c-52ee7285eaeb</RequestId></ErrorResponse>" }

The same operation, in the same environment, using the CLI, works correctly.

@albx79 albx79 reopened this Aug 19, 2022
@rcoh
Copy link
Contributor

rcoh commented Aug 19, 2022

Can you use debug information to ensure you're hitting the same region and resolving the same credentials in the CLI and the Rust SDK? It appears to be dispatching the request successfully

Seems like you're using the receive_message API here based on the logs? Please send the CLI command your invoking along side the Rust code

@albx79
Copy link
Author

albx79 commented Aug 22, 2022

This is the CLI command:

aws sqs receive-message --queue-url=https://sqs.eu-west-1.amazonaws.com/087849981844/excalibur-company-materializer-qrp-events

The logs are a bit verbose, but we can already see that I'm hitting the correct URL/endpoint:

QueueUrl=https://sqs.eu-west-1.amazonaws.com/087849981844/excalibur-company-materializer-qrp-events

I'm not sure how to find out what credentials it's resolving. My understanding is that I'm not sending any credentials, and indeed I can't find any in the logs, but my knowledge of AWS credentials is still a bit sketchy.

@Velfi
Copy link
Contributor

Velfi commented Aug 22, 2022

@albx79 Would you be able to send us a minimum reproduction of what you're doing in your Rust code?

@albx79
Copy link
Author

albx79 commented Aug 24, 2022

Hi, here's the minimal example:

fn make_unsigned<I, R>(op: &mut Operation<I, R>) {
    let mut props = op.properties_mut();
    let mut signing_config = props
        .get_mut::<OperationSigningConfig>()
        .expect("has signing_config");
    signing_config.signing_requirements = SigningRequirements::Disabled;
}

#[tokio::main]
async fn main() -> () {
    let client = aws_smithy_client::Builder::dyn_https()
        .middleware(aws_sdk_sqs::middleware::DefaultMiddleware::new())
        .build();
    let conf = aws_config::from_env()
        .credentials_provider(aws_types::Credentials::new("stub", "stub", None, None, "faked"))
        .load()
        .await;
    let conf = aws_sdk_sqs::config::Builder::from(&conf).build();

    let mut rcv_msg_op = aws_sdk_sqs::operation::ReceiveMessage::builder()
        .queue_url("https://sqs.eu-west-1.amazonaws.com/087849981844/excalibur-company-materializer-qrp-events")
        .build().unwrap()
        .make_operation(&conf)
        .await.unwrap();
    make_unsigned(&mut rcv_msg_op);
    client.call(rcv_msg_op).await.unwrap();
}

And this is the output I get:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ServiceError { err: ReceiveMessageError { kind: Unhandled(Error { code: Some("AccessDenied"), message: Some("Access to the resource https://sqs.eu-west-1.amazonaws.com/ is denied."), request_id: Some("d672c10b-207d-5da9-9f2e-9a8b6b589100"), extras: {} }), meta: Error { code: Some("AccessDenied"), message: Some("Access to the resource https://sqs.eu-west-1.amazonaws.com/ is denied."), request_id: Some("d672c10b-207d-5da9-9f2e-9a8b6b589100"), extras: {} } }, raw: Response { inner: Response { status: 403, version: HTTP/1.1, headers: {"x-amzn-requestid": "d672c10b-207d-5da9-9f2e-9a8b6b589100", "connection": "close", "date": "Wed, 24 Aug 2022 15:51:01 GMT", "content-type": "text/xml", "content-length": "319"}, body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\"?><ErrorResponse xmlns=\"http://queue.amazonaws.com/doc/2012-11-05/\"><Error><Type>Sender</Type><Code>AccessDenied</Code><Message>Access to the resource https://sqs.eu-west-1.amazonaws.com/ is denied.</Message><Detail/></Error><RequestId>d672c10b-207d-5da9-9f2e-9a8b6b589100</RequestId></ErrorResponse>")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }', src/main.rs:66:35
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at ./rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: core::result::unwrap_failed
             at ./rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1785:5
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   4: std::thread::local::LocalKey<T>::with
   5: tokio::park::thread::CachedParkThread::block_on
   6: tokio::runtime::thread_pool::ThreadPool::block_on
   7: excalibur_company_materializer::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Thanks!

@Velfi
Copy link
Contributor

Velfi commented Aug 26, 2022

I finally got a k8s test environment set up and I'm looking into this.

@Velfi
Copy link
Contributor

Velfi commented Aug 26, 2022

Hey @albx70, after looking at this, I wasn't able to repro your issue so I have something for you to try (if you wouldn't mind.)

// [dependencies]
// aws-config = "0.47.0"
// aws-sdk-sqs = "0.17.0"
// aws-sdk-sts = "0.17.0"
// tokio = { version = "1.20.1", features = ["full"] }
// tracing = "0.1.36"
// tracing-subscriber = "0.3.15"


#[tokio::main]
async fn main() {
    tracing_subscriber::fmt::init();

    let conf = aws_config::load_from_env().await;

    {
        let client = aws_sdk_sts::Client::new(&conf);
        let res = client.get_caller_identity().send().await;
        tracing::info!("caller_identity = {:#?}", res)
    }

    let queue_name = std::env::var("SQS_QUEUE_NAME").unwrap();
    let client = aws_sdk_sqs::Client::new(&conf);

    let res = client
        .get_queue_url()
        .queue_name(queue_name)
        .send()
        .await
        .unwrap();
    let queue_url = res.queue_url().unwrap();

    let res = client.receive_message().queue_url(queue_url).send().await;

    match res {
        Ok(res) => tracing::info!("success: {:#?}", res),
        Err(e) => tracing::error!("failure: {:#?}", e),
    }
}

Run this like so:

RUST_LOG=debug cargo run

That will emit a bunch of logs related to resolving your credentials and they'll look something like this:

2022-08-26T15:50:00.558820Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
2022-08-26T15:50:00.558912Z DEBUG load_config_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2022-08-26T15:50:00.559024Z DEBUG load_config_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2022-08-26T15:50:00.559212Z DEBUG load_config_file: aws_config::profile::parser::source: config file not found path=~/.aws/config
2022-08-26T15:50:00.559344Z DEBUG load_config_file: aws_config::profile::parser::source: config file loaded path=~/.aws/config size=0
2022-08-26T15:50:00.559534Z DEBUG load_credentials_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2022-08-26T15:50:00.559694Z DEBUG load_credentials_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2022-08-26T15:50:00.559771Z DEBUG load_credentials_file: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2022-08-26T15:50:00.559821Z DEBUG load_credentials_file: aws_config::profile::parser::source: config file loaded path=~/.aws/credentials size=0
2022-08-26T15:50:00.560030Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
2022-08-26T15:50:00.560107Z DEBUG load_config_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2022-08-26T15:50:00.560170Z DEBUG load_config_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2022-08-26T15:50:00.560232Z DEBUG load_config_file: aws_config::profile::parser::source: config file not found path=~/.aws/config
2022-08-26T15:50:00.560280Z DEBUG load_config_file: aws_config::profile::parser::source: config file loaded path=~/.aws/config size=0
2022-08-26T15:50:00.560379Z DEBUG load_credentials_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2022-08-26T15:50:00.560438Z DEBUG load_credentials_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2022-08-26T15:50:00.560500Z DEBUG load_credentials_file: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2022-08-26T15:50:00.560559Z DEBUG load_credentials_file: aws_config::profile::parser::source: config file loaded path=~/.aws/credentials size=0
2022-08-26T15:50:00.574696Z DEBUG rustls::anchors: add_pem_file processed 137 valid and 0 invalid certs    
2022-08-26T15:50:00.574890Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
2022-08-26T15:50:00.575080Z DEBUG load_config_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2022-08-26T15:50:00.575153Z DEBUG load_config_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2022-08-26T15:50:00.575216Z DEBUG load_config_file: aws_config::profile::parser::source: config file not found path=~/.aws/config
2022-08-26T15:50:00.575265Z DEBUG load_config_file: aws_config::profile::parser::source: config file loaded path=~/.aws/config size=0
2022-08-26T15:50:00.575366Z DEBUG load_credentials_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2022-08-26T15:50:00.575511Z DEBUG load_credentials_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2022-08-26T15:50:00.575667Z DEBUG load_credentials_file: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2022-08-26T15:50:00.575722Z DEBUG load_credentials_file: aws_config::profile::parser::source: config file loaded path=~/.aws/credentials size=0
2022-08-26T15:50:00.579122Z DEBUG aws_endpoint: resolved endpoint endpoint=AwsEndpoint { endpoint: Endpoint { uri: https://sts.us-east-1.amazonaws.com/, immutable: false }, credential_scope: CredentialScope { region: Some(SigningRegion("us-east-1")), service: None } } base_region=Region("us-east-1")
2022-08-26T15:50:00.579487Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=environment variable not set
2022-08-26T15:50:00.579642Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2022-08-26T15:50:00.579912Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2022-08-26T15:50:00.580176Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2022-08-26T15:50:00.580385Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: config file not found path=~/.aws/config
2022-08-26T15:50:00.580459Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file: aws_config::profile::parser::source: config file loaded path=~/.aws/config size=0
2022-08-26T15:50:00.580590Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2022-08-26T15:50:00.580772Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2022-08-26T15:50:00.581278Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2022-08-26T15:50:00.581351Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_credentials_file: aws_config::profile::parser::source: config file loaded path=~/.aws/credentials size=0
2022-08-26T15:50:00.581522Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=No profiles were defined
2022-08-26T15:50:00.584774Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}: aws_endpoint: resolved endpoint endpoint=AwsEndpoint { endpoint: Endpoint { uri: https://sts.us-east-1.amazonaws.com/, immutable: false }, credential_scope: CredentialScope { region: Some(SigningRegion("us-east-1")), service: None } } base_region=Region("us-east-1")
2022-08-26T15:50:00.584990Z  INFO send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: aws_http::auth: provider returned CredentialsNotLoaded, ignoring
2022-08-26T15:50:00.585587Z DEBUG hyper::client::connect::dns: resolving host="sts.us-east-1.amazonaws.com"
2022-08-26T15:50:00.593329Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: hyper::client::connect::http: connecting to 209.54.177.185:443
2022-08-26T15:50:00.595556Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: hyper::client::connect::http: connected to 209.54.177.185:443
2022-08-26T15:50:00.595715Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: rustls::client::hs: No cached session for DNSNameRef("sts.us-east-1.amazonaws.com")    
2022-08-26T15:50:00.595841Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: rustls::client::hs: Not resuming any session    
2022-08-26T15:50:00.598147Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: rustls::client::hs: ALPN protocol is Some(b"http/1.1")    
2022-08-26T15:50:00.598230Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: rustls::client::hs: Using ciphersuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    
2022-08-26T15:50:00.598845Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: rustls::client::tls12: ECDHE curve is ECParameters { curve_type: NamedCurve, named_group: secp256r1 }    
2022-08-26T15:50:00.598971Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: rustls::client::tls12: Server DNS name is DNSName("sts.us-east-1.amazonaws.com")    
2022-08-26T15:50:00.602227Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: rustls::client::tls12: Session saved    
2022-08-26T15:50:00.603036Z DEBUG hyper::proto::h1::io: flushed 1592 bytes
2022-08-26T15:50:00.813101Z DEBUG hyper::proto::h1::io: parsed 4 headers
2022-08-26T15:50:00.813163Z DEBUG hyper::proto::h1::conn: incoming body is content-length (2454 bytes)
2022-08-26T15:50:00.813270Z DEBUG hyper::proto::h1::conn: incoming body completed
2022-08-26T15:50:00.813660Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}:load_credentials{provider=WebIdentityToken}:load_credentials{provider="WebIdentityToken"}:send_operation{operation="AssumeRoleWithWebIdentity" service="sts"}: hyper::client::pool: pooling idle connection for ("https", sts.us-east-1.amazonaws.com)
2022-08-26T15:50:00.814788Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=WebIdentityToken
2022-08-26T15:50:00.816147Z DEBUG hyper::client::connect::dns: resolving host="sts.us-east-1.amazonaws.com"
2022-08-26T15:50:00.822082Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: hyper::client::connect::http: connecting to 209.54.177.185:443
2022-08-26T15:50:00.824170Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: hyper::client::connect::http: connected to 209.54.177.185:443
2022-08-26T15:50:00.824338Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: rustls::client::hs: Resuming session    
2022-08-26T15:50:00.826515Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: rustls::client::hs: ALPN protocol is Some(b"http/1.1")    
2022-08-26T15:50:00.826576Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: rustls::client::hs: Using ciphersuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    
2022-08-26T15:50:00.827123Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: rustls::client::tls12: ECDHE curve is ECParameters { curve_type: NamedCurve, named_group: secp256r1 }    
2022-08-26T15:50:00.827207Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: rustls::client::tls12: Server DNS name is DNSName("sts.us-east-1.amazonaws.com")    
2022-08-26T15:50:00.829804Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: rustls::client::tls12: Session saved    
2022-08-26T15:50:00.830450Z DEBUG hyper::proto::h1::io: flushed 1894 bytes
2022-08-26T15:50:00.844332Z DEBUG hyper::proto::h1::io: parsed 4 headers
2022-08-26T15:50:00.844365Z DEBUG hyper::proto::h1::conn: incoming body is content-length (535 bytes)
2022-08-26T15:50:00.844433Z DEBUG hyper::proto::h1::conn: incoming body completed
2022-08-26T15:50:00.844669Z DEBUG send_operation{operation="GetCallerIdentity" service="sts"}: hyper::client::pool: pooling idle connection for ("https", sts.us-east-1.amazonaws.com)
2022-08-26T15:50:00.845486Z  INFO imds_creds_missing_in_k8s: caller_identity = Ok(
    GetCallerIdentityOutput {
        user_id: Some(
            "AROA2LYEZOZTAQGKPRZO2:web-identity-token-1661529000581",
        ),
        account: Some(
            "712437757542",
        ),
        arn: Some(
            "arn:aws:sts::712437757542:assumed-role/EksCredentialsStack-helloekseksserviceaccountRole7-1ANCUXW66N12E/web-identity-token-1661529000581",
        ),
    },
)
2022-08-26T15:50:00.845648Z DEBUG rustls::session: Sending warning alert CloseNotify    
2022-08-26T15:50:00.849351Z DEBUG aws_endpoint: resolved endpoint endpoint=AwsEndpoint { endpoint: Endpoint { uri: https://sqs.us-east-1.amazonaws.com/, immutable: false }, credential_scope: CredentialScope { region: Some(SigningRegion("us-east-1")), service: None } } base_region=Region("us-east-1")
2022-08-26T15:50:00.849601Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2022-08-26T15:50:00.850525Z DEBUG hyper::client::connect::dns: resolving host="sqs.us-east-1.amazonaws.com"
2022-08-26T15:50:00.857497Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: hyper::client::connect::http: connecting to 3.236.169.32:443
2022-08-26T15:50:00.860509Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: hyper::client::connect::http: connected to 3.236.169.32:443
2022-08-26T15:50:00.860635Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: rustls::client::hs: No cached session for DNSNameRef("sqs.us-east-1.amazonaws.com")    
2022-08-26T15:50:00.860704Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: rustls::client::hs: Not resuming any session    
2022-08-26T15:50:00.863080Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: rustls::client::hs: ALPN protocol is Some(b"http/1.1")    
2022-08-26T15:50:00.863159Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: rustls::client::hs: Using ciphersuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    
2022-08-26T15:50:00.864002Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: rustls::client::tls12: ECDHE curve is ECParameters { curve_type: NamedCurve, named_group: secp256r1 }    
2022-08-26T15:50:00.864085Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: rustls::client::tls12: Server DNS name is DNSName("sqs.us-east-1.amazonaws.com")    
2022-08-26T15:50:00.866850Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: rustls::client::tls12: Session saved    
2022-08-26T15:50:00.867443Z DEBUG hyper::proto::h1::io: flushed 1960 bytes
2022-08-26T15:50:00.889446Z DEBUG hyper::proto::h1::io: parsed 4 headers
2022-08-26T15:50:00.889488Z DEBUG hyper::proto::h1::conn: incoming body is content-length (366 bytes)
2022-08-26T15:50:00.889572Z DEBUG hyper::proto::h1::conn: incoming body completed
2022-08-26T15:50:00.889883Z DEBUG send_operation{operation="GetQueueUrl" service="sqs"}: hyper::client::pool: pooling idle connection for ("https", sqs.us-east-1.amazonaws.com)
2022-08-26T15:50:00.890730Z DEBUG aws_endpoint: resolved endpoint endpoint=AwsEndpoint { endpoint: Endpoint { uri: https://sqs.us-east-1.amazonaws.com/, immutable: false }, credential_scope: CredentialScope { region: Some(SigningRegion("us-east-1")), service: None } } base_region=Region("us-east-1")
2022-08-26T15:50:00.890973Z DEBUG send_operation{operation="ReceiveMessage" service="sqs"}:provide_credentials{provider=default_chain}: aws_config::meta::credentials::lazy_caching: loaded credentials from cache
2022-08-26T15:50:00.891897Z DEBUG send_operation{operation="ReceiveMessage" service="sqs"}: hyper::client::pool: reuse idle connection for ("https", sqs.us-east-1.amazonaws.com)
2022-08-26T15:50:00.892229Z DEBUG hyper::proto::h1::io: flushed 2022 bytes
2022-08-26T15:50:01.006567Z DEBUG hyper::proto::h1::io: parsed 4 headers
2022-08-26T15:50:01.006619Z DEBUG hyper::proto::h1::conn: incoming body is content-length (240 bytes)
2022-08-26T15:50:01.006714Z DEBUG hyper::proto::h1::conn: incoming body completed
2022-08-26T15:50:01.007061Z DEBUG send_operation{operation="ReceiveMessage" service="sqs"}: hyper::client::pool: pooling idle connection for ("https", sqs.us-east-1.amazonaws.com)
2022-08-26T15:50:01.007642Z  INFO imds_creds_missing_in_k8s: success: ReceiveMessageOutput {
    messages: None,
}
2022-08-26T15:50:01.007777Z DEBUG rustls::session: Sending warning alert CloseNotify    
2022-08-26T15:50:01.008784Z DEBUG rustls::session: Sending warning alert CloseNotify

What I'm hoping is that we'll see that it's either not loading your creds OR it's loading the wrong creds. Also, this will log the output of sts#GetCallerIdentity. Make sure that it returns the same output as when you run the AWS CLI because that'll show if the CLI and your app are accessing different identities.

@albx79
Copy link
Author

albx79 commented Aug 29, 2022

I'm not sure what changed, but it's working now. Thanks for your help, and sorry for the inconvenience.

@albx79 albx79 closed this as completed Aug 29, 2022
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@Velfi
Copy link
Contributor

Velfi commented Aug 29, 2022

Glad it's working!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
Archived in project
Development

No branches or pull requests

3 participants