Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

put_snapshot_block consistently fails with InvalidSignatureException #930

Closed
anelson opened this issue Oct 28, 2023 · 8 comments
Closed

put_snapshot_block consistently fails with InvalidSignatureException #930

anelson opened this issue Oct 28, 2023 · 8 comments
Labels
bug This issue is a bug. p2 This is a standard priority issue pending-release This issue will be fixed by an approved PR that hasn't been released yet.

Comments

@anelson
Copy link

anelson commented Oct 28, 2023 •
edited
Loading

Describe the bug

The put_snapshot_block function in aws-sdk-ebs seems to be completely broken in version 0.34.0 (and possibly other version; only tried 0.34 and 0.33).

The function call always fails with error InvalidSignatureException, although I'm sure it's not an issue w/ my AWS credential config or with the computation of the checksum for the block data.

Expected Behavior

When calling put_snapshot_block with a valid 512KiB block and corresponding SHA256 checksum, I expect the call to succeed.

Current Behavior

The function call always fails with an error:

Error: Unhandled(Unhandled { source: ErrorMetadata { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), extras: Some({"aws_request_id": "0ffad7cf-a867-4ff2-9401-a32a558e2219"}) }, meta: ErrorMetadata { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), extras: Some({"aws_request_id": "0ffad7cf-a867-4ff2-9401-a32a558e2219"}) } })

Reproduction Steps

I discovered this when writing my own test project to explore EBS direct using the AWS SDK for Rust, but actually the SDK's own example code appears to fail the same way.

This code:

aws-sdk-rust/examples/examples/ebs/src/bin/create-snapshot.rs

Lines 57 to 66 in 464d782

client
.put_snapshot_block()
.snapshot_id(id)
.block_index(idx as i32)
.block_data(ByteStream::from(block))
.checksum(checksum)
.checksum_algorithm(ChecksumAlgorithm::ChecksumAlgorithmSha256)
.data_length(EBS_BLOCK_SIZE as i32)
.send()
.await?;

always fails for me.

From the examples/examples/ebs directory in the aws-sdk-rust repo I run this command:

cargo run --bin create-snapshot -- --description "snapshot from AWS SDK examples"

The resulting output is:

Error: Unhandled(Unhandled { source: ErrorMetadata { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), extras: Some({"aws_request_id": "0ffad7cf-a867-4ff2-9401-a32a558e2219"}) }, meta: ErrorMetadata { code: Some("InvalidSignatureException"), message: Some("The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."), extras: Some({"aws_request_id": "0ffad7cf-a867-4ff2-9401-a32a558e2219"}) } })

Note that I'm sure the error is on the put_snapshot_block call, not the earlier create_snapshot call. I know this because I modified the code to println right before the put_snapshot_block call, and also because I can see in the AWS console that a Pending snapshot has been created:

image

So my credentials themselves are configured correctly. I've tried both my usual SSO-based authentication, as well as creating an IAM user with static credentials; both fail the same way.

Possible Solution

I assume that what makes put_snapshot_block unusual is that the block payload is not supposed to be signed with the rest of the request, because the caller must provide a checksum explicitly. Perhaps there's a bug in the generated client code that is including the block data in the signature? Otherwise I have no idea.

Additional Information/Context

I modified my code to use the Rusoto Rust bindings instead, and it works correctly. So it's definitely not environmental, it's specific to the AWS SDK implementation somehow.

Version

ebs-code-examples v0.1.0 (/home/cornelius/sources/aws-sdk-rust/examples/examples/ebs)
├── aws-config v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-config)
│   ├── aws-credential-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-credential-types)
│   │   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async)
│   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types)
│   ├── aws-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-http)
│   │   ├── aws-credential-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http)
│   │   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-types)
│   │   │   ├── aws-credential-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   │   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-client v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-client)
│   │   │   │   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   │   ├── aws-smithy-http-tower v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http-tower)
│   │   │   │   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   │   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   │   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-sdk-sso v0.34.0 (/home/cornelius/sources/aws-sdk-rust/sdk/sso)
│   │   ├── aws-credential-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-http) (*)
│   │   ├── aws-runtime v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-runtime)
│   │   │   ├── aws-credential-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   │   ├── aws-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-http) (*)
│   │   │   ├── aws-sigv4 v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-sigv4)
│   │   │   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   ├── aws-smithy-runtime-api v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime-api)
│   │   │   │   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   │   ├── aws-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-types) (*)
│   │   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-client v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-client) (*)
│   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-json v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-json)
│   │   │   └── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime)
│   │   │   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   │   ├── aws-smithy-client v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-client) (*)
│   │   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   │   ├── aws-smithy-runtime-api v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime-api v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-types) (*)
│   ├── aws-sdk-sts v0.34.0 (/home/cornelius/sources/aws-sdk-rust/sdk/sts)
│   │   ├── aws-credential-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-credential-types) (*)
│   │   ├── aws-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-http) (*)
│   │   ├── aws-runtime v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-runtime) (*)
│   │   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   │   ├── aws-smithy-client v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-client) (*)
│   │   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   │   ├── aws-smithy-json v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   │   ├── aws-smithy-query v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-query)
│   │   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-runtime v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   │   ├── aws-smithy-runtime-api v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   │   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   │   ├── aws-smithy-xml v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-xml)
│   │   ├── aws-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-types) (*)
│   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   ├── aws-smithy-client v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-client) (*)
│   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   ├── aws-smithy-http-tower v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http-tower) (*)
│   ├── aws-smithy-json v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-types) (*)
├── aws-sdk-ebs v0.34.0 (/home/cornelius/sources/aws-sdk-rust/sdk/ebs)
│   ├── aws-credential-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-credential-types) (*)
│   ├── aws-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-http) (*)
│   ├── aws-runtime v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-runtime) (*)
│   ├── aws-sigv4 v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-sigv4) (*)
│   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   ├── aws-smithy-client v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-client) (*)
│   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   ├── aws-smithy-json v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   ├── aws-smithy-runtime v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   ├── aws-smithy-runtime-api v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-types) (*)
├── aws-sdk-ec2 v0.34.0 (/home/cornelius/sources/aws-sdk-rust/sdk/ec2)
│   ├── aws-credential-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-credential-types) (*)
│   ├── aws-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-http) (*)
│   ├── aws-runtime v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-runtime) (*)
│   ├── aws-smithy-async v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-async) (*)
│   ├── aws-smithy-client v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-client) (*)
│   ├── aws-smithy-http v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-http) (*)
│   ├── aws-smithy-json v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-json) (*)
│   ├── aws-smithy-query v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-query) (*)
│   ├── aws-smithy-runtime v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime) (*)
│   ├── aws-smithy-runtime-api v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-runtime-api) (*)
│   ├── aws-smithy-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-types) (*)
│   ├── aws-smithy-xml v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-smithy-xml) (*)
│   ├── aws-types v0.56.1 (/home/cornelius/sources/aws-sdk-rust/sdk/aws-types) (*)

Environment details (OS name and version, etc.)

Fedora Linux 37

Logs

2023-10-28T18:00:08.064079Z DEBUG load_region{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }, ImdsRegionProvider { client: LazyClient { client: OnceCell { value: None }, builder: Builder { max_attempts: None, endpoint: None, mode_override: None, token_ttl: None, connect_timeout: None, read_timeout: None, config: Some(ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None }) } }, env: Env(Real) }] })}:load_region{provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }}: aws_config::fs_util: loaded home directory src="HOME"
2023-10-28T18:00:08.064131Z DEBUG load_region{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }, ImdsRegionProvider { client: LazyClient { client: OnceCell { value: None }, builder: Builder { max_attempts: None, endpoint: None, mode_override: None, token_ttl: None, connect_timeout: None, read_timeout: None, config: Some(ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None }) } }, env: Env(Real) }] })}:load_region{provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/home/cornelius" path="~/.aws/config"
2023-10-28T18:00:08.064161Z DEBUG load_region{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }, ImdsRegionProvider { client: LazyClient { client: OnceCell { value: None }, builder: Builder { max_attempts: None, endpoint: None, mode_override: None, token_ttl: None, connect_timeout: None, read_timeout: None, config: Some(ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None }) } }, env: Env(Real) }] })}:load_region{provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/home/cornelius/.aws/config"
2023-10-28T18:00:08.064239Z DEBUG load_region{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }, ImdsRegionProvider { client: LazyClient { client: OnceCell { value: None }, builder: Builder { max_attempts: None, endpoint: None, mode_override: None, token_ttl: None, connect_timeout: None, read_timeout: None, config: Some(ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None }) } }, env: Env(Real) }] })}:load_region{provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/home/cornelius/.aws/config") size=67616
2023-10-28T18:00:08.064293Z DEBUG load_region{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }, ImdsRegionProvider { client: LazyClient { client: OnceCell { value: None }, builder: Builder { max_attempts: None, endpoint: None, mode_override: None, token_ttl: None, connect_timeout: None, read_timeout: None, config: Some(ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None }) } }, env: Env(Real) }] })}:load_region{provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/home/cornelius" path="~/.aws/credentials"
2023-10-28T18:00:08.064315Z DEBUG load_region{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }, ImdsRegionProvider { client: LazyClient { client: OnceCell { value: None }, builder: Builder { max_attempts: None, endpoint: None, mode_override: None, token_ttl: None, connect_timeout: None, read_timeout: None, config: Some(ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None }) } }, env: Env(Real) }] })}:load_region{provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/home/cornelius/.aws/credentials"
2023-10-28T18:00:08.064342Z DEBUG load_region{provider=DefaultRegionChain(RegionProviderChain { providers: [EnvironmentVariableRegionProvider { env: Env(Real) }, ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }, ImdsRegionProvider { client: LazyClient { client: OnceCell { value: None }, builder: Builder { max_attempts: None, endpoint: None, mode_override: None, token_ttl: None, connect_timeout: None, read_timeout: None, config: Some(ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None }) } }, env: Env(Real) }] })}:load_region{provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs: Fs(Real), sleep: Some(SharedAsyncSleep(TokioSleep)), region: None } }}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/home/cornelius/.aws/credentials") size=1100
2023-10-28T18:00:08.072150Z DEBUG aws_config::fs_util: loaded home directory src="HOME"
2023-10-28T18:00:08.072185Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/home/cornelius" path="~/.aws/config"
2023-10-28T18:00:08.072205Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/home/cornelius/.aws/config"
2023-10-28T18:00:08.072270Z DEBUG load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/home/cornelius/.aws/config") size=67616
2023-10-28T18:00:08.072304Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/home/cornelius" path="~/.aws/credentials"
2023-10-28T18:00:08.072320Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/home/cornelius/.aws/credentials"
2023-10-28T18:00:08.072341Z DEBUG load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/home/cornelius/.aws/credentials") size=1100
2023-10-28T18:00:08.083030Z DEBUG build_profile_provider: hyper_rustls::config: with_native_roots processed 140 valid and 0 invalid certs    
2023-10-28T18:00:08.103700Z DEBUG aws_sdk_sts::config: using retry strategy with partition 'sts'
2023-10-28T18:00:08.124321Z DEBUG aws_sdk_ebs::config: using retry strategy with partition 'ebs'
2023-10-28T18:00:08.124691Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase
2023-10-28T18:00:08.124799Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase
2023-10-28T18:00:08.124855Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op: aws_smithy_runtime::client::retries::strategy::standard: no client rate limiter configured, so no token is required for the initial request.
2023-10-28T18:00:08.124874Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op: aws_smithy_runtime::client::orchestrator: retry strategy has OKed initial request
2023-10-28T18:00:08.124897Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op: aws_smithy_runtime::client::orchestrator: beginning attempt #1
2023-10-28T18:00:08.124955Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: will use endpoint Endpoint { url: "https://ebs.us-east-2.amazonaws.com", headers: {}, properties: {} }
2023-10-28T18:00:08.125175Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "environment variable not set" }))
2023-10-28T18:00:08.125256Z  INFO invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: CredentialProcess("/home/cornelius/.local/bin/aws-sso ** arguments redacted **"), chain: [] }
2023-10-28T18:00:08.125298Z  INFO invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::profile::credentials::exec: first credentials will be loaded from CredentialProcess("/home/cornelius/.local/bin/aws-sso ** arguments redacted **") base=CredentialProcess("/home/cornelius/.local/bin/aws-sso ** arguments redacted **")
2023-10-28T18:00:08.125341Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_base_credentials: aws_config::credential_process: loading credentials from external process command=/home/cornelius/.local/bin/aws-sso ** arguments redacted **
2023-10-28T18:00:08.451703Z  INFO invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::profile::credentials: loaded base credentials creds=Credentials { provider_name: "CredentialProcess", access_key_id: "ASIA6C7DXBZJSY7DQJAS", secret_access_key: "** redacted **", expires_after: "2023-10-29T00:24:06Z" }
2023-10-28T18:00:08.451794Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Profile
2023-10-28T18:00:08.451847Z  INFO invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt:lazy_load_credentials: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; added new AWS credentials (took Ok(326.761043ms))
2023-10-28T18:00:08.451881Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: aws_credential_types::cache::lazy_caching: loaded credentials
2023-10-28T18:00:08.452586Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'transmit' phase
2023-10-28T18:00:08.452907Z DEBUG hyper::client::connect::dns: resolving host="ebs.us-east-2.amazonaws.com"
2023-10-28T18:00:08.454427Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: hyper::client::connect::http: connecting to 3.141.102.191:443
2023-10-28T18:00:08.568978Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: hyper::client::connect::http: connected to 3.141.102.191:443
2023-10-28T18:00:08.569053Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: rustls::client::hs: No cached session for DnsName("ebs.us-east-2.amazonaws.com")    
2023-10-28T18:00:08.569513Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: rustls::client::hs: Not resuming any session    
2023-10-28T18:00:08.683884Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: rustls::client::hs: ALPN protocol is Some(b"http/1.1")    
2023-10-28T18:00:08.683943Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: rustls::client::hs: Using ciphersuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256    
2023-10-28T18:00:08.684608Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: rustls::client::tls12: ECDHE curve is ECParameters { curve_type: NamedCurve, named_group: secp256r1 }    
2023-10-28T18:00:08.684669Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: rustls::client::tls12: Server DNS name is DnsName("ebs.us-east-2.amazonaws.com")    
2023-10-28T18:00:08.800511Z DEBUG hyper::proto::h1::io: flushed 1705 bytes
2023-10-28T18:00:09.034437Z DEBUG hyper::proto::h1::io: parsed 4 headers
2023-10-28T18:00:09.034500Z DEBUG hyper::proto::h1::conn: incoming body is content-length (261 bytes)
2023-10-28T18:00:09.034552Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-10-28T18:00:09.034664Z DEBUG hyper::client::pool: pooling idle connection for ("https", ebs.us-east-2.amazonaws.com)
2023-10-28T18:00:09.034649Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'before deserialization' phase
2023-10-28T18:00:09.034815Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'deserialization' phase
2023-10-28T18:00:09.034985Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt:deserialization:deserialize_nonstreaming: aws_sdk_ebs::operation::start_snapshot: request_id=Some("77ca3861-1fc2-49da-b0db-ed68227bf448")
2023-10-28T18:00:09.035115Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'after deserialization' phase
2023-10-28T18:00:09.035253Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op: aws_smithy_runtime::client::retries::strategy::standard: request succeeded, no retry necessary
2023-10-28T18:00:09.035288Z DEBUG invoke{service=ebs operation=StartSnapshot}:try_op: aws_smithy_runtime::client::orchestrator: a retry is either unnecessary or not possible, exiting attempt loop
Adding block 0 to snapshot snap-0d8c55964e06c3435
2023-10-28T18:00:09.061192Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'serialization' phase
2023-10-28T18:00:09.061313Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op: aws_smithy_runtime_api::client::interceptors::context: entering 'before transmit' phase
2023-10-28T18:00:09.061396Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op: aws_smithy_runtime::client::retries::strategy::standard: no client rate limiter configured, so no token is required for the initial request.
2023-10-28T18:00:09.061422Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op: aws_smithy_runtime::client::orchestrator: retry strategy has OKed initial request
2023-10-28T18:00:09.061460Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op: aws_smithy_runtime::client::orchestrator: beginning attempt #1
2023-10-28T18:00:09.061531Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt: aws_smithy_runtime::client::orchestrator::endpoints: will use endpoint Endpoint { url: "https://ebs.us-east-2.amazonaws.com", headers: {}, properties: {} }
2023-10-28T18:00:09.061691Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt: aws_credential_types::cache::lazy_caching: loaded credentials from cache
2023-10-28T18:00:09.062205Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'transmit' phase
2023-10-28T18:00:09.062309Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt: hyper::client::pool: reuse idle connection for ("https", ebs.us-east-2.amazonaws.com)
2023-10-28T18:00:09.063099Z DEBUG hyper::proto::h1::io: flushed 131072 bytes
2023-10-28T18:00:09.063176Z DEBUG hyper::proto::h1::io: flushed 12424 bytes
2023-10-28T18:00:09.177451Z DEBUG hyper::proto::h1::io: flushed 65536 bytes
2023-10-28T18:00:09.177514Z DEBUG hyper::proto::h1::io: flushed 18303 bytes
2023-10-28T18:00:09.291671Z DEBUG hyper::proto::h1::io: flushed 65536 bytes
2023-10-28T18:00:09.291750Z DEBUG hyper::proto::h1::io: flushed 41442 bytes
2023-10-28T18:00:09.405508Z DEBUG hyper::proto::h1::io: flushed 131072 bytes
2023-10-28T18:00:09.405566Z DEBUG hyper::proto::h1::io: flushed 7617 bytes
2023-10-28T18:00:09.406841Z DEBUG hyper::proto::h1::io: flushed 53080 bytes
2023-10-28T18:00:09.751014Z DEBUG hyper::proto::h1::io: parsed 5 headers
2023-10-28T18:00:09.751077Z DEBUG hyper::proto::h1::conn: incoming body is content-length (192 bytes)
2023-10-28T18:00:09.751149Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-10-28T18:00:09.751298Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt: hyper::client::pool: pooling idle connection for ("https", ebs.us-east-2.amazonaws.com)
2023-10-28T18:00:09.751405Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'before deserialization' phase
2023-10-28T18:00:09.751545Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'deserialization' phase
2023-10-28T18:00:09.751735Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt:deserialization:deserialize_nonstreaming: aws_sdk_ebs::operation::put_snapshot_block: request_id=Some("399593ed-2251-424f-b719-d29d0a43f157")
2023-10-28T18:00:09.751866Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'after deserialization' phase
2023-10-28T18:00:09.751996Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op: aws_smithy_runtime::client::retries::strategy::standard: encountered unretryable error attempts=1 max_attempts=3
2023-10-28T18:00:09.752044Z DEBUG invoke{service=ebs operation=PutSnapshotBlock}:try_op: aws_smithy_runtime::client::orchestrator: a retry is either unnecessary or not possible, exiting attempt loop
2023-10-28T18:00:09.752554Z DEBUG rustls::common_state: Sending warning alert CloseNotify
@anelson anelson added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 28, 2023
@rcoh rcoh removed the needs-triage This issue or PR still needs to be triaged. label Oct 30, 2023
@rcoh
Copy link
Contributor

rcoh commented Oct 30, 2023

Seems like a regression. We'll take a look ASAP

@rcoh rcoh added the p2 This is a standard priority issue label Oct 30, 2023
@rcoh
Copy link
Contributor

rcoh commented Nov 1, 2023

Looks like this regressed between 0.28 (works) and 0.29 (broken) when we rolled out the orchestrator. We'll get a fix out shortly.

@rcoh
Copy link
Contributor

rcoh commented Nov 1, 2023

Fix: smithy-lang/smithy-rs#3126

@rcoh rcoh added the pending-release This issue will be fixed by an approved PR that hasn't been released yet. label Nov 1, 2023
@anelson
Copy link
Author

anelson commented Nov 1, 2023

Thanks @rcoh for the very rapid turnaround!

@rcoh
Copy link
Contributor

rcoh commented Nov 7, 2023

this missed the train for 0.57.1 but will go out in 0.57.2

@anelson
Copy link
Author

anelson commented Nov 7, 2023

Cool, thanks for the update. Any sense of when 0.57.2 would be going out? I'm not familiar w/ the typical SDK release cadence...

@rcoh
Copy link
Contributor

rcoh commented Nov 11, 2023

should go out early next week

@rcoh rcoh closed this as completed Nov 15, 2023
@github-actions GitHub Actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. p2 This is a standard priority issue pending-release This issue will be fixed by an approved PR that hasn't been released yet.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@anelson @rcoh