Extra whitespace in S3 object metadata causes SignatureDoesNotMatch error

[mlafeldt]

Bug Report

Today I noticed that a specific PutObject operation failed due to a SignatureDoesNotMatch error:

[2021-10-10T13:19:18Z INFO smithy_http_tower::parse_response] send_operation; operation="PutObject"
--
  | [2021-10-10T13:19:18Z INFO smithy_http_tower::parse_response] send_operation; service="s3"
  | [2021-10-10T13:19:18Z INFO aws_config::meta::credentials::chain] load_credentials; provider=Environment
  | [2021-10-10T13:19:18Z INFO aws_config::meta::credentials::chain] loaded credentials provider=Environment
  | [2021-10-10T13:19:18Z INFO smithy_http_tower::parse_response] send_operation; status="service_err"
  | [2021-10-10T13:19:18Z INFO smithy_http_tower::parse_response] send_operation; message=Error { code: "SignatureDoesNotMatch", message: "The request signature we calculated does not match the signature you provided. Check your key and signing method.", request_id: "FDMZ3T5JHGXBGVF3", s3_extended_request_id: "GTg448Gmgjt6kQCqCOchyKu0H13tM3+XMPmS39MuaQ49empi7t7AGaz09wHBuqmeUA05msHijo0=" }
  | [2021-10-10T13:19:18Z ERROR get_strip] failed to put object strips/2021-10-10.gif
  | Caused by:
  | 0: Error { code: "SignatureDoesNotMatch", message: "The request signature we calculated does not match the signature you provided. Check your key and signing method.", request_id: "FDMZ3T5JHGXBGVF3", s3_extended_request_id: "GTg448Gmgjt6kQCqCOchyKu0H13tM3+XMPmS39MuaQ49empi7t7AGaz09wHBuqmeUA05msHijo0=" }
  | 1: Error { code: "SignatureDoesNotMatch", message: "The request signature we calculated does not match the signature you provided. Check your key and signing method.", request_id: "FDMZ3T5JHGXBGVF3", s3_extended_request_id: "GTg448Gmgjt6kQCqCOchyKu0H13tM3+XMPmS39MuaQ49empi7t7AGaz09wHBuqmeUA05msHijo0=" }
  | 2: Error { code: "SignatureDoesNotMatch", message: "The request signature we calculated does not match the signature you provided. Check your key and signing method.", request_id: "FDMZ3T5JHGXBGVF3", s3_extended_request_id: "GTg448Gmgjt6kQCqCOchyKu0H13tM3+XMPmS39MuaQ49empi7t7AGaz09wHBuqmeUA05msHijo0=" }
  | [2021-10-10T13:19:18Z ERROR lambda_runtime] failed to put object strips/2021-10-10.gif

This can be reproduced locally with this code:

#[tokio::main]
async fn main() {
    aws_sdk_s3::Client::new(&aws_config::load_from_env().await)
        .put_object()
        .bucket("some-bucket")
        .key("strips/2021-10-10.gif")
        .body(aws_sdk_s3::ByteStream::from_static(b"hello"))
        .metadata("title", "Sunday   Data Looks Two Ways") // here's the problem
        .send()
        .await
        .unwrap();
}

Note that removing the repeated whitespace from metadata will fix the issue. May or may not be related to #226.

Full backtrace:

$ RUST_BACKTRACE=full cargo run --bin bug
    Finished dev [unoptimized + debuginfo] target(s) in 0.37s
     Running `target/debug/bug`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ServiceError { err: PutObjectError { kind: Unhandled(Error { code: Some("SignatureDoesNotMatch"), message: Some("The request signature we calculated does not match the signature you provided. Check your key and signing method."), request_id: Some("2RYD4W7SB5S3MMKC"), extras: {"s3_extended_request_id": "djjMKM1bJwU4dBdCsoIcEVFHYw2vGb2EtMw4Z7Fq7DvOE/mFfQDqU1ZaOanFHclRh0P1aIAjZac="} }), meta: Error { code: Some("SignatureDoesNotMatch"), message: Some("The request signature we calculated does not match the signature you provided. Check your key and signing method."), request_id: Some("2RYD4W7SB5S3MMKC"), extras: {"s3_extended_request_id": "djjMKM1bJwU4dBdCsoIcEVFHYw2vGb2EtMw4Z7Fq7DvOE/mFfQDqU1ZaOanFHclRh0P1aIAjZac="} } }, raw: Response { inner: Response { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "2RYD4W7SB5S3MMKC", "x-amz-id-2": "djjMKM1bJwU4dBdCsoIcEVFHYw2vGb2EtMw4Z7Fq7DvOE/mFfQDqU1ZaOanFHclRh0P1aIAjZac=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Sun, 10 Oct 2021 13:47:40 GMT", "server": "AmazonS3", "connection": "close"}, body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>XXX</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256\n20211010T134741Z\n20211010/eu-central-1/s3/aws4_request\n1dffeae4c737c7f129d65b5af94149c9d7ea1c54121524131e874050699e0b00</StringToSign><SignatureProvided>506e7587127cdb7ada909bf17114e7ba3a2acfcf03471362ab264fb3e3369fed</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 31 31 30 31 30 54 31 33 34 37 34 31 5a 0a 32 30 32 31 31 30 31 30 2f 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 31 64 66 66 65 61 65 34 63 37 33 37 63 37 66 31 32 39 64 36 35 62 35 61 66 39 34 31 34 39 63 39 64 37 65 61 31 63 35 34 31 32 31 35 32 34 31 33 31 65 38 37 34 30 35 30 36 39 39 65 30 62 30 30</StringToSignBytes><CanonicalRequest>PUT\n/some-bucket/strips/2021-10-10.gif\nx-id=PutObject\ncontent-length:5\ncontent-type:application/octet-stream\nhost:s3.eu-central-1.amazonaws.com\nx-amz-content-sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824\nx-amz-date:20211010T134741Z\nx-amz-meta-title:Sunday Data Looks Two Ways\nx-amz-security-token:FwoGZXIvYXdzEG8aDFnLPe3Jc5bePtvkliKCAe3uKdp31TD0x7fcvY1J4AA/u2AWvAwQpIgoo+KRmEbVYCnKomK8PnBHhz86VJzyYTFfR+b0+KT5ZGWvuY4ETe2vpZCKQsQDmRtxcYw0wSyNJiaJ4crJA4zcUlz0oyC/ygVk3K2cme/fH0Tt8d7V1DdbksI1KrRxfD/PYl0NvM45cT8o8tKLiwYyKO+msnG8FQAW8p5/fKWI0QNC28jgjVE04GbXEjxw1H3dsVLMTQB1Gps=\nx-amz-user-agent:aws-sdk-rust/0.1.0 api/s3/0.0.19-alpha os/macos lang/rust/1.55.0\n\ncontent-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-title;x-amz-security-token;x-amz-user-agent\n2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824</CanonicalRequest><CanonicalRequestBytes>50 55 54 0a 2f 64 69 6c 62 65 72 74 2d 66 65 65 64 2d 64 65 76 2d 62 75 63 6b 65 74 38 33 39 30 38 65 37 37 2d 31 76 66 36 32 71 38 6f 6d 6b 35 62 66 2f 73 74 72 69 70 73 2f 32 30 32 31 2d 31 30 2d 31 30 2e 67 69 66 0a 78 2d 69 64 3d 50 75 74 4f 62 6a 65 63 74 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a 35 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0a 68 6f 73 74 3a 73 33 2e 65 75 2d 63 65 6e 74 72 61 6c 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 32 63 66 32 34 64 62 61 35 66 62 30 61 33 30 65 32 36 65 38 33 62 32 61 63 35 62 39 65 32 39 65 31 62 31 36 31 65 35 63 31 66 61 37 34 32 35 65 37 33 30 34 33 33 36 32 39 33 38 62 39 38 32 34 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 31 31 30 31 30 54 31 33 34 37 34 31 5a 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 74 69 74 6c 65 3a 53 75 6e 64 61 79 20 44 61 74 61 20 4c 6f 6f 6b 73 20 54 77 6f 20 57 61 79 73 0a 78 2d 61 6d 7a 2d 73 65 63 75 72 69 74 79 2d 74 6f 6b 65 6e 3a 46 77 6f 47 5a 58 49 76 59 58 64 7a 45 47 38 61 44 46 6e 4c 50 65 33 4a 63 35 62 65 50 74 76 6b 6c 69 4b 43 41 65 33 75 4b 64 70 33 31 54 44 30 78 37 66 63 76 59 31 4a 34 41 41 2f 75 32 41 57 76 41 77 51 70 49 67 6f 6f 2b 4b 52 6d 45 62 56 59 43 6e 4b 6f 6d 4b 38 50 6e 42 48 68 7a 38 36 56 4a 7a 79 59 54 46 66 52 2b 62 30 2b 4b 54 35 5a 47 57 76 75 59 34 45 54 65 32 76 70 5a 43 4b 51 73 51 44 6d 52 74 78 63 59 77 30 77 53 79 4e 4a 69 61 4a 34 63 72 4a 41 34 7a 63 55 6c 7a 30 6f 79 43 2f 79 67 56 6b 33 4b 32 63 6d 65 2f 66 48 30 54 74 38 64 37 56 31 44 64 62 6b 73 49 31 4b 72 52 78 66 44 2f 50 59 6c 30 4e 76 4d 34 35 63 54 38 6f 38 74 4b 4c 69 77 59 79 4b 4f 2b 6d 73 6e 47 38 46 51 41 57 38 70 35 2f 66 4b 57 49 30 51 4e 43 32 38 6a 67 6a 56 45 30 34 47 62 58 45 6a 78 77 31 48 33 64 73 56 4c 4d 54 51 42 31 47 70 73 3d 0a 78 2d 61 6d 7a 2d 75 73 65 72 2d 61 67 65 6e 74 3a 61 77 73 2d 73 64 6b 2d 72 75 73 74 2f 30 2e 31 2e 30 20 61 70 69 2f 73 33 2f 30 2e 30 2e 31 39 2d 61 6c 70 68 61 20 6f 73 2f 6d 61 63 6f 73 20 6c 61 6e 67 2f 72 75 73 74 2f 31 2e 35 35 2e 30 0a 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3b 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 3b 78 2d 61 6d 7a 2d 6d 65 74 61 2d 74 69 74 6c 65 3b 78 2d 61 6d 7a 2d 73 65 63 75 72 69 74 79 2d 74 6f 6b 65 6e 3b 78 2d 61 6d 7a 2d 75 73 65 72 2d 61 67 65 6e 74 0a 32 63 66 32 34 64 62 61 35 66 62 30 61 33 30 65 32 36 65 38 33 62 32 61 63 35 62 39 65 32 39 65 31 62 31 36 31 65 35 63 31 66 61 37 34 32 35 65 37 33 30 34 33 33 36 32 39 33 38 62 39 38 32 34</CanonicalRequestBytes><RequestId>2RYD4W7SB5S3MMKC</RequestId><HostId>djjMKM1bJwU4dBdCsoIcEVFHYw2vGb2EtMw4Z7Fq7DvOE/mFfQDqU1ZaOanFHclRh0P1aIAjZac=</HostId></Error>")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } }', src/bin/bug.rs:11:10
stack backtrace:
   0:        0x104d80ad4 - std::backtrace_rs::backtrace::libunwind::trace::hd9b9204470182ffc
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:        0x104d80ad4 - std::backtrace_rs::backtrace::trace_unsynchronized::hdc48b9180be04761
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:        0x104d80ad4 - std::sys_common::backtrace::_print_fmt::h8824de041faf4112
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:67:5
   3:        0x104d80ad4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8d168763f08a78b9
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:46:22
   4:        0x104d9e44c - core::fmt::write::h0fa46d4891ed63f1
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/fmt/mod.rs:1115:17
   5:        0x104d7b2aa - std::io::Write::write_fmt::hacbf9e10e17149ce
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/io/mod.rs:1665:15
   6:        0x104d8285f - std::sys_common::backtrace::_print::h26213154a400cfbe
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:49:5
   7:        0x104d8285f - std::sys_common::backtrace::print::h0f4ab60f53d1f93b
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:36:9
   8:        0x104d8285f - std::panicking::default_hook::{{closure}}::hee7cf38dedfa76fc
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:208:50
   9:        0x104d8235d - std::panicking::default_hook::hd12252412832a45f
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:225:9
  10:        0x104d82f60 - std::panicking::rust_panic_with_hook::h521fd09e07057893
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:622:17
  11:        0x104d82a05 - std::panicking::begin_panic_handler::{{closure}}::h3588a5a9cca9e8e5
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:519:13
  12:        0x104d80f78 - std::sys_common::backtrace::__rust_end_short_backtrace::h86522c151a195ff9
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:141:18
  13:        0x104d8296a - rust_begin_unwind
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
  14:        0x104db039f - core::panicking::panic_fmt::h8ab5bf5f27d0a26a
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
  15:        0x104db0485 - core::result::unwrap_failed::ha249df8fad19d4e8
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/result.rs:1599:5
  16:        0x1044e916a - core::result::Result<T,E>::unwrap::h0c65e654c90f892a
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/result.rs:1281:23
  17:        0x1044d39d3 - bug::main::{{closure}}::hc469cb26ae58d822
                               at /Users/mathias/devel/projects/dilbert-feed/src/bin/bug.rs:3:5
  18:        0x1044dbf10 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hc5c4e63963a7b68d
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/future/mod.rs:80:19
  19:        0x1044cfbba - tokio::park::thread::CachedParkThread::block_on::{{closure}}::h4d99078977019589
                               at /Users/mathias/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.12.0/src/park/thread.rs:263:54
  20:        0x1044d4289 - tokio::coop::with_budget::{{closure}}::h43c273399f8a156d
                               at /Users/mathias/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.12.0/src/coop.rs:106:9
  21:        0x1044dd5d9 - std::thread::local::LocalKey<T>::try_with::h19213e27fe909076
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/thread/local.rs:399:16
  22:        0x1044dd40c - std::thread::local::LocalKey<T>::with::hc0f29a262cc8e7b1
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/thread/local.rs:375:9
  23:        0x1044cf9fa - tokio::coop::with_budget::h8cc039aa7be4b57e
                               at /Users/mathias/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.12.0/src/coop.rs:99:5
  24:        0x1044cf9fa - tokio::coop::budget::h23576021aa847f56
                               at /Users/mathias/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.12.0/src/coop.rs:76:5
  25:        0x1044cf9fa - tokio::park::thread::CachedParkThread::block_on::hcd7fd8bd35625dce
                               at /Users/mathias/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.12.0/src/park/thread.rs:263:31
  26:        0x1044e09e1 - tokio::runtime::enter::Enter::block_on::h11b832b622ba3cdd
                               at /Users/mathias/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.12.0/src/runtime/enter.rs:151:13
  27:        0x1044d0226 - tokio::runtime::thread_pool::ThreadPool::block_on::h0aca672ec27fe7de
                               at /Users/mathias/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.12.0/src/runtime/thread_pool/mod.rs:77:9
  28:        0x1044daa83 - tokio::runtime::Runtime::block_on::hd72f7e9fc69057ea
                               at /Users/mathias/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.12.0/src/runtime/mod.rs:463:43
  29:        0x1044d710c - bug::main::h7c608f44b2172786
                               at /Users/mathias/devel/projects/dilbert-feed/src/bin/bug.rs:3:5
  30:        0x1044e4e6e - core::ops::function::FnOnce::call_once::hf1f47b605a607abd
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/ops/function.rs:227:5
  31:        0x1044d44e1 - std::sys_common::backtrace::__rust_begin_short_backtrace::h7525043c864d00dc
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:125:18
  32:        0x1044d4d14 - std::rt::lang_start::{{closure}}::hb5431c90f1b746f1
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:63:18
  33:        0x104d83489 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h5adc1669b25f044b
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/ops/function.rs:259:13
  34:        0x104d83489 - std::panicking::try::do_call::h405f88a3fec16587
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
  35:        0x104d83489 - std::panicking::try::h86d9af92fb3c6022
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
  36:        0x104d83489 - std::panic::catch_unwind::h67150880d2b87901
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
  37:        0x104d83489 - std::rt::lang_start_internal::{{closure}}::h548e43d03023c928
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:48
  38:        0x104d83489 - std::panicking::try::do_call::h1383affbfec23244
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
  39:        0x104d83489 - std::panicking::try::hf51149bda74e8587
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
  40:        0x104d83489 - std::panic::catch_unwind::h8280208a4f5e7cf4
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
  41:        0x104d83489 - std::rt::lang_start_internal::h5f9b030f0a63c040
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:20
  42:        0x1044d4cde - std::rt::lang_start::h3ed2256927ced8b2
                               at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:62:5
  43:        0x1044d7186 - _main

Version

v0.0.19-alpha

[Velfi]

Thanks for finding this, I'll look into and let you know when it's fixed

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[rcoh]

keeping this open until we land this in the next release

[Velfi]

that dang autoclose

[rcoh]

fix shipped in 0.0.23-alpha

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.