SES V1/2 Reliability Issues #844

AnBowell · 2023-07-10T13:32:49Z

Describe the bug

I have a lambda function which I would like to use to send an email via SES.

Following the example in this repo, I achieved this functionality; though, around 50% of the time the emails fail to go through and are reported as a "dispatch failure" or upon further investigation, an HTTP response timeout.

My AWS account has been approved with a sending limit of 50k emails per day / 14 per second, and it can sometimes fail on the first email of the day, so I don't believe it to be a throttling problem. No bounces or failures are reported in the SES console.

I have found a very similar issue reported here: knadh/listmonk#81

Expected Behavior

Emails sends are successful 100% of the time.

Current Behavior

Emails are only successfully sent about 50% of the time.

Reproduction Steps

use aws_sdk_ses::Client as SesClient;
use lambda_http::Error as LambdaError;

#[tokio::main]
async fn main() -> Result<(), LambdaError> {
let shared_config = aws_config::load_from_env().await;
let ses_client = SesClient::new(&shared_config);
let ses_client_ref = &ses_client;
let func = service_fn(move |event| async move {
    function_handler(
        event,
        ses_client_ref,
    )
    .await
});
run(func).await
}

async fn function_handler(
    event: Request,
    ses_client: &SesClient,
) -> Result<Response<String>, Error> {

    let email_res = client
        .send_email()
        .set_source(Some("email_here".to_string()))
        .destination("dest_here")
        .message("msg_here") 
        .send()
        .await.unwrap();

    // Have taken out the return.
}

Possible Solution

No response

Additional Information/Context

No response

Version

├── aws-config v0.55.3
│   ├── aws-credential-types v0.55.3
│   │   ├── aws-smithy-async v0.55.3
│   │   │   ├── futures-util v0.3.28
│   │   │   │   ├── futures-channel v0.3.28
│   │   │   │   │   ├── futures-core v0.3.28
│   │   │   │   │   └── futures-sink v0.3.28
│   │   │   │   ├── futures-core v0.3.28
│   │   │   │   ├── futures-io v0.3.28
│   │   │   │   ├── futures-macro v0.3.28 (proc-macro)
│   │   │   │   │   ├── proc-macro2 v1.0.63
│   │   │   │   │   │   └── unicode-ident v1.0.10     
│   │   │   │   │   ├── quote v1.0.29
│   │   │   │   │   │   └── proc-macro2 v1.0.63 (*)
│   │   │   │   │   └── syn v2.0.23
│   │   │   │   │       ├── proc-macro2 v1.0.63 (*)
│   │   │   │   │       ├── quote v1.0.29 (*)
│   │   │   │   │       └── unicode-ident v1.0.10
│   │   │   │   ├── futures-sink v0.3.28
│   │   │   │   ├── futures-task v0.3.28
│   │   │   │   ├── memchr v2.5.0
│   │   │   │   ├── pin-project-lite v0.2.10
│   │   │   │   ├── pin-utils v0.1.0
│   │   │   │   └── slab v0.4.8
│   │   │   │       [build-dependencies]
│   │   │   │       └── autocfg v1.1.0
│   │   │   ├── pin-project-lite v0.2.10
│   │   │   ├── tokio v1.29.1
│   │   │   │   ├── bytes v1.4.0
│   │   │   │   │   └── serde v1.0.167
│   │   │   │   │       └── serde_derive v1.0.167 (proc-macro)
│   │   │   │   │           ├── proc-macro2 v1.0.63 (*)
│   │   │   │   │           ├── quote v1.0.29 (*)
│   │   │   │   │           └── syn v2.0.23 (*)
│   │   │   │   ├── mio v0.8.8
│   │   │   │   │   └── windows-sys v0.48.0
│   │   │   │   │       └── windows-targets v0.48.1
│   │   │   │   │           └── windows_x86_64_msvc v0.48.0
│   │   │   │   ├── num_cpus v1.16.0
│   │   │   │   ├── pin-project-lite v0.2.10
│   │   │   │   ├── socket2 v0.4.9
│   │   │   │   │   └── winapi v0.3.9
│   │   │   │   ├── tokio-macros v2.1.0 (proc-macro)
│   │   │   │   │   ├── proc-macro2 v1.0.63 (*)
│   │   │   │   │   ├── quote v1.0.29 (*)
│   │   │   │   │   └── syn v2.0.23 (*)
│   │   │   │   └── windows-sys v0.48.0 (*)
│   │   │   │   [build-dependencies]
│   │   │   │   └── autocfg v1.1.0
│   │   │   └── tokio-stream v0.1.14
│   │   │       ├── futures-core v0.3.28
│   │   │       ├── pin-project-lite v0.2.10
│   │   │       └── tokio v1.29.1 (*)
│   │   ├── aws-smithy-types v0.55.3
│   │   │   ├── base64-simd v0.8.0
│   │   │   │   ├── outref v0.5.1
│   │   │   │   └── vsimd v0.8.0
│   │   │   ├── itoa v1.0.8
│   │   │   ├── num-integer v0.1.45
│   │   │   │   └── num-traits v0.2.15
│   │   │   │       [build-dependencies]
│   │   │   │       └── autocfg v1.1.0
│   │   │   │   [build-dependencies]
│   │   │   │   └── autocfg v1.1.0
│   │   │   ├── ryu v1.0.14
│   │   │   └── time v0.3.22
│   │   │       └── time-core v0.1.1
│   │   ├── fastrand v1.9.0
│   │   ├── tokio v1.29.1 (*)
│   │   ├── tracing v0.1.37
│   │   │   ├── cfg-if v1.0.0
│   │   │   ├── log v0.4.19
│   │   │   ├── pin-project-lite v0.2.10
│   │   │   ├── tracing-attributes v0.1.26 (proc-macro)
│   │   │   │   ├── proc-macro2 v1.0.63 (*)
│   │   │   │   ├── quote v1.0.29 (*)
│   │   │   │   └── syn v2.0.23 (*)
│   │   │   └── tracing-core v0.1.31
│   │   │       └── once_cell v1.18.0
│   │   └── zeroize v1.6.0
│   ├── aws-http v0.55.3
│   │   ├── aws-credential-types v0.55.3 (*)
│   │   ├── aws-smithy-http v0.55.3
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   ├── bytes v1.4.0 (*)
│   │   │   ├── bytes-utils v0.1.3
│   │   │   │   ├── bytes v1.4.0 (*)
│   │   │   │   └── either v1.8.1
│   │   │   ├── futures-core v0.3.28
│   │   │   ├── http v0.2.9
│   │   │   │   ├── bytes v1.4.0 (*)
│   │   │   │   ├── fnv v1.0.7
│   │   │   │   └── itoa v1.0.8
│   │   │   ├── http-body v0.4.5
│   │   │   │   ├── bytes v1.4.0 (*)
│   │   │   │   ├── http v0.2.9 (*)
│   │   │   │   └── pin-project-lite v0.2.10
│   │   │   ├── hyper v0.14.27
│   │   │   │   ├── bytes v1.4.0 (*)
│   │   │   │   ├── futures-channel v0.3.28 (*)
│   │   │   │   ├── futures-core v0.3.28
│   │   │   │   ├── futures-util v0.3.28 (*)
│   │   │   │   ├── h2 v0.3.20
│   │   │   │   │   ├── bytes v1.4.0 (*)
│   │   │   │   │   ├── fnv v1.0.7
│   │   │   │   │   ├── futures-core v0.3.28
│   │   │   │   │   ├── futures-sink v0.3.28
│   │   │   │   │   ├── futures-util v0.3.28 (*)
│   │   │   │   │   ├── http v0.2.9 (*)
│   │   │   │   │   ├── indexmap v1.9.3
│   │   │   │   │   │   └── hashbrown v0.12.3
│   │   │   │   │   │   [build-dependencies]
│   │   │   │   │   │   └── autocfg v1.1.0
│   │   │   │   │   ├── slab v0.4.8 (*)
│   │   │   │   │   ├── tokio v1.29.1 (*)
│   │   │   │   │   ├── tokio-util v0.7.8
│   │   │   │   │   │   ├── bytes v1.4.0 (*)
│   │   │   │   │   │   ├── futures-core v0.3.28
│   │   │   │   │   │   ├── futures-sink v0.3.28
│   │   │   │   │   │   ├── pin-project-lite v0.2.10
│   │   │   │   │   │   ├── tokio v1.29.1 (*)
│   │   │   │   │   │   └── tracing v0.1.37 (*)
│   │   │   │   │   └── tracing v0.1.37 (*)
│   │   │   │   ├── http v0.2.9 (*)
│   │   │   │   ├── http-body v0.4.5 (*)
│   │   │   │   ├── httparse v1.8.0
│   │   │   │   ├── httpdate v1.0.2
│   │   │   │   ├── itoa v1.0.8
│   │   │   │   ├── pin-project-lite v0.2.10
│   │   │   │   ├── socket2 v0.4.9 (*)
│   │   │   │   ├── tokio v1.29.1 (*)
│   │   │   │   ├── tower-service v0.3.2
│   │   │   │   ├── tracing v0.1.37 (*)
│   │   │   │   └── want v0.3.1
│   │   │   │       └── try-lock v0.2.4
│   │   │   ├── once_cell v1.18.0
│   │   │   ├── percent-encoding v2.3.0
│   │   │   ├── pin-project-lite v0.2.10
│   │   │   ├── pin-utils v0.1.0
│   │   │   ├── tokio v1.29.1 (*)
│   │   │   ├── tokio-util v0.7.8 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-types v0.55.3
│   │   │   ├── aws-credential-types v0.55.3 (*)
│   │   │   ├── aws-smithy-async v0.55.3 (*)
│   │   │   ├── aws-smithy-client v0.55.3
│   │   │   │   ├── aws-smithy-async v0.55.3 (*)
│   │   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.55.3
│   │   │   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   │   │   ├── bytes v1.4.0 (*)
│   │   │   │   │   ├── http v0.2.9 (*)
│   │   │   │   │   ├── http-body v0.4.5 (*)
│   │   │   │   │   ├── pin-project-lite v0.2.10
│   │   │   │   │   ├── tower v0.4.13
│   │   │   │   │   │   ├── futures-core v0.3.28
│   │   │   │   │   │   ├── futures-util v0.3.28 (*)
│   │   │   │   │   │   ├── pin-project v1.1.2
│   │   │   │   │   │   │   └── pin-project-internal v1.1.2 (proc-macro)
│   │   │   │   │   │   │       ├── proc-macro2 v1.0.63 (*)
│   │   │   │   │   │   │       ├── quote v1.0.29 (*)
│   │   │   │   │   │   │       └── syn v2.0.23 (*)
│   │   │   │   │   │   ├── pin-project-lite v0.2.10
│   │   │   │   │   │   ├── tokio v1.29.1 (*)
│   │   │   │   │   │   ├── tower-layer v0.3.2
│   │   │   │   │   │   ├── tower-service v0.3.2
│   │   │   │   │   │   └── tracing v0.1.37 (*)
│   │   │   │   │   └── tracing v0.1.37 (*)
│   │   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   │   ├── bytes v1.4.0 (*)
│   │   │   │   ├── fastrand v1.9.0
│   │   │   │   ├── http v0.2.9 (*)
│   │   │   │   ├── http-body v0.4.5 (*)
│   │   │   │   ├── hyper v0.14.27 (*)
│   │   │   │   ├── hyper-rustls v0.23.2
│   │   │   │   │   ├── http v0.2.9 (*)
│   │   │   │   │   ├── hyper v0.14.27 (*)
│   │   │   │   │   ├── log v0.4.19
│   │   │   │   │   ├── rustls v0.20.8
│   │   │   │   │   │   ├── log v0.4.19
│   │   │   │   │   │   ├── ring v0.16.20
│   │   │   │   │   │   │   ├── spin v0.5.2
│   │   │   │   │   │   │   ├── untrusted v0.7.1
│   │   │   │   │   │   │   └── winapi v0.3.9
│   │   │   │   │   │   │   [build-dependencies]
│   │   │   │   │   │   │   └── cc v1.0.79
│   │   │   │   │   │   ├── sct v0.7.0
│   │   │   │   │   │   │   ├── ring v0.16.20 (*)
│   │   │   │   │   │   │   └── untrusted v0.7.1
│   │   │   │   │   │   └── webpki v0.22.0
│   │   │   │   │   │       ├── ring v0.16.20 (*)
│   │   │   │   │   │       └── untrusted v0.7.1
│   │   │   │   │   ├── rustls-native-certs v0.6.3
│   │   │   │   │   │   ├── rustls-pemfile v1.0.3
│   │   │   │   │   │   │   └── base64 v0.21.2
│   │   │   │   │   │   └── schannel v0.1.22
│   │   │   │   │   │       └── windows-sys v0.48.0 (*)
│   │   │   │   │   ├── tokio v1.29.1 (*)
│   │   │   │   │   └── tokio-rustls v0.23.4
│   │   │   │   │       ├── rustls v0.20.8 (*)
│   │   │   │   │       ├── tokio v1.29.1 (*)
│   │   │   │   │       └── webpki v0.22.0 (*)
│   │   │   │   ├── lazy_static v1.4.0
│   │   │   │   ├── pin-project-lite v0.2.10
│   │   │   │   ├── rustls v0.20.8 (*)
│   │   │   │   ├── tokio v1.29.1 (*)
│   │   │   │   ├── tower v0.4.13 (*)
│   │   │   │   └── tracing v0.1.37 (*)
│   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   ├── http v0.2.9 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   │   [build-dependencies]
│   │   │   └── rustc_version v0.4.0
│   │   │       └── semver v1.0.17
│   │   ├── bytes v1.4.0 (*)
│   │   ├── http v0.2.9 (*)
│   │   ├── http-body v0.4.5 (*)
│   │   ├── lazy_static v1.4.0
│   │   ├── percent-encoding v2.3.0
│   │   ├── pin-project-lite v0.2.10
│   │   └── tracing v0.1.37 (*)
│   ├── aws-sdk-sso v0.28.0
│   │   ├── aws-credential-types v0.55.3 (*)
│   │   ├── aws-endpoint v0.55.3
│   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   ├── aws-types v0.55.3 (*)
│   │   │   ├── http v0.2.9 (*)
│   │   │   ├── regex v1.9.0
│   │   │   │   ├── aho-corasick v1.0.2
│   │   │   │   │   └── memchr v2.5.0
│   │   │   │   ├── memchr v2.5.0
│   │   │   │   ├── regex-automata v0.3.0
│   │   │   │   │   ├── aho-corasick v1.0.2 (*)
│   │   │   │   │   ├── memchr v2.5.0
│   │   │   │   │   └── regex-syntax v0.7.3
│   │   │   │   └── regex-syntax v0.7.3
│   │   │   └── tracing v0.1.37 (*)
│   │   ├── aws-http v0.55.3 (*)
│   │   ├── aws-sig-auth v0.55.3
│   │   │   ├── aws-credential-types v0.55.3 (*)
│   │   │   ├── aws-sigv4 v0.55.3
│   │   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   │   ├── form_urlencoded v1.2.0
│   │   │   │   │   └── percent-encoding v2.3.0
│   │   │   │   ├── hex v0.4.3
│   │   │   │   ├── hmac v0.12.1
│   │   │   │   │   └── digest v0.10.7
│   │   │   │   │       ├── block-buffer v0.10.4
│   │   │   │   │       │   └── generic-array v0.14.7
│   │   │   │   │       │       └── typenum v1.16.0
│   │   │   │   │       │       [build-dependencies]
│   │   │   │   │       │       └── version_check v0.9.4
│   │   │   │   │       ├── crypto-common v0.1.6
│   │   │   │   │       │   ├── generic-array v0.14.7 (*)
│   │   │   │   │       │   └── typenum v1.16.0
│   │   │   │   │       └── subtle v2.5.0
│   │   │   │   ├── http v0.2.9 (*)
│   │   │   │   ├── once_cell v1.18.0
│   │   │   │   ├── percent-encoding v2.3.0
│   │   │   │   ├── regex v1.9.0 (*)
│   │   │   │   ├── sha2 v0.10.7
│   │   │   │   │   ├── cfg-if v1.0.0
│   │   │   │   │   ├── cpufeatures v0.2.9
│   │   │   │   │   └── digest v0.10.7 (*)
│   │   │   │   ├── time v0.3.22 (*)
│   │   │   │   └── tracing v0.1.37 (*)
│   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   ├── aws-types v0.55.3 (*)
│   │   │   ├── http v0.2.9 (*)
│   │   │   └── tracing v0.1.37 (*)
│   │   ├── aws-smithy-async v0.55.3 (*)
│   │   ├── aws-smithy-client v0.55.3 (*)
│   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-smithy-http-tower v0.55.3 (*)
│   │   ├── aws-smithy-json v0.55.3
│   │   │   └── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-types v0.55.3 (*)
│   │   ├── bytes v1.4.0 (*)
│   │   ├── http v0.2.9 (*)
│   │   ├── regex v1.9.0 (*)
│   │   ├── tokio-stream v0.1.14 (*)
│   │   ├── tower v0.4.13 (*)
│   │   └── tracing v0.1.37 (*)
│   ├── aws-sdk-sts v0.28.0
│   │   ├── aws-credential-types v0.55.3 (*)
│   │   ├── aws-endpoint v0.55.3 (*)
│   │   ├── aws-http v0.55.3 (*)
│   │   ├── aws-sig-auth v0.55.3 (*)
│   │   ├── aws-smithy-async v0.55.3 (*)
│   │   ├── aws-smithy-client v0.55.3 (*)
│   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-smithy-http-tower v0.55.3 (*)
│   │   ├── aws-smithy-json v0.55.3 (*)
│   │   ├── aws-smithy-query v0.55.3
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   └── urlencoding v2.1.2
│   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-smithy-xml v0.55.3
│   │   │   └── xmlparser v0.13.5
│   │   ├── aws-types v0.55.3 (*)
│   │   ├── bytes v1.4.0 (*)
│   │   ├── http v0.2.9 (*)
│   │   ├── regex v1.9.0 (*)
│   │   ├── tower v0.4.13 (*)
│   │   └── tracing v0.1.37 (*)
│   ├── aws-smithy-async v0.55.3 (*)
│   ├── aws-smithy-client v0.55.3 (*)
│   ├── aws-smithy-http v0.55.3 (*)
│   ├── aws-smithy-http-tower v0.55.3 (*)
│   ├── aws-smithy-json v0.55.3 (*)
│   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-types v0.55.3 (*)
│   ├── bytes v1.4.0 (*)
│   ├── fastrand v1.9.0
│   ├── hex v0.4.3
│   ├── http v0.2.9 (*)
│   ├── hyper v0.14.27 (*)
│   ├── ring v0.16.20 (*)
│   ├── time v0.3.22 (*)
│   ├── tokio v1.29.1 (*)
│   ├── tower v0.4.13 (*)
│   ├── tracing v0.1.37 (*)
│   └── zeroize v1.6.0
├── aws-sdk-lambda v0.28.0
│   ├── aws-credential-types v0.55.3 (*)
│   ├── aws-endpoint v0.55.3 (*)
│   ├── aws-http v0.55.3 (*)
│   ├── aws-sig-auth v0.55.3 (*)
│   ├── aws-smithy-async v0.55.3 (*)
│   ├── aws-smithy-client v0.55.3 (*)
│   ├── aws-smithy-http v0.55.3 (*)
│   ├── aws-smithy-http-tower v0.55.3 (*)
│   ├── aws-smithy-json v0.55.3 (*)
│   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-types v0.55.3 (*)
│   ├── bytes v1.4.0 (*)
│   ├── http v0.2.9 (*)
│   ├── regex v1.9.0 (*)
│   ├── tokio-stream v0.1.14 (*)
│   ├── tower v0.4.13 (*)
│   └── tracing v0.1.37 (*)
├── aws-sdk-ses v0.28.0
│   ├── aws-credential-types v0.55.3 (*)
│   ├── aws-endpoint v0.55.3 (*)
│   ├── aws-http v0.55.3 (*)
│   ├── aws-sig-auth v0.55.3 (*)
│   ├── aws-smithy-async v0.55.3 (*)
│   ├── aws-smithy-client v0.55.3 (*)
│   ├── aws-smithy-http v0.55.3 (*)
│   ├── aws-smithy-http-tower v0.55.3 (*)
│   ├── aws-smithy-json v0.55.3 (*)
│   ├── aws-smithy-query v0.55.3 (*)
│   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-smithy-xml v0.55.3 (*)
│   ├── aws-types v0.55.3 (*)
│   ├── bytes v1.4.0 (*)
│   ├── http v0.2.9 (*)
│   ├── regex v1.9.0 (*)
│   ├── tokio-stream v0.1.14 (*)
│   ├── tower v0.4.13 (*)
│   └── tracing v0.1.37 (*)
├── http v0.2.9 (*)
├── lambda_http v0.8.1
│   ├── aws_lambda_events v0.10.0
│   │   ├── base64 v0.21.2
│   │   ├── bytes v1.4.0 (*)
│   │   ├── http v0.2.9 (*)
│   │   ├── http-body v0.4.5 (*)
│   │   ├── http-serde v1.1.2
│   │   │   ├── http v0.2.9 (*)
│   │   │   └── serde v1.0.167 (*)
│   │   ├── query_map v0.6.0
│   │   │   ├── form_urlencoded v1.2.0 (*)
│   │   │   ├── serde v1.0.167 (*)
│   │   │   └── serde_derive v1.0.167 (proc-macro) (*)
│   │   ├── serde v1.0.167 (*)
│   │   └── serde_json v1.0.100
│   │       ├── itoa v1.0.8
│   │       ├── ryu v1.0.14
│   │       └── serde v1.0.167 (*)
│   ├── base64 v0.21.2
│   ├── bytes v1.4.0 (*)
│   ├── encoding_rs v0.8.32
│   │   └── cfg-if v1.0.0
│   ├── futures v0.3.28
│   │   ├── futures-channel v0.3.28 (*)
│   │   ├── futures-core v0.3.28
│   │   ├── futures-executor v0.3.28
│   │   │   ├── futures-core v0.3.28
│   │   │   ├── futures-task v0.3.28
│   │   │   └── futures-util v0.3.28 (*)
│   │   ├── futures-io v0.3.28
│   │   ├── futures-sink v0.3.28
│   │   ├── futures-task v0.3.28
│   │   └── futures-util v0.3.28 (*)
│   ├── http v0.2.9 (*)
│   ├── http-body v0.4.5 (*)
│   ├── hyper v0.14.27 (*)
│   ├── lambda_runtime v0.8.1
│   │   ├── async-stream v0.3.5
│   │   │   ├── async-stream-impl v0.3.5 (proc-macro)
│   │   │   │   ├── proc-macro2 v1.0.63 (*)
│   │   │   │   ├── quote v1.0.29 (*)
│   │   │   │   └── syn v2.0.23 (*)
│   │   │   ├── futures-core v0.3.28
│   │   │   └── pin-project-lite v0.2.10
│   │   ├── bytes v1.4.0 (*)
│   │   ├── futures v0.3.28 (*)
│   │   ├── http v0.2.9 (*)
│   │   ├── hyper v0.14.27 (*)
│   │   ├── lambda_runtime_api_client v0.8.0
│   │   │   ├── http v0.2.9 (*)
│   │   │   ├── hyper v0.14.27 (*)
│   │   │   ├── tokio v1.29.1 (*)
│   │   │   └── tower-service v0.3.2
│   │   ├── serde v1.0.167 (*)
│   │   ├── serde_json v1.0.100 (*)
│   │   ├── serde_path_to_error v0.1.13
│   │   │   ├── itoa v1.0.8
│   │   │   └── serde v1.0.167 (*)
│   │   ├── tokio v1.29.1 (*)
│   │   ├── tokio-stream v0.1.14 (*)
│   │   ├── tower v0.4.13 (*)
│   │   └── tracing v0.1.37 (*)
│   ├── mime v0.3.17
│   ├── percent-encoding v2.3.0
│   ├── serde v1.0.167 (*)
│   ├── serde_json v1.0.100 (*)
│   ├── serde_urlencoded v0.7.1
│   │   ├── form_urlencoded v1.2.0 (*)
│   │   ├── itoa v1.0.8
│   │   ├── ryu v1.0.14
│   │   └── serde v1.0.167 (*)
│   └── url v2.4.0
│       ├── form_urlencoded v1.2.0 (*)
│       ├── idna v0.4.0
│       │   ├── unicode-bidi v0.3.13
│       │   └── unicode-normalization v0.1.22
│       │       └── tinyvec v1.6.0
│       │           └── tinyvec_macros v0.1.1
│       └── percent-encoding v2.3.0
├── postgres-types v0.2.5
│   ├── bytes v1.4.0 (*)
│   ├── chrono v0.4.26
│   │   ├── num-traits v0.2.15 (*)
│   │   ├── serde v1.0.167 (*)
│   │   └── winapi v0.3.9
│   ├── fallible-iterator v0.2.0
│   ├── postgres-derive v0.4.4 (proc-macro)
│   │   ├── proc-macro2 v1.0.63 (*)
│   │   ├── quote v1.0.29 (*)
│   │   └── syn v2.0.23 (*)
│   ├── postgres-protocol v0.6.5
│   │   ├── base64 v0.21.2
│   │   ├── byteorder v1.4.3
│   │   ├── bytes v1.4.0 (*)
│   │   ├── fallible-iterator v0.2.0
│   │   ├── hmac v0.12.1 (*)
│   │   ├── md-5 v0.10.5
│   │   │   └── digest v0.10.7 (*)
│   │   ├── memchr v2.5.0
│   │   ├── rand v0.8.5
│   │   │   ├── rand_chacha v0.3.1
│   │   │   │   ├── ppv-lite86 v0.2.17
│   │   │   │   └── rand_core v0.6.4
│   │   │   │       └── getrandom v0.2.10
│   │   │   │           └── cfg-if v1.0.0
│   │   │   └── rand_core v0.6.4 (*)
│   │   ├── sha2 v0.10.7 (*)
│   │   └── stringprep v0.1.2
│   │       ├── unicode-bidi v0.3.13
│   │       └── unicode-normalization v0.1.22 (*)
│   ├── serde v1.0.167 (*)
│   └── serde_json v1.0.100 (*)
├── serde v1.0.167 (*)
├── serde_json v1.0.100 (*)
├── serde_with v2.3.3
│   ├── chrono v0.4.26 (*)
│   ├── serde v1.0.167 (*)
│   └── serde_with_macros v2.3.3 (proc-macro)
│       ├── darling v0.20.1
│       │   ├── darling_core v0.20.1
│       │   │   ├── fnv v1.0.7
│       │   │   ├── ident_case v1.0.1
│       │   │   ├── proc-macro2 v1.0.63 (*)
│       │   │   ├── quote v1.0.29 (*)
│       │   │   ├── strsim v0.10.0
│       │   │   └── syn v2.0.23 (*)
│       │   └── darling_macro v0.20.1 (proc-macro)
│       │       ├── darling_core v0.20.1 (*)
│       │       ├── quote v1.0.29 (*)
│       │       └── syn v2.0.23 (*)
│       ├── proc-macro2 v1.0.63 (*)
│       ├── quote v1.0.29 (*)
│       └── syn v2.0.23 (*)
├── strum v0.25.0
├── strum_macros v0.25.1 (proc-macro)
│   ├── heck v0.4.1
│   ├── proc-macro2 v1.0.63 (*)
│   ├── quote v1.0.29 (*)
│   ├── rustversion v1.0.13 (proc-macro)
│   └── syn v2.0.23 (*)
├── tokio v1.29.1 (*)
├── tokio-postgres v0.7.8
│   ├── async-trait v0.1.71 (proc-macro)
│   │   ├── proc-macro2 v1.0.63 (*)
│   │   ├── quote v1.0.29 (*)
│   │   └── syn v2.0.23 (*)
│   ├── byteorder v1.4.3
│   ├── bytes v1.4.0 (*)
│   ├── fallible-iterator v0.2.0
│   ├── futures-channel v0.3.28 (*)
│   ├── futures-util v0.3.28 (*)
│   ├── log v0.4.19
│   ├── parking_lot v0.12.1
│   │   ├── lock_api v0.4.10
│   │   │   └── scopeguard v1.1.0
│   │   │   [build-dependencies]
│   │   │   └── autocfg v1.1.0
│   │   └── parking_lot_core v0.9.8
│   │       ├── cfg-if v1.0.0
│   │       ├── smallvec v1.11.0
│   │       └── windows-targets v0.48.1 (*)
│   ├── percent-encoding v2.3.0
│   ├── phf v0.11.2
│   │   └── phf_shared v0.11.2
│   │       └── siphasher v0.3.10
│   ├── pin-project-lite v0.2.10
│   ├── postgres-protocol v0.6.5 (*)
│   ├── postgres-types v0.2.5 (*)
│   ├── socket2 v0.5.3
│   │   └── windows-sys v0.48.0 (*)
│   ├── tokio v1.29.1 (*)
│   └── tokio-util v0.7.8 (*)
├── tracing v0.1.37 (*)
└── tracing-subscriber v0.3.17
    ├── sharded-slab v0.1.4
    │   └── lazy_static v1.4.0
    ├── thread_local v1.1.7
    │   ├── cfg-if v1.0.0
    │   └── once_cell v1.18.0
    └── tracing-core v0.1.31 (*)

Environment details (OS name and version, etc.)

Windows 10 - cargo-lambda 0.19.3 (2023-06-07Z)

Logs

Internal message | Cannot send email for some reason: dispatch failure

The text was updated successfully, but these errors were encountered:

Velfi · 2023-07-10T14:20:11Z

Hey @AnBowell, thanks for submitting this bug. Can you enable ~~trace~~ debug logging, reproduce the issue, and then post the logs here? That'll help us better understand the issue.

AnBowell · 2023-07-11T10:45:52Z

No worries, thanks for the super quick response!

So at the top level, I get "dispatch failure".

With the debug/trace logging enabled using tracing_subscriber I also see this:

DEBUG ...:send_operation{operation="SendTemplatedEmail" service="ses" status="ok"}:dispatch:Connection{peer=Client}:poll: h2::proto::connection: Connection::poll; connection error error=GoAway(b"", NO_ERROR, Library).

TRACE ...:send_operation{operation="SendTemplatedEmail" service="ses"}: aws_smithy_client::poison: No smithy connection found! The underlying HTTP connection never set a connection.

TRACE ...:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: checkout dropped for ("https", email.eu-west-2.amazonaws.com)

TRACE ...:send_operation{operation="SendTemplatedEmail" service="ses"}: aws_smithy_client::retry: retry action retry=Some((RetryHandler { local: RequestLocalRetryState { attempts: 3, last_quota_usage: Some(10) }, shared: CrossRequestRetryState { quota_available: Mutex { data: 460, poisoned: false, .. } }, config: Config { initial_retry_tokens: 500, retry_cost: 5, no_retry_increment: 1, timeout_retry_cost: 10, max_attempts: 3, initial_backoff: 1s, max_backoff: 20s, base: 0xaaaaca6ce730 }, sleep_impl: Some(TokioSleep) }, 1.833846278s)) retry_kind=Error(TransientError)

I've now also matched on the error and printed that too.

:send_operation{operation="SendTemplatedEmail" service="ses"}: aws_smithy_client::retry: retry classification retry_kind=Error(TransientError)

Velfi · 2023-07-11T15:55:09Z

I think the core thing to establish is this:

Timeouts are getting retried
Timeouts are considered a transient error so the retry strategy should be requesting a new connection when retrying them.
The new connections should be unlikely to run into the same problem. Is that the case or not?
What happens if request timeout time is set to something really low (< 1s) and max_retry_attempts is increased?

You can set timeouts and retry behavior on the sdk config if you use from_env().retry_config(...).timeout_config(...).load(). Let me know if that reveals anything interesting in the logs. When trace logging, hyper should note that it's throwing away the old connection and using a new one.

I'd like to look further into this myself but my week is rather busy. Hopefully the above tips will help reveal the real issue, but if they don't, I'll test this myself as soon as I'm able.

rcoh · 2023-07-11T18:19:22Z

Ah—I've seen this GoAway error before. I had a really hard time reproducing it. Do you see the retries happening? Does it fail even after retries? Do you see logs re-establishing the connection in Hyper?

AnBowell · 2023-07-13T13:53:31Z

Thanks again for the quick responses. I tried a new test of sending four emails, about a minute apart using the settings @Velfi recommended.

I decreased the operation timeout to 200 milliseconds and increased the number of retries to 10. I also tried the with and without the "ReuseAllConnections" Reconnect mode. With these settings I possibly got some interesting logs.

On the second email, I could see Hyper waiting for the connection:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: checkout waiting for idle connection: ("https", email.eu-west-2.amazonaws.com)

then attempting to connect:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("email.eu-west-2.amazonaws.com"), port=None

and then pooling:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: pooling idle connection for ("https", email.eu-west-2.amazonaws.com)

and then dropping the connection:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: checkout dropped for ("https", email.eu-west-2.amazonaws.com)

which then resulted in a timeout error:
:send_email: TimeoutError TimeoutError { source: RequestTimeoutError { kind: "operation timeout (all attempts including retries)", duration: 200ms } }

However, on the third email I first got an attempt to take the connection:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: take? ("https", email.eu-west-2.amazonaws.com): expiration = Some(90s)

a connection was then closed:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: removing closed connection for ("https", email.eu-west-2.amazonaws.com)

and then checked out:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: checkout waiting for idle connection: ("https", email.eu-west-2.amazonaws.com)

and then a connection request:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("email.eu-west-2.amazonaws.com"), port=None

pooling:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: pooling idle connection for ("https", email.eu-west-2.amazonaws.com)

dropped:
:send_operation{operation="SendTemplatedEmail" service="ses"}:dispatch: hyper::client::pool: checkout dropped for ("https", email.eu-west-2.amazonaws.com)

but then a successfully parsed response:
:send_operation{operation="SendTemplatedEmail" service="ses"}:load_response:parse_loaded: aws_sdk_ses::operation::send_templated_email: request_id=Some("<request_id_was_here>")

I can't seem to see in the logs where requests are being retried at all. As for the GoAway error, it seems really transient and I can't reproduce it with every run of the test either.

I've managed to get a better success rate by upping the operation timeout to 10 seconds (from 1s) and giving it 10 maximum retry attempts. Is it possible it's just not connecting within the default timeout duration? When emails are to be sent from the Lambda, I see they run for much longer than I'd expect.

For now I hope this "solution" holds up, sorry I can't be more helpful, this is my first time using the SES crate.

rcoh · 2023-08-08T17:09:05Z

Thanks for the details logs. We're going to try to reproduce this on our side and see what we can do. The issue you linked seems probably a red herring—I think they're using the SMTP API instead of the HTTP API for SES. It seems like there is a likely bug in the connection management somewhere.

johnm · 2023-11-21T01:58:54Z

Thanks for the details logs. We're going to try to reproduce this on our side and see what we can do. The issue you linked seems probably a red herring—I think they're using the SMTP API instead of the HTTP API for SES. It seems like there is a likely bug in the connection management somewhere.

Were you able to figure out the culprit here? I'm getting consistent "dispatch failure" errors too when I'm trying to start instances. It's happening from one ec2 instance but when I try it from my laptop it's working fine.

jdisanti · 2023-12-05T01:07:17Z

The GOAWAY error is being addressed in #738 and will be fixed in an upcoming release.

jdisanti · 2023-12-06T00:26:04Z

The fix for GOAWAY retries has been released.

rcoh · 2024-01-24T19:20:39Z

Has anyone run into this on the latest code? I'm going to close this for now, but feel free to reopen if you hit this issue!

github-actions · 2024-01-24T19:20:55Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

AnBowell added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jul 10, 2023

Velfi removed the needs-triage This issue or PR still needs to be triaged. label Jul 10, 2023

rcoh added p2 This is a standard priority issue needs-reproduction This issue needs reproduction. labels Aug 8, 2023

polvorin mentioned this issue Dec 12, 2023

update aws-sdk-kms version to v1.5.0 build-trust/ockam#7119

Merged

rcoh closed this as completed Jan 24, 2024

jszwedko mentioned this issue Apr 18, 2024

S3/SQS source events fails sporadically vectordotdev/vector#20314

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SES V1/2 Reliability Issues #844

SES V1/2 Reliability Issues #844

AnBowell commented Jul 10, 2023 •

edited

Velfi commented Jul 10, 2023 •

edited

AnBowell commented Jul 11, 2023 •

edited

Velfi commented Jul 11, 2023

rcoh commented Jul 11, 2023

AnBowell commented Jul 13, 2023

rcoh commented Aug 8, 2023

johnm commented Nov 21, 2023

jdisanti commented Dec 5, 2023

jdisanti commented Dec 6, 2023

rcoh commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

SES V1/2 Reliability Issues #844

SES V1/2 Reliability Issues #844

Comments

AnBowell commented Jul 10, 2023 • edited

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

Version

Environment details (OS name and version, etc.)

Logs

Velfi commented Jul 10, 2023 • edited

AnBowell commented Jul 11, 2023 • edited

Velfi commented Jul 11, 2023

rcoh commented Jul 11, 2023

AnBowell commented Jul 13, 2023

rcoh commented Aug 8, 2023

johnm commented Nov 21, 2023

jdisanti commented Dec 5, 2023

jdisanti commented Dec 6, 2023

rcoh commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

⚠️COMMENT VISIBILITY WARNING⚠️

AnBowell commented Jul 10, 2023 •

edited

Velfi commented Jul 10, 2023 •

edited

AnBowell commented Jul 11, 2023 •

edited