Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error during dispatch: connection closed before message completed #1766

Closed
lucdew opened this issue Jun 4, 2020 · 2 comments
Closed

Error during dispatch: connection closed before message completed #1766

lucdew opened this issue Jun 4, 2020 · 2 comments

Comments

@lucdew
Copy link

lucdew commented Jun 4, 2020 •
edited
Loading

Hi,
I get the following error with Rusoto 0.44 (Linux ubuntu LTS 20.04 x64, 5.4.0-33-generic kernel)
Error during dispatch: connection closed before message completed
My code runs in a tokio 0.2.2 async runtime and performs STS, ECS requests using the same rusoto_core::HttpClient. Meaning that throug reference counting (Arc) the same httpClient is passed to multiple ECSClients peforming requests concurrently.

The code is available here (https://github.com/lucdew/rusoto-example) and especially the httpClient that is created here https://github.com/lucdew/rusoto-example/blob/master/src/client.rs

It occurs when concurrency is increased (up to 28req/s) with 4 ecs clients.

It is a issue with the way I use the httpClient (RequestDispatcher) that is unsafe to use in different AWS services clients ?

It could be the server closing the connection (when I test I don't go through a proxy), I could try to capture the https traffic to see. But it does not occur when I limit the number of ecs clients to 2 for intance.

Thanks in advance
@lucdew
Copy link
Author

lucdew commented Jun 5, 2020 •
edited
Loading

Ok I am closing the issue.
It is an hyper "issue" and is related to the server closing an idle connection while at the same time the client tries to reuse and write on it hyperium/hyper#2136
I decreased the connection idle timeout on the client to 15s, still did not fix the issue (but it was less frequent), but disabling pooling on the client solve the issue (but degrades performance obviously)

@lucdew lucdew closed this as completed Jun 5, 2020
@silverjam
Copy link

Probably related to #1686

ricoLv added a commit to ricoLv/tonic that referenced this issue Mar 17, 2022
@ricoLv
fix pool_max_idle_per_host
ea64eaa 
// disabling due to connection closed issue
    build.pool_max_idle_per_host(0);
    /* still getting connection closed before message completed
     * rusoto/rusoto#1766
   */
@yorhodes yorhodes mentioned this issue Jun 7, 2023
Merged
yorhodes added a commit to hyperlane-xyz/hyperlane-monorepo that referenced this issue Jun 8, 2023
@yorhodes @mattiekat
Implement single signer task in validator (#2342)
a93daec 
### Description

Implements singleton signer which allows multiple tasks to use a shared
signer that is signing on a single task and migrates validator to use a
singleton KMS signer

### Drive-by changes

Modifies base agent to allow self to be mutable in run

### Related issues

- Fixes error from sharing KMS signer connection across validator tasks
rusoto/rusoto#1766

### Backward compatibility

Yes

### Testing

E2E Tests

---------

Co-authored-by: Mattie Conover <git@mconover.dev>
@daniel-savu daniel-savu mentioned this issue Feb 19, 2024
Merged
daniel-savu added a commit to hyperlane-xyz/hyperlane-monorepo that referenced this issue Feb 19, 2024
@daniel-savu
fix: lower rusoto timeout to 15s (#3283)
1fdd619 
### Description

Applies the fix in
#2384 everywhere
an `HttpClient` is constructed via rusoto.

It lowers the S3 timeout to 15s based on tips in [this
thread](hyperium/hyper#2136 (comment)),
to avoid `Error during dispatch: connection closed before message
completed` errors. Note that we'll probably still run into these issues,
but less frequently
([source](rusoto/rusoto#1766 (comment))).


### Drive-by changes

<!--
Are there any minor or drive-by changes also included?
-->

### Related issues

<!--
- Fixes #[issue number here]
-->

### Backward compatibility

<!--
Are these changes backward compatible? Are there any infrastructure
implications, e.g. changes that would prohibit deploying older commits
using this infra tooling?

Yes/No
-->

### Testing

<!--
What kind of testing have these changes undergone?

None/Manual/Unit Tests
-->
ltyu pushed a commit to ltyu/hyperlane-monorepo that referenced this issue Mar 13, 2024
@daniel-savu @ltyu
fix: lower rusoto timeout to 15s (hyperlane-xyz#3283)
e16cc70 
### Description

Applies the fix in
hyperlane-xyz#2384 everywhere
an `HttpClient` is constructed via rusoto.

It lowers the S3 timeout to 15s based on tips in [this
thread](hyperium/hyper#2136 (comment)),
to avoid `Error during dispatch: connection closed before message
completed` errors. Note that we'll probably still run into these issues,
but less frequently
([source](rusoto/rusoto#1766 (comment))).


### Drive-by changes

<!--
Are there any minor or drive-by changes also included?
-->

### Related issues

<!--
- Fixes #[issue number here]
-->

### Backward compatibility

<!--
Are these changes backward compatible? Are there any infrastructure
implications, e.g. changes that would prohibit deploying older commits
using this infra tooling?

Yes/No
-->

### Testing

<!--
What kind of testing have these changes undergone?

None/Manual/Unit Tests
-->
daniel-savu added a commit to hyperlane-xyz/hyperlane-monorepo that referenced this issue May 30, 2024
@daniel-savu
fix: lower rusoto timeout to 15s (#3283)
b243823 
Applies the fix in
#2384 everywhere
an `HttpClient` is constructed via rusoto.

It lowers the S3 timeout to 15s based on tips in [this
thread](hyperium/hyper#2136 (comment)),
to avoid `Error during dispatch: connection closed before message
completed` errors. Note that we'll probably still run into these issues,
but less frequently
([source](rusoto/rusoto#1766 (comment))).

<!--
Are there any minor or drive-by changes also included?
-->

<!--
- Fixes #[issue number here]
-->

<!--
Are these changes backward compatible? Are there any infrastructure
implications, e.g. changes that would prohibit deploying older commits
using this infra tooling?

Yes/No
-->

<!--
What kind of testing have these changes undergone?

None/Manual/Unit Tests
-->
@daniel-savu daniel-savu mentioned this issue May 30, 2024
Merged
daniel-savu added a commit to hyperlane-xyz/hyperlane-monorepo that referenced this issue Jun 4, 2024
@daniel-savu
fix: backport rusoto timeout change to v2 (#3872)
3813233 
Backport of
#3283

Applies the fix in
#2384 everywhere
an `HttpClient` is constructed via rusoto.

It lowers the S3 timeout to 15s based on tips in [this
thread](hyperium/hyper#2136 (comment)),
to avoid `Error during dispatch: connection closed before message
completed` errors. Note that we'll probably still run into these issues,
but less frequently

([source](rusoto/rusoto#1766 (comment))).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@silverjam @lucdew