UploadPart request occassionally fails with broken pipe error

[Gun9niR]

When uploading an object with multipart upload. I occassionally get error writing a body to connection: Broken pipe (os error 32) or TransientError: connection closed before message completed error.

I also noticed that this error would only happen when the data passed into body needs to be polled multiple times. In my case,

fn get_upload_body(data: Vec<Bytes>) -> aws_sdk_s3::types::ByteStream {
    Body::wrap_stream(stream::iter(data.into_iter().map(ObjectResult::Ok))).into()
}

client_cloned
    .upload_part()
    .bucket(bucket)
    .key(key)
    .upload_id(upload_id)
    .part_number(part_id)
    .body(get_upload_body(data))
    .content_length(len as i64)
    .send()
    .await
    .map_err(ObjectError::s3);

will occasionally return with an error, while concatenating the Vec<Bytes> and changing get_upload_body to

fn get_upload_body(data: Bytes) -> aws_sdk_s3::types::ByteStream {
    aws_sdk_s3::types::ByteStream::from(data)
}

will never return an error.

---

I have searched it online a bit already, and according to this issue, it seems to be related to a race condition in hyper. However, no matter how short I set the client-side keep-alive timeout (by passing in a custom hyper client), the error persists. With all that said, my questions are:

• What is the connection keep-alive timeout on S3 server side? I failed to find any documentation for this.
• If there is, is it configurable with a keep-alive HTTP header or API?
• Any other possible causes for my problem?

[jdisanti]

I think that by converting hyper::Body into a ByteStream, you're inadvertently disabling retry for the operation since the SDK has no way to restart the stream in the event of failure. If you're streaming from a file, then you can use ByteStream::from_path to get streaming retry. Otherwise, you should use SdkBody::retryable to create an SdkBody, and then convert that SdkBody into a ByteStream. For example:

use aws_smithy_http::body::SdkBody;
use std::convert::Infallible;
use tokio_stream as stream;

fn get_upload_body(data: Vec<bytes::Bytes>) -> aws_sdk_s3::types::ByteStream {
    SdkBody::retryable(move || {
        SdkBody::from(hyper::Body::wrap_stream(stream::iter(
            data.clone().into_iter().map(Result::<_, Infallible>::Ok),
        )))
    })
    .into()
}

Also, if you're using Tokio's single-threaded runtime, there is a bug that makes IncompleteMessage failures occur more frequently.

[Gun9niR]

Thanks for the quick response!

I'm not streaming from a file. So are you suggesting that I never experience failure in the second case because retry is not disabled so the error rate is extremely low?

[jdisanti]

In all likelihood, it is failing, but that failure is getting retried automatically. I think it logs retry at the debug level too, so you wouldn't see that unless you had your logging configured for it. That said, I don't have any evidence to suggest that. I would be interested to know more if you dig into it, since there definitely could be a bug hiding in here.

[Gun9niR]

I tried to add the retry and the error was gone, so I guess that was the problem. Thanks!

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.