(object_store) Error broken pipe on S3 multipart upload

[Samrose-Ahmed]

Which part is this question about

objec_store

Describe your question

We are seeing error of type:

0: Generic S3 error: response error "request error", after 0 retries: error sending request for url (https://s3.us-west-2.amazonaws.com/bucket/3bb9fe5d-dd42-4137-9383-828cae7c363a.zstd.parquet?partNumber=4&uploadId=uN4MB5OPLXo7uM7cv.ZM2YBuIQNgTSpXQNkgLDE8A2_mFP1LwNlDScnhAyQmj5f8_cLnit.OJ_zBbTzFZ_ANENjAZB_T9cjOFLec3ELLKDvPmhxl3nmjrM8ZqIoTFXRnLMKSCuXkZMuaSfqc8KTShYoobTOgmOV99_O3A4ZY1pU-): error writing a body to connection: Broken pipe (os error 32) | main

1: response error "request error", after 0 retries: error sending request for url (https://s3.us-west-2.amazonaws.com/bucket/3bb9fe5d-dd42-4137-9383-828cae7c363a.zstd.parquet?partNumber=4&uploadId=uN4MB5OPLXo7uM7cv.ZM2YBuIQNgTSpXQNkgLDE8A2_mFP1LwNlDScnhAyQmj5f8_cLnit.OJ_zBbTzFZ_ANENjAZB_T9cjOFLec3ELLKDvPmhxl3nmjrM8ZqIoTFXRnLMKSCuXkZMuaSfqc8KTShYoobTOgmOV99_O3A4ZY1pU-): error writing a body to connection: Broken pipe (os error 32) | main
2: error sending request for url (https://s3.us-west-2.amazonaws.com/bucket/3bb9fe5d-dd42-4137-9383-828cae7c363a.zstd.parquet?partNumber=4&uploadId=uN4MB5OPLXo7uM7cv.ZM2YBuIQNgTSpXQNkgLDE8A2_mFP1LwNlDScnhAyQmj5f8_cLnit.OJ_zBbTzFZ_ANENjAZB_T9cjOFLec3ELLKDvPmhxl3nmjrM8ZqIoTFXRnLMKSCuXkZMuaSfqc8KTShYoobTOgmOV99_O3A4ZY1pU-): error writing a body to connection: Broken pipe (os error 32) | main
3: error writing a body to connection: Broken pipe (os error 32) | main
4: Broken pipe (os error 32)

Is there any guidance on this? I've tried tweaking the timeouts.

Additional context

[tustvold]

This is an error coming from the operating system saying that either the upstream or something between you and the upstream dropped the TCP connection. This could be for any number of reasons, from dodgy middleware boxes, to exceeding bandwidth limits of some appliance, or any number of other things...

Are you encountering this whilst running in AWS, or on some other network?

[Samrose-Ahmed]

Yes this is on AWS Fargate

[tustvold]

You could possibly look at the VPC flow logs, assuming they work for Fargate, but I don't really have any other suggestions on how to debug this

[Samrose-Ahmed]

Ok thanks for helping, it migh be bandwidth limits, I will investigate more.