Operations hang indefinitely after opening too many files

[passaro]

Mountpoint for Amazon S3 version

mount-s3 1.5.0

AWS Region

eu-west-1

Describe the running environment

EC2 instance c5.4xlarge, Amazon Linux 2023

Mountpoint options

mount-s3 BUCKET mnt --maximum-throughput-gbps=8 --debug-crt

What happened?

When too many files are open for writing, Mountpoint becomes unresponsive and operations hangs indefinitely.

Reproduction:

• Create and open 20 new files inside the mounted directory and do not close them
• Try ls mnt

The command never returns.

Relevant log output

2024-03-27T14:09:42.198800Z  INFO awscrt::S3ClientStats: id=0x5632618b1230 Requests-in-flight(approx/exact):20/20  Requests-preparing:20  Requests-queued:0  Requests-network(get/put/default/total):0/0/0/0  Requests-streaming-waiting:0  Requests-streaming-response:0  Endpoints(in-table/allocated):1/1

[passaro]

The problem is caused by an issue with how Mountpoint issues upload requests to the underlying Common Runtime client, leading to situations where the client is waiting for the data to upload rather than processing other requests.

We are working on a fix, which will rely on the corresponding change under review here: awslabs/aws-c-s3#418.

[passaro]

Fix available in mountpoint-s3 v1.6.0.