bug: S3 put-object hangs indefinitely #10340

mgrundie-r7 · 2024-02-27T14:49:07Z

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Single part upload of a 22M file using put-object hangs indefinitely. We noticed that cucumber scenarios in our tests were hanging when the aws sdk for ruby was uploading test file. I was able to replicate the behaviour (described below) using the awslocal cli.

Expected Behavior

The file should upload successfully or fail with an error

How are you starting LocalStack?

With a docker run command

Steps To Reproduce

How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

docker run \
  --rm -it \
  -p 4566:4566 \
  -p 4510-4559:4510-4559 \
  localstack/localstack:3.1.0

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

// Create a 22mb file
dd if=/dev/zero of=samplefile.dat  bs=1m count=22
awslocal s3api create-bucket --bucket testbucket

Repeat the below command until it hangs, usually < 20 times 
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat --body ./samplefile.dat

Environment

- OS: MacOS Ventura 13.6.4
- LocalStack: 3.1.0

Anything else?

We think this issue may have started happening when we updated docker to 4.27.0 as it works as expected when we downgraded docker to 4.26.1

The text was updated successfully, but these errors were encountered:

bentsku · 2024-02-28T20:10:42Z

Hello @mgrundie-r7 and thanks for the report!

I was on Docker v4.26.1 and could not reproduce the issue (running the command 50 times in a row with a 22mb file). I've now updated to 4.28.0 and still cannot reproduce the issue. Could it have been a bug in Docker? It seems they released patch versions pretty quickly after. Could you please give it a try with maybe 4.27.2 or 4.28.0 and see if that fixes the issue for you?

I'm not sure the issue is with LocalStack directly. Thanks!

mgrundie-r7 · 2024-02-28T21:08:05Z

@bentsku I've tried it with 4.28 also and still having the issue, it could be specific to docker for MacOS (ARM), which OS did you try on?

mgrundie-r7 · 2024-02-28T21:32:41Z

❯ for i in {1..50}; do
time awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat --body ./samplefile.dat &> /dev/null;
done
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.18s system 63% cpu 0.852 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 64% cpu 0.837 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.18s system 66% cpu 0.817 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 64% cpu 0.835 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.18s system 64% cpu 0.844 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 64% cpu 0.823 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 63% cpu 0.830 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.35s user 0.17s system 64% cpu 0.818 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 62% cpu 0.845 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 64% cpu 0.820 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.18s system 52% cpu 1.038 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.19s system 52% cpu 1.062 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.18s system 63% cpu 0.849 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.18s system 52% cpu 1.049 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.38s user 0.20s system 65% cpu 0.881 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.35s user 0.17s system 62% cpu 0.839 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 65% cpu 0.817 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.19s system 65% cpu 0.853 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.46s user 0.26s system 0% cpu 3:13.53 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.45s user 0.27s system 0% cpu 3:07.84 total

Notice the last 2 hang for longer than 3 mins. I cancel at this point because every attempt at this point hangs.

❯ awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat

Read timeout on endpoint URL: "http://localhost:4566/testbucket/somekey/samplefile.dat"

With docker desktop 4.26.1 I ran it 150 times without issue

bentsku · 2024-02-29T11:50:36Z

@mgrundie-r7 I'm on MacOS Sonoma 14.3 (23D56) on an Apple M3 Max.
Docker version 25.0.3, build 4debf41
Docker Desktop for Mac: 4.28.0 (139021)

This S3 implementation only use filesystem operations from Python, so I don't see it doing anything crazy. Do you have any LocalStack logs before it hangs? You can start LocalStack with DEBUG=1 and see if anything shows up.

Could you share how you are starting LocalStack? Is it the simple Docker command above?

Does the issue happens if the object key is not always the same? Trying to narrow down why this is happening.

I ran the script myself and could not reproduce:

Results

$ docker --version
Docker version 25.0.3, build 4debf41
$ docker run  --rm -it -p 4566:4566 -p 4510-4559:4510-4559 -d localstack/localstack:3.1.0
$ awslocal s3 mb s3://testbucket
$ for i in {1..50}; do          
time awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat --body ./samplefile.dat &> /dev/null;
done
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.07s system 60% cpu 0.486 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 62% cpu 0.441 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 64% cpu 0.454 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 62% cpu 0.437 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 62% cpu 0.445 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.452 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.07s system 64% cpu 0.461 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.454 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.449 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 68% cpu 0.422 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.450 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 64% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 63% cpu 0.447 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.24s user 0.07s system 46% cpu 0.667 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.25s user 0.07s system 46% cpu 0.675 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 62% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 62% cpu 0.448 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 64% cpu 0.447 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.458 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 62% cpu 0.450 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 62% cpu 0.455 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.25s user 0.07s system 46% cpu 0.678 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 62% cpu 0.478 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 62% cpu 0.455 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 63% cpu 0.441 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.25s user 0.07s system 45% cpu 0.689 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 62% cpu 0.457 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 67% cpu 0.432 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 62% cpu 0.455 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 67% cpu 0.416 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 63% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.447 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 63% cpu 0.449 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 63% cpu 0.441 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 66% cpu 0.419 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.458 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.452 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 62% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.25s user 0.07s system 46% cpu 0.680 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.25s user 0.07s system 46% cpu 0.683 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 62% cpu 0.468 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.457 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 62% cpu 0.462 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.23s user 0.06s system 63% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.22s user 0.06s system 63% cpu 0.445 total

mgrundie-r7 · 2024-02-29T11:57:51Z

I'll get all requested details shortly. Yes, that is the run command I use:

❯ docker run \
  --rm -it \
  -p 4566:4566 \
  -p 4510-4559:4510-4559 \
  localstack/localstack:3.1.0

Just one clarification, I'm not running awslocal s3api put-object... inside the localstack container. I am running in my mac user shell

❯ which localstack
/opt/homebrew/bin/localstack

bentsku · 2024-02-29T11:59:18Z

Ah yes sorry, I'm running it from my Mac user shell as well, sorry for the results not showing it 😅

mgrundie-r7 · 2024-02-29T13:31:07Z

Mac: Ventura 13.6.4 (22G513) Apple M1 Pro
Docker Desktop: 4.28.0 (139021)

❯ docker --version
Docker version 25.0.3, build 4debf41

❯ for i in {1..50}; do time awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat --body ./samplefile.dat &> /dev/null; done
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.38s user 0.19s system 59% cpu 0.956 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.38s user 0.18s system 63% cpu 0.883 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.38s user 0.18s system 62% cpu 0.896 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.18s system 64% cpu 0.842 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.39s user 0.18s system 64% cpu 0.891 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.40s user 0.18s system 64% cpu 0.900 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.18s system 61% cpu 0.900 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 62% cpu 0.856 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 63% cpu 0.837 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 62% cpu 0.848 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.18s system 64% cpu 0.850 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.38s user 0.18s system 64% cpu 0.860 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.39s user 0.19s system 64% cpu 0.891 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 61% cpu 0.888 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 61% cpu 0.880 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 63% cpu 0.853 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 64% cpu 0.845 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 63% cpu 0.848 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 63% cpu 0.848 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.38s user 0.19s system 63% cpu 0.879 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.39s user 0.19s system 65% cpu 0.882 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.39s user 0.20s system 64% cpu 0.923 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.38s user 0.17s system 62% cpu 0.876 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.38s user 0.17s system 64% cpu 0.850 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.18s system 63% cpu 0.863 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 63% cpu 0.851 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 63% cpu 0.853 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.16s system 63% cpu 0.843 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 63% cpu 0.843 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.37s user 0.17s system 60% cpu 0.876 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.36s user 0.17s system 63% cpu 0.841 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat     0.45s user 0.26s system 0% cpu 3:17.45 tota

Localstack debug log

2024-02-29T13:25:46.990  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200
2024-02-29T13:25:47.739 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile.dat
2024-02-29T13:25:47.861  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200
2024-02-29T13:25:48.588 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile.dat
2024-02-29T13:25:48.712  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200
2024-02-29T13:25:49.453 DEBUG --- [   asgi_gw_0] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile.dat
2024-02-29T13:25:53.141 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:26:03.366 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:26:13.586 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:26:23.821 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:26:34.049 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:26:44.244 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:26:54.474 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:26:54.852 DEBUG --- [   asgi_gw_1] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile.dat
2024-02-29T13:27:04.689 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:27:14.895 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:27:25.119 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:27:35.320 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:27:45.544 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:27:55.764 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:28:00.962 DEBUG --- [   asgi_gw_2] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile.dat
2024-02-29T13:28:05.996 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:28:16.217 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:28:26.441 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:28:36.676 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:28:46.898 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:28:57.120 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:29:07.145 DEBUG --- [   asgi_gw_3] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile.dat
2024-02-29T13:29:07.312 DEBUG --- [   asgi_gw_4] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:29:17.536 DEBUG --- [   asgi_gw_4] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:29:27.741 DEBUG --- [   asgi_gw_4] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:29:37.957 DEBUG --- [   asgi_gw_4] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:29:48.179 DEBUG --- [   asgi_gw_4] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:29:58.419 DEBUG --- [   asgi_gw_4] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:30:08.660 DEBUG --- [   asgi_gw_4] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health

mgrundie-r7 · 2024-02-29T13:41:20Z

Changing the key by appending $i doesn't resolve the issue.

❯ for i in {1..50}; do time awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat" --body ./samplefile.dat &> /dev/null; done
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.19s system 55% cpu 1.017 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.19s system 65% cpu 0.879 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 64% cpu 0.864 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.36s user 0.17s system 63% cpu 0.834 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 63% cpu 0.870 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 64% cpu 0.865 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.36s user 0.17s system 64% cpu 0.839 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.16s system 63% cpu 0.843 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.17s system 63% cpu 0.850 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.36s user 0.17s system 59% cpu 0.896 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 63% cpu 0.866 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.36s user 0.17s system 63% cpu 0.851 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.36s user 0.17s system 63% cpu 0.833 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 64% cpu 0.870 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 64% cpu 0.867 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 64% cpu 0.867 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.17s system 61% cpu 0.886 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 51% cpu 1.089 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 61% cpu 0.883 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 62% cpu 0.887 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.36s user 0.17s system 61% cpu 0.863 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.17s system 63% cpu 0.847 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 61% cpu 0.886 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.17s system 62% cpu 0.872 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.40s user 0.19s system 62% cpu 0.942 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.39s user 0.18s system 59% cpu 0.961 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.17s system 62% cpu 0.878 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 62% cpu 0.884 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 63% cpu 0.891 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 61% cpu 0.888 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.39s user 0.19s system 63% cpu 0.908 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 61% cpu 0.898 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.17s system 63% cpu 0.863 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 62% cpu 0.883 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.17s system 63% cpu 0.860 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.17s system 63% cpu 0.861 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 63% cpu 0.879 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.39s user 0.19s system 66% cpu 0.871 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 63% cpu 0.871 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.36s user 0.17s system 60% cpu 0.880 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.38s user 0.18s system 62% cpu 0.888 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.18s system 63% cpu 0.864 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.39s user 0.18s system 62% cpu 0.910 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.39s user 0.18s system 63% cpu 0.900 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.39s user 0.18s system 62% cpu 0.902 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.17s system 62% cpu 0.865 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.37s user 0.17s system 61% cpu 0.893 total
awslocal s3api put-object --bucket testbucket --key "somekey/samplefile$i.dat  0.46s user 0.24s system 0% cpu 3:12.68 total

Debug log I can see that it attempted to put samplefile48.dat 3 times

2024-02-29T13:35:43.794  INFO --- [   asgi_gw_6] localstack.request.aws     : AWS s3.PutObject => 200
2024-02-29T13:35:44.511 DEBUG --- [   asgi_gw_7] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile48.dat
2024-02-29T13:35:45.863 DEBUG --- [   asgi_gw_6] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:35:56.093 DEBUG --- [   asgi_gw_6] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:36:06.290 DEBUG --- [   asgi_gw_6] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:36:16.506 DEBUG --- [   asgi_gw_6] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:36:26.717 DEBUG --- [   asgi_gw_6] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:36:36.944 DEBUG --- [   asgi_gw_6] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:36:44.645 DEBUG --- [   asgi_gw_6] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile48.dat
2024-02-29T13:36:47.168 DEBUG --- [   asgi_gw_8] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:36:57.390 DEBUG --- [   asgi_gw_8] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:37:07.611 DEBUG --- [   asgi_gw_8] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:37:17.831 DEBUG --- [   asgi_gw_8] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:37:28.065 DEBUG --- [   asgi_gw_8] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:37:38.282 DEBUG --- [   asgi_gw_8] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:37:48.494 DEBUG --- [   asgi_gw_8] rolo.gateway.wsgi          : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-29T13:37:51.310 DEBUG --- [   asgi_gw_8] rolo.gateway.wsgi          : PUT localhost:4566/testbucket/somekey/samplefile48.dat

mgrundie-r7 · 2024-02-29T14:03:10Z

I confirmed colleagues with the same mac hardware (M1) can replicate this issue . A Colleague with intel mac cannot replicate this issue

bentsku · 2024-02-29T14:32:39Z

This is getting interesting 😬 I wonder if this is related to Docker or something we're doing.
Are you all using Ventura 13.6.4 and not Sonoma?
I only have an Intel Mac with Ventura on my hand. 😕

Could you make more tests with maybe a bigger file, see if that would fail faster? Can you put more files with a 5MB file and less file with a 50MB one for example?

I can also see the new feature starting at 4.27, the Synchronized file shares, but those are not enabled for me.

edit: and seeing the file n.48 taking 3min seems to be because of retries, it hangs for 1min, timeout then the CLI retries 3 times and finally gives up.

bentsku · 2024-02-29T14:47:12Z

I'm ready to try anything at this point, do you think you could maybe try with an older LocalStack image, just in case? You could try with localstack:1.2 and localstack:2.1 for example, as the S3 implementations were different.

I could also help by making a minimal Docker image reproducing the bug for the Docker issue you've opened.

mgrundie-r7 · 2024-02-29T14:56:05Z

I feel like this is a docker issue since downgrading docker stops it happening. I'm surprised it does not happen on your arm Mac, I would have thought that if it is a docker issue in the arm build then all arm mac users would be affected. I'll try the steps you've described.

Synchronized file shares are disabled for me also
We all use Ventura because our company is slow is support new MacOS versions

mgrundie-r7 · 2024-02-29T14:56:24Z

Thanks for your time, much appreciated

bentsku · 2024-02-29T14:59:23Z

Yes, I agree with you regarding the fact that it is most probably related to Docker, as we can run this code locally in a Python interpreter and not run into any issue.

I'm surprised too! I'll try running it around 150 times too, just to see if I can reproduce.
edit: can't reproduce, even with 150 iterations 😞
Maybe a combination of macOS version + Docker, can't be sure. 🤷‍♂️

mgrundie-r7 · 2024-02-29T15:51:58Z

Interesting update: Colleague with same M1 hardware who has Sonoma is NOT affected

bentsku · 2024-02-29T18:11:31Z

Oh wow, alright. I guess you will have to keep Docker at 4.26.1 until you can upgrade to Sonoma then. I wonder what is the reason behind it... do you have any issue with any other Docker image than LocalStack?

mgrundie-r7 · 2024-03-05T09:21:59Z

None that I've noticed. Hopefully something comes from the docker issue I've raised, Not sure how to proceed with this issue as it might be useful for others experiencing this problem.

bentsku · 2024-03-05T17:28:41Z

Let's keep it opened for now. Seems a bit weird that only LocalStack would be affected by this issue 😬

You could maybe try starting LocalStack with PROVIDER_OVERRIDE_S3=legacy_v2 to see if the problem is still present, or if there's something we could do to improve the issue. Maybe we could improve something?

bentsku · 2024-03-19T23:11:00Z

Hello again @mgrundie-r7!

Is it still ongoing? We never know, but could you try with localstack/localstack:latest with GATEWAY_SERVER=twisted? We've merged a new implementation of our HTTP server, and it could maybe improve the situation, we never know. Thanks!

mgrundie-r7 · 2024-03-20T09:43:57Z

Hey @bentsku it was still happening yea, I've tried your steps an the good news is that seems to a have resolved the issue. I ran the loop for 400+ iterations. One iteration did pause for 1 min but then it resumed.

Please have a look at the comments in docker/for-mac#7207 as packet capture shows repeating ZeroWindowProbe that continued even after I killed the loop

Below are the steps I followed and some errors I seen in localstack logs during startup

docker run \
  --rm -it \
  -p 4566:4566 \
  -p 4510-4559:4510-4559 \
  -e DEBUG=1 -e GATEWAY_SERVER=twisted localstack/localstack:latest

LocalStack version: 3.2.1.dev
LocalStack build date: 2024-03-20
LocalStack build git hash: 9d29716

Errors during localstack startup

2024-03-20T09:20:44.150 DEBUG --- [  MainThread] localstack.dns.server      : DNS server startup finished.
2024-03-20T09:20:44.155 DEBUG --- [-functhread2] l.u.c.docker_sdk_client    : Creating Docker SDK client failed: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1293, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1052, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 990, in send
    self.connect()
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1293, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1052, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 990, in send
    self.connect()
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/localstack/utils/container_utils/docker_sdk_client.py", line 75, in _create_client
    return docker.from_env(timeout=DOCKER_SDK_DEFAULT_TIMEOUT_SECONDS)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-03-20T09:20:44.166 DEBUG --- [-functhread2] l.u.c.docker_sdk_client    : Creating Docker SDK client failed: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1293, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1052, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 990, in send
    self.connect()
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1293, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1052, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 990, in send
    self.connect()
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/code/localstack/localstack/utils/container_utils/docker_sdk_client.py", line 75, in _create_client
    return docker.from_env(timeout=DOCKER_SDK_DEFAULT_TIMEOUT_SECONDS)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/client.py", line 96, in from_env
    return cls(
           ^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-03-20T09:20:44.157 DEBUG --- [  MainThread] localstack.runtime.init    : Init scripts discovered: {BOOT: [], START: [], READY: [], SHUTDOWN: []}

bentsku · 2024-03-20T11:21:11Z

Thanks so much for the quick response @mgrundie-r7!

Right, the packet capture looks very interesting... we've had issue with our web server before, which is why we're working on improving it. At least the fact that it doesn't hang fully anymore is great progress.

The issue in the logs might be related to the fact the docker socket is not mounted in the container. It's not necessary for S3 and can be disregarded, but would be for many other services like lambda .I think this can be safe to disregard, but it might be a bit too verbose. Were those errors not present before?

If you're only using S3 in your tests, you could use the following tag: localstack/localstack:s3-latest, which only contains the S3 service and is lighter, it also has GATEWAY_SERVER=twisted as default. Not sure if that's relevant for you.

mgrundie-r7 · 2024-03-20T11:30:20Z

@bentsku apologies I should have checked that. Yes those errors are also present in localstack 3.1.0 (with both new and old docker)

bentsku · 2024-03-20T13:25:40Z

@mgrundie-r7 no worries, I was just checking we did not introduce something 😄 do you think we can consider the issue closed now with the new HTTP server?

mgrundie-r7 · 2024-03-20T13:48:32Z

Yea, I think we can. When will this change be available in a stable release?

I wonder if I should close the docker one or not, I still think that this was a docker issue since it only broke when docker was upgraded and not localstack

Thanks for your help on this!

bentsku · 2024-03-20T13:55:00Z

We are planning 3.3 for next week on the 27th. I can ping you here once it is available if you wish.
This change will normally become default only in the next major release.

For Docker, I agree, we might have been doing something wrong, but it is still weird it broke all of a sudden.
Thank you for the detailed report! Feel free to comment again or re-open if there's anything.

mgrundie-r7 · 2024-04-10T16:16:14Z

For anyone who comes across this. Docker found the cause of this issue and have a fix

docker/for-mac#7207

mgrundie-r7 added status: triage needed Requires evaluation by maintainers type: bug Bug report labels Feb 27, 2024

bentsku added aws:s3 Amazon Simple Storage Service status: in progress Currently being worked on and removed status: triage needed Requires evaluation by maintainers labels Feb 27, 2024

bentsku self-assigned this Feb 27, 2024

bentsku added status: response required Waiting for a response from the reporter and removed status: in progress Currently being worked on labels Feb 28, 2024

localstack-bot removed the status: response required Waiting for a response from the reporter label Feb 28, 2024

mgrundie-r7 mentioned this issue Feb 29, 2024

Docker Desktop breaks localstack s3 transfer on M1 mac since 4.27.0 (Unknown network issue/limiting) docker/for-mac#7207

Closed

bentsku added the status: response required Waiting for a response from the reporter label Mar 1, 2024

localstack-bot removed the status: response required Waiting for a response from the reporter label Mar 5, 2024

bentsku added the area: docker Use Docker with LocalStack label Mar 5, 2024

bentsku added the status: in progress Currently being worked on label Mar 5, 2024

bentsku added status: response required Waiting for a response from the reporter and removed status: in progress Currently being worked on labels Mar 19, 2024

localstack-bot removed the status: response required Waiting for a response from the reporter label Mar 20, 2024

bentsku added the status: response required Waiting for a response from the reporter label Mar 20, 2024

localstack-bot removed the status: response required Waiting for a response from the reporter label Mar 20, 2024

bentsku closed this as completed Mar 20, 2024

daniel-frak mentioned this issue Mar 26, 2024

S3 timing out on CI since 3.0.3 (tests using LocalStack) awspring/spring-cloud-aws#1118

Closed

MarcelStranak mentioned this issue May 16, 2024

bug: hanging during docker run with docker v4.30.0 #10832

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug: S3 put-object hangs indefinitely #10340

bug: S3 put-object hangs indefinitely #10340

mgrundie-r7 commented Feb 27, 2024 •

edited

Loading

bentsku commented Feb 28, 2024 •

edited

Loading

mgrundie-r7 commented Feb 28, 2024 •

edited

Loading

mgrundie-r7 commented Feb 28, 2024 •

edited

Loading

bentsku commented Feb 29, 2024 •

edited

Loading

mgrundie-r7 commented Feb 29, 2024 •

edited

Loading

bentsku commented Feb 29, 2024

mgrundie-r7 commented Feb 29, 2024

mgrundie-r7 commented Feb 29, 2024

mgrundie-r7 commented Feb 29, 2024

bentsku commented Feb 29, 2024 •

edited

Loading

bentsku commented Feb 29, 2024 •

edited

Loading

mgrundie-r7 commented Feb 29, 2024

mgrundie-r7 commented Feb 29, 2024

bentsku commented Feb 29, 2024 •

edited

Loading

mgrundie-r7 commented Feb 29, 2024

bentsku commented Feb 29, 2024 •

edited

Loading

mgrundie-r7 commented Mar 5, 2024

bentsku commented Mar 5, 2024

bentsku commented Mar 19, 2024

mgrundie-r7 commented Mar 20, 2024

bentsku commented Mar 20, 2024

mgrundie-r7 commented Mar 20, 2024

bentsku commented Mar 20, 2024

mgrundie-r7 commented Mar 20, 2024

bentsku commented Mar 20, 2024

mgrundie-r7 commented Apr 10, 2024

bug: S3 put-object hangs indefinitely #10340

bug: S3 put-object hangs indefinitely #10340

Comments

mgrundie-r7 commented Feb 27, 2024 • edited Loading

Is there an existing issue for this?

Current Behavior

Expected Behavior

How are you starting LocalStack?

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Environment

Anything else?

bentsku commented Feb 28, 2024 • edited Loading

mgrundie-r7 commented Feb 28, 2024 • edited Loading

mgrundie-r7 commented Feb 28, 2024 • edited Loading

bentsku commented Feb 29, 2024 • edited Loading

mgrundie-r7 commented Feb 29, 2024 • edited Loading

bentsku commented Feb 29, 2024

mgrundie-r7 commented Feb 29, 2024

mgrundie-r7 commented Feb 29, 2024

mgrundie-r7 commented Feb 29, 2024

bentsku commented Feb 29, 2024 • edited Loading

bentsku commented Feb 29, 2024 • edited Loading

mgrundie-r7 commented Feb 29, 2024

mgrundie-r7 commented Feb 29, 2024

bentsku commented Feb 29, 2024 • edited Loading

mgrundie-r7 commented Feb 29, 2024

bentsku commented Feb 29, 2024 • edited Loading

mgrundie-r7 commented Mar 5, 2024

bentsku commented Mar 5, 2024

bentsku commented Mar 19, 2024

mgrundie-r7 commented Mar 20, 2024

bentsku commented Mar 20, 2024

mgrundie-r7 commented Mar 20, 2024

bentsku commented Mar 20, 2024

mgrundie-r7 commented Mar 20, 2024

bentsku commented Mar 20, 2024

mgrundie-r7 commented Apr 10, 2024

mgrundie-r7 commented Feb 27, 2024 •

edited

Loading

How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

bentsku commented Feb 28, 2024 •

edited

Loading

mgrundie-r7 commented Feb 28, 2024 •

edited

Loading

mgrundie-r7 commented Feb 28, 2024 •

edited

Loading

bentsku commented Feb 29, 2024 •

edited

Loading

mgrundie-r7 commented Feb 29, 2024 •

edited

Loading

bentsku commented Feb 29, 2024 •

edited

Loading

bentsku commented Feb 29, 2024 •

edited

Loading

bentsku commented Feb 29, 2024 •

edited

Loading

bentsku commented Feb 29, 2024 •

edited

Loading