-
-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bug: S3 put-object hangs indefinitely #10340
Comments
Hello @mgrundie-r7 and thanks for the report! I was on Docker v4.26.1 and could not reproduce the issue (running the command 50 times in a row with a 22mb file). I've now updated to 4.28.0 and still cannot reproduce the issue. Could it have been a bug in Docker? It seems they released patch versions pretty quickly after. Could you please give it a try with maybe I'm not sure the issue is with LocalStack directly. Thanks! |
@bentsku I've tried it with 4.28 also and still having the issue, it could be specific to docker for MacOS (ARM), which OS did you try on? |
Notice the last 2 hang for longer than 3 mins. I cancel at this point because every attempt at this point hangs.
With docker desktop 4.26.1 I ran it 150 times without issue |
@mgrundie-r7 I'm on MacOS Sonoma 14.3 (23D56) on an Apple M3 Max. This S3 implementation only use filesystem operations from Python, so I don't see it doing anything crazy. Do you have any LocalStack logs before it hangs? You can start LocalStack with Could you share how you are starting LocalStack? Is it the simple Docker command above? Does the issue happens if the object key is not always the same? Trying to narrow down why this is happening. I ran the script myself and could not reproduce: Results
$ docker --version
Docker version 25.0.3, build 4debf41
$ docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 -d localstack/localstack:3.1.0
$ awslocal s3 mb s3://testbucket
$ for i in {1..50}; do
time awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat --body ./samplefile.dat &> /dev/null;
done
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.07s system 60% cpu 0.486 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 62% cpu 0.441 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 64% cpu 0.454 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 62% cpu 0.437 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 62% cpu 0.445 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.452 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.07s system 64% cpu 0.461 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.454 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.449 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 68% cpu 0.422 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.450 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 64% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 63% cpu 0.447 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.24s user 0.07s system 46% cpu 0.667 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.25s user 0.07s system 46% cpu 0.675 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 62% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 62% cpu 0.448 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 64% cpu 0.447 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.458 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 62% cpu 0.450 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 62% cpu 0.455 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.25s user 0.07s system 46% cpu 0.678 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 62% cpu 0.478 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 62% cpu 0.455 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 63% cpu 0.441 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.25s user 0.07s system 45% cpu 0.689 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 62% cpu 0.457 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 67% cpu 0.432 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 62% cpu 0.455 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 67% cpu 0.416 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 63% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.447 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 63% cpu 0.449 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 63% cpu 0.441 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 66% cpu 0.419 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.458 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.452 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 62% cpu 0.453 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.25s user 0.07s system 46% cpu 0.680 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.25s user 0.07s system 46% cpu 0.683 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 62% cpu 0.468 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.457 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 62% cpu 0.462 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.23s user 0.06s system 63% cpu 0.451 total
awslocal s3api put-object --bucket testbucket --key somekey/samplefile.dat 0.22s user 0.06s system 63% cpu 0.445 total |
I'll get all requested details shortly. Yes, that is the run command I use:
Just one clarification, I'm not running
|
Ah yes sorry, I'm running it from my Mac user shell as well, sorry for the results not showing it 😅 |
Mac: Ventura 13.6.4 (22G513) Apple M1 Pro
Localstack debug log
|
Changing the key by appending $i doesn't resolve the issue.
Debug log I can see that it attempted to put samplefile48.dat 3 times
|
I confirmed colleagues with the same mac hardware (M1) can replicate this issue . A Colleague with intel mac cannot replicate this issue |
This is getting interesting 😬 I wonder if this is related to Docker or something we're doing. Could you make more tests with maybe a bigger file, see if that would fail faster? Can you put more files with a 5MB file and less file with a 50MB one for example? I can also see the new feature starting at edit: and seeing the file n.48 taking 3min seems to be because of retries, it hangs for 1min, timeout then the CLI retries 3 times and finally gives up. |
I'm ready to try anything at this point, do you think you could maybe try with an older LocalStack image, just in case? You could try with I could also help by making a minimal Docker image reproducing the bug for the Docker issue you've opened. |
I feel like this is a docker issue since downgrading docker stops it happening. I'm surprised it does not happen on your arm Mac, I would have thought that if it is a docker issue in the arm build then all arm mac users would be affected. I'll try the steps you've described. Synchronized file shares are disabled for me also |
Thanks for your time, much appreciated |
Yes, I agree with you regarding the fact that it is most probably related to Docker, as we can run this code locally in a Python interpreter and not run into any issue. I'm surprised too! I'll try running it around 150 times too, just to see if I can reproduce. |
Interesting update: Colleague with same M1 hardware who has Sonoma is NOT affected |
Oh wow, alright. I guess you will have to keep Docker at |
None that I've noticed. Hopefully something comes from the docker issue I've raised, Not sure how to proceed with this issue as it might be useful for others experiencing this problem. |
Let's keep it opened for now. Seems a bit weird that only LocalStack would be affected by this issue 😬 You could maybe try starting LocalStack with |
Hello again @mgrundie-r7! Is it still ongoing? We never know, but could you try with |
Hey @bentsku it was still happening yea, I've tried your steps an the good news is that seems to a have resolved the issue. I ran the loop for 400+ iterations. One iteration did pause for 1 min but then it resumed. Please have a look at the comments in docker/for-mac#7207 as packet capture shows repeating ZeroWindowProbe that continued even after I killed the loop Below are the steps I followed and some errors I seen in localstack logs during startup
Errors during localstack startup
|
Thanks so much for the quick response @mgrundie-r7! Right, the packet capture looks very interesting... we've had issue with our web server before, which is why we're working on improving it. At least the fact that it doesn't hang fully anymore is great progress. The issue in the logs might be related to the fact the docker socket is not mounted in the container. It's not necessary for S3 and can be disregarded, but would be for many other services like lambda .I think this can be safe to disregard, but it might be a bit too verbose. Were those errors not present before? If you're only using S3 in your tests, you could use the following tag: |
@bentsku apologies I should have checked that. Yes those errors are also present in localstack 3.1.0 (with both new and old docker) |
@mgrundie-r7 no worries, I was just checking we did not introduce something 😄 do you think we can consider the issue closed now with the new HTTP server? |
Yea, I think we can. When will this change be available in a stable release? I wonder if I should close the docker one or not, I still think that this was a docker issue since it only broke when docker was upgraded and not localstack Thanks for your help on this! |
We are planning For Docker, I agree, we might have been doing something wrong, but it is still weird it broke all of a sudden. |
For anyone who comes across this. Docker found the cause of this issue and have a fix |
Is there an existing issue for this?
Current Behavior
Single part upload of a 22M file using put-object hangs indefinitely. We noticed that cucumber scenarios in our tests were hanging when the aws sdk for ruby was uploading test file. I was able to replicate the behaviour (described below) using the awslocal cli.
Expected Behavior
The file should upload successfully or fail with an error
How are you starting LocalStack?
With a
docker run
commandSteps To Reproduce
How are you starting localstack (e.g.,
bin/localstack
command, arguments, ordocker-compose.yml
)Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
Environment
Anything else?
We think this issue may have started happening when we updated docker to 4.27.0 as it works as expected when we downgraded docker to 4.26.1
The text was updated successfully, but these errors were encountered: