-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Boto3 Client Put Object Hangs Indefinitely #3657
Comments
Hi @jacob-aegis, thanks for reaching out. Just to be clear, do you mean that around 1000 images are hanging out of those that you upload, or that they continuously hang after reaching your 1000th upload? Also, could you provide debug logs of an upload that hangs? You can get debug logs by adding |
We have about 2000+ pods on Kubernetes each day and each pod uploads images to s3 during it's lifecycle. Across those 2000+ pods we upload up to 10M images a day. We are seeing around 10 pod's s3 client running into this hanging issue, which I estimate leads to about 1000 hanging uploads. The pods themselves are able to upload to other services (dynamo, RDS, etc.) and have internet access otherwise. I'm hesitant to turn on debug logging since i'd have to do so on all 2000+ pods since there is no pattern to why the uploads are sometimes hanging. |
We changed our code to generate a presigned url and then PUT with requests. It's still showing intermittent hanging on the PUT and adding a timeout to the request seems to have no effect
|
Thanks for following up. In your config, how do you have retries configured? |
We're not passing retries in our config. We're only using requests for the put at the moment, but can switch back to uploading via the S3 Client's put_object. What config do you suggest? |
Realizing we actually already tested with this before but still saw hanging: botoconfig = BotoConfig(max_pool_connections=50, read_timeout=1, retries={'max_attempts': 1}) |
By specifying the max_attempts at 1, and not specifying the mode, you're actually overriding legacy mode's default of 5. I would recommend trying retries with standard mode and without overriding max_attempts, and letting me know how that works. Thanks! |
We tried standard mode and unfortunately we are still seeing the same issues. In the interim we have added a mechanism to reboot the application if it notices hanging, which is a sufficient workaround for the time being. |
Glad to hear you have a workaround while we figure this out. I do want to ask again if there is any way you can get debug logs of a failed upload. Those would really help me narrow this issue down. Otherwise, is there any consistency in the failed uploads that might point us in the right direction? |
Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one. |
Hi, |
For anyone coming across this thread with similar issues to the one I described above, we managed to resolve it by updating our base Docker image of our app to |
Describe the bug
We're uploading image bytes frequently to the same bucket in s3. We upload several million images each day using this same code snippet, but we are finding that put_object has intermittent problems with hanging indefinitely (around 1000 uploads each day). The only resolution has been to relaunch the application pod with the faulty s3 client,
Any suggestions to help catch why this is hanging? It's in a try except/block within our application, but no exceptions are being logged.
Expected Behavior
Successful upload or an exception
Current Behavior
Indefinite hanging
Reproduction Steps
Possible Solution
No response
Additional Information/Context
No response
SDK version used
boto3-1.26.108
Environment details (OS name and version, etc.)
Amazon Linux 2023 ARM
The text was updated successfully, but these errors were encountered: