Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken uploads on Conda with S3 storage #1343

Closed
ChGen opened this issue Nov 29, 2023 · 3 comments
Closed

Broken uploads on Conda with S3 storage #1343

ChGen opened this issue Nov 29, 2023 · 3 comments
Assignees

Comments

@ChGen
Copy link
Collaborator

ChGen commented Nov 29, 2023

Broken upload operations for conda repositories with S3 storage. Looks like filesystem storage isn't affected. Could be reproduced with local instance of artipie with local minIO S3 server. It works mostly for the first package upload, subsequent uploads are getting struck in .upload folder.

Also:

  • amazon s3 client should be updated to the new version.
  • asto-core should be buildable & testable with jdk21
  • more integration/smoke tests with s3 storage servers required

See also logs below:

Full logs.zip

Uploading file "aiteam/scipy/1.9.3/linux-64/scipy-1.9.3-py310hdfbd76f_2.tar.bz2"
26.2MB [00:04, 6.17MB/s]
java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: You must provide the Content-Length HTTP header. (Service: S3, Status Code: 411, Request ID: C4PMMMQ7Y57S90N7, Extended Request ID: 2Hkds1PSqtfYKGb/PguPDXoKAbCBFJu/kN5K0/VuPyrCp7MH7Ezvhe86ZWQQpwh/sbh1U1hal44=)
.....
Ram, [24.11.2023 17:33]
INFO: Exists 'conda/linux-64/repodata.json': false
Nov 24, 2023 2:26:07 PM com.artipie.asto.LoggingStorage log
INFO: Save 'conda/linux-64/repodata.json': Optional.empty
Nov 24, 2023 2:26:08 PM com.artipie.asto.LoggingStorage log
INFO: Move 'conda/.upload/linux-64/ai-utils-0.2.0-py310_0.tar.bz2' 'conda/linux-64/ai-utils-0.2.0-py310_0.tar.bz2'

Ram, [24.11.2023 17:34]
here when Exists 'conda/linux-64/repodata.json': false
it worked

Ram, [24.11.2023 17:35]
there were no errors

Ram, [24.11.2023 17:35]
It didn't complain about anything in case of success scenario

Ram, [24.11.2023 17:40]
If I tried to restart and re-login in to the artipie server between the uploads, it is not hanging, but it is throwing like this

Ram, [24.11.2023 17:40]
bade02r@MAC-Y42VW654JQ Downloads % anaconda upload ai-utils-0.2.0-py310_0.tar.bz2
Using Anaconda API: http://localhost:8080/conda
Using "aiteam" as upload username
Processing "ai-utils-0.2.0-py310_0.tar.bz2"
Detecting file type...
File type is "Conda"
Extracting conda attributes for upload
Creating package "ai-utils"
Creating release "0.2.0"
Uploading file "aiteam/ai-utils/0.2.0/linux-64/ai-utils-0.2.0-py310_0.tar.bz2"
24.4kB [00:01, 13.9kB/s]

[ERROR] no element found: line 1, column 0
@ChGen ChGen self-assigned this Nov 29, 2023
@ChGen
Copy link
Collaborator Author

ChGen commented Dec 6, 2023

Hotfix provided in #1351

@ChGen
Copy link
Collaborator Author

ChGen commented Dec 22, 2023

Note: bug could be reproduced by conda test canMultiUploadSameArchTest() when hotfix commit is reverted.

@ChGen
Copy link
Collaborator Author

ChGen commented Jan 24, 2024

Fix and extra tests provided in #1373

@ChGen ChGen closed this as completed Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant