Parallel conan upload to artifactory fail

[klausholstjacobsen]

Environment details

• OS: Ubuntu 20.04
• Conan 2.0.6:
• Artifactory: 7.49.5

Steps to reproduce

I have 3 parallel gitlab pipeline jobs building packages for the same conan recipe (same rrev due to revision_mode = "scm") to different platforms, and uploading these to my artifactory remote.

Upload command: conan upload myrecipe/1.0.0 --confirm --remote my-virtual-remote
Sometimes the upload fails with a very strange error:

Preparing artifacts to upload
Compressing recipe sources...
Compressing None:conan_sources.tgz
Compressing package...
Uploading artifacts
Uploading recipe 'myrecipe/1.0.0#f0885d238a01f6416656eec66a8b696e84ae384d'
ERROR: 502 Server Error: Bad Gateway for url: https://[hidden]/v2/conans/myrecipe/1.0.0/_/_/revisions/f0885d238a01f6416656eec66a8b696e84ae384d/files/conan_sources.tgz
Waiting 5 seconds to retry...
ERROR: 
Error uploading file: conan_sources.tgz, '502 Server Error: Bad Gateway for url: https://[hidden-url]/v2/conans/myrecipe/1.0.0/_/_/revisions/f0885d238a01f6416656eec66a8b696e84ae384d/files/conan_sources.tgz'
ERROR: Permission denied for user: 'myuser': 403: Not enough permissions to delete/overwrite artifact 'my-virtual-remote:myrecipe/_/1.0.0/_/f0885d238a01f6416656eec66a8b696e84ae384d/export/conanmanifest.txt' (user: 'myuser' needs DELETE permission).. [Remote: my-virtual-remote]

I can read the error message, but I am pretty sure it has nothing to do with permissions as the jobs often succeeds.
My own theory is that it happens when 2 of the parallel jobs finish and uploads at the same time and fails because the upload is not an atomic operation, causing some kind of race condition, but it is all just speculation.

What could be wrong?

Regards
Klaus

[memsharded]

Hi @klausholstjacobsen

Thanks for your report.
This would be an Artifactory issue. If you are a JFrog customer, could you please open them a ticket reporting it? You can share it with me so I can also track it and try to push it.

I'd say that you are mostly right it looks like a race condition, due to not atomic uploads. The way to mitigate it, is avoiding the parallel upload of the recipe, limiting it to one of the CI jobs, uploading only the recipe, then the rest of binaries can be uploaded in parallel, as they will not compete for writing in the same recipe location.

If the remote is a virtual one, it would be better to rule out that as a factor, and reproduce the same issue uploading directly against the local repo, not the virtual redirecting to the local.

[klausholstjacobsen]

https://support.jfrog.com/s/tickets/5006900006hm7qwAAA/parallel-conan-upload-to-artifactory-fail

Ticket Number 263881

[docbrown1955]

Seems to be duplicate of #14175 which is also still open

My personal workaround for CI/CD: if the upload fails, just retry! I don't know the cause of the issue on Artifactory's side, but it is indeed a race condition on the recipe upload because one job already wrote the recipe on the remote and another tries to do it as well (no race condition on packages as they should have different package ids).

Having a (small) logic to retry once - after a second - if the upload fails is enough as the race condition will not happen again, the recipe being already on the remote server. It is only a work-around but it is a robust solution.

[memsharded]

Closing in favor of #16703, that will be used to centralize future 2.X investigations, please track that ticket instead, thanks!