Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Fix corrupted packages with missing conanmanifest.txt files #4662
Changelog: Fix: Fix corrupted packages with missing conanmanifest.txt files
We found an issue on Bintray about missing files in a package, in that case only the conaninfo.txt file was uploaded out of the conan_package.tgz and conanmanifest.txt.
In that case the problematic comes when you want to upload a new package as ir fais becaouse of the following:
To mitigate this I propose a behavior that is already being done in the recipe upload: Check that the manifest is in the snapshot (actually in the recipe upload it checks the availability of the digest and if not there -404- it assusmes there is no package uploaded).
I the digest is for some reason not available it assumes there is no existing package uploaded and it uploads everything.
Another useful change to mitigate this (not proposed at the time of opening this PR) is changing the order of the files upload. Now the files are uploaded in this order: conanmanifest.txt --> conaninfo.txt --> conan_package.tgz and it is done intetionally this way for some reason
From my point of view, the most problematic file during the upload is the conan_package.tgz and it should be uploaded in first place, then the conaninfo.txt and finally the conanmanifest.txt to wrap up the complete upload.
I don't think changing the order helps here. upload has to be transnational operation - conan either uploads everything or nothing. if it's interrupted during the upload (e.g. Ctrl-C pressed, conan crashed, network error occurred or whatever else happened), it should never finalize the transaction and files that were already uploaded or partially uploaded by that time should be discarded.
This was referenced
Mar 6, 2019
So finally I think I have a bunch of interesting tests here bringing some additional information to the original situation like the one described in #4672
To me, this is something that it is worth to have consistent behavior and probably a naive solution would be to check in install/download and upload of packages that the 3 files that define a package are correctly in place, although that would mean that if in the future we decide to remove one of those files the old clients will be broken (although this is something that would require a 2.0 or equivalent).
Having all this into account I would like to know your opinion or at least bring this discussion to one of our whiteboard sessions @conan-io/barbarians
PS: Don't worry about the prints in tests, those are just a proof of concept to get the idea. I will rework them when we agree on a solution
referenced this pull request
Mar 7, 2019
changed the title
WIP: Fix corrupted packages with missing conanmanifest.txt files
Mar 15, 2019
Added the changes to improve the integrity of packages in the complete loop of upload-search-install-upload.
The search couldn't be improved as the values returned by the server are the conaninfo content directly, so not much we can do.
Please take note that this makes an extra API call to check the snapshot in the package installation (download) from the server