Fix corrupted packages with missing conanmanifest.txt files

[danimtb]

Changelog: Fix: Fix corrupted packages with missing conanmanifest.txt files
Docs: omit

Close #4698
@revisions: 1

We found an issue on Bintray about missing files in a package, in that case only the conaninfo.txt file was uploaded out of the conan_package.tgz and conanmanifest.txt.

In that case the problematic comes when you want to upload a new package as ir fais becaouse of the following:

• Conan checks in the package upload that the package exist requesting the SNAPSHOT and only checking if there is something (in this case it is true, the conaninfo.txt is there)
• Then it tries to get the digest of the package to perform the diff (in this case it fails as there is no conanmanifest.txt)
• The conan_package.tgz is never uploaded as the client fails in the previous step.

To mitigate this I propose a behavior that is already being done in the recipe upload: Check that the manifest is in the snapshot (actually in the recipe upload it checks the availability of the digest and if not there -404- it assusmes there is no package uploaded).

I the digest is for some reason not available it assumes there is no existing package uploaded and it uploads everything.

---

Another useful change to mitigate this (not proposed at the time of opening this PR) is changing the order of the files upload. Now the files are uploaded in this order: conanmanifest.txt --> conaninfo.txt --> conan_package.tgz and it is done intetionally this way for some reason

conan/conans/client/rest/rest_client_v1.py

Line 40 in a52dcdc

def _download_files(self, file_urls, quiet=False):

From my point of view, the most problematic file during the upload is the conan_package.tgz and it should be uploaded in first place, then the conaninfo.txt and finally the conanmanifest.txt to wrap up the complete upload.

[danimtb]

Included some test to check also the search and install commands

[SSE4]

I don't think changing the order helps here. upload has to be transnational operation - conan either uploads everything or nothing. if it's interrupted during the upload (e.g. Ctrl-C pressed, conan crashed, network error occurred or whatever else happened), it should never finalize the transaction and files that were already uploaded or partially uploaded by that time should be discarded.

[danimtb]

That is the best solution but the upload is not done in one transaction but file by file and it seems that cannot be changed in the server side. I know this fix does not fix the issue but it tries to minimize it as much as we can

[SSE4]

still better than nothing. if it reduces the rate of problem occurrence, I fully support applying it.
however, that seems to be a design problem, which has to be solved may be in the next version of conan REST API, if it's ever planned.

[danimtb]

So finally I think I have a bunch of interesting tests here bringing some additional information to the original situation like the one described in #4672

To me, this is something that it is worth to have consistent behavior and probably a naive solution would be to check in install/download and upload of packages that the 3 files that define a package are correctly in place, although that would mean that if in the future we decide to remove one of those files the old clients will be broken (although this is something that would require a 2.0 or equivalent).

Having all this into account I would like to know your opinion or at least bring this discussion to one of our whiteboard sessions @conan-io/barbarians

PS: Don't worry about the prints in tests, those are just a proof of concept to get the idea. I will rework them when we agree on a solution 😜

[danimtb]

This test was failing before at this point https://github.com/conan-io/conan/pull/4662/files#diff-9e273c58f528ec321db5954d73096c3aR133 due to bad credentials for the remote. I am not sure if this PR is changing any behavior or the test was wrong 😕

[danimtb]

Added the changes to improve the integrity of packages in the complete loop of upload-search-install-upload.

The search couldn't be improved as the values returned by the server are the conaninfo content directly, so not much we can do.

Please take note that this makes an extra API call to check the snapshot in the package installation (download) from the server

[jgsogo]

LGTM, this will help the user to cope with some errors

[danimtb]

PR is ready. Please take a look so it can be merged