Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload packages in parallel #5856

Merged
merged 141 commits into from Nov 26, 2019
Merged

Upload packages in parallel #5856

merged 141 commits into from Nov 26, 2019

Conversation

czoido
Copy link
Contributor

@czoido czoido commented Oct 2, 2019

Changelog: Feature: New argument of conan upload command --parallel to upload packages using multithreading.
Docs: conan-io/docs#1250

  • Uses a ThreadPool to parallelize upload across recipes and packages upload.
  • Fixes output so that output messages and progress bars do not overlap.
  • If --parallel option is present, the upload will use 8 threads to upload and use only one thread in the case it is missing.

Closes #3452

  • Refer to the issue that supports this Pull Request.
  • If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
  • I've read the Contributing guide.
  • I've followed the PEP8 style guides for Python code.
  • I've opened another PR in the Conan docs repo to the develop branch, documenting this one.

Note: By default this PR will skip the slower tests and will use a limited set of python versions. Check here how to increase the testing level by writing some tags in the current PR body text.

czoido added 30 commits Apr 4, 2019
conans/client/cmd/uploader.py Outdated Show resolved Hide resolved
conans/test/functional/command/upload_complete_test.py Outdated Show resolved Hide resolved
conans/util/progress_bar.py Outdated Show resolved Hide resolved
conans/test/functional/command/upload_complete_test.py Outdated Show resolved Hide resolved
@memsharded memsharded removed this from the 1.20 milestone Oct 22, 2019
@memsharded
Copy link
Member

@memsharded memsharded commented Oct 22, 2019

I have added a mutex to avoid problems while loading hooks.

This PR is failing, not because of the PR itself but because of the ConanApiAuthManager layer, that is not designed for concurrency

I have opened #5957, which must be fixed first before this PR can be merged.

@lasote lasote added this to the 1.21 milestone Nov 19, 2019
@memsharded
Copy link
Member

@memsharded memsharded commented Nov 19, 2019

I have merged develop and pushed. Then testing a little bit:

  • seems more robust, all seems working fine.
  • The final output messages become useless when parallel, it is impossible to know what has been uploaded. Lots of interleaved Uploading conan_package.tgz completed [10943.20k] messages. I guess they should be replaced with something like Uploaded pkg/0.1:ID conan_package.tgz [...]
  • I haven't been able to make it fail with Artifactory, but it is also true that the deduplication of Artifactory has made difficult to repeat the testing several times.

conans/util/progress_bar.py Outdated Show resolved Hide resolved
Copy link
Member

@memsharded memsharded left a comment

Good!
Make sure to do a last round of manual testing (py27, py36, a couple of platforms) to check everything is good, and can be merged.
Create a UX issue to improve:

  • the text of the downloaded messages, something that is useful as a log of what exactly has been uploaded
  • In platforms like CI or cmake.conan, the output is a few lines of blank text, which reads weird. Try if that output can be improved too.

@memsharded memsharded removed their assignment Nov 21, 2019
@lasote
Copy link
Contributor

@lasote lasote commented Nov 22, 2019

@czoido create the issue and merge.

@czoido czoido merged commit b9f82c3 into conan-io:develop Nov 26, 2019
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants