Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload packages in parallel #5856

Merged
merged 141 commits into from Nov 26, 2019
Merged

Upload packages in parallel #5856

merged 141 commits into from Nov 26, 2019

Conversation

@czoido
Copy link
Contributor

czoido commented Oct 2, 2019

Changelog: Feature: New argument of conan upload command --parallel to upload packages using multithreading.
Docs: conan-io/docs#1250

  • Uses a ThreadPool to parallelize upload across recipes and packages upload.
  • Fixes output so that output messages and progress bars do not overlap.
  • If --parallel option is present, the upload will use 8 threads to upload and use only one thread in the case it is missing.

Closes #3452

  • Refer to the issue that supports this Pull Request.
  • If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
  • I've read the Contributing guide.
  • I've followed the PEP8 style guides for Python code.
  • I've opened another PR in the Conan docs repo to the develop branch, documenting this one.

Note: By default this PR will skip the slower tests and will use a limited set of python versions. Check here how to increase the testing level by writing some tags in the current PR body text.

czoido added 30 commits Apr 4, 2019
czoido added 2 commits Oct 18, 2019
conans/client/cmd/uploader.py Outdated Show resolved Hide resolved
conans/util/progress_bar.py Outdated Show resolved Hide resolved
@memsharded memsharded removed this from the 1.20 milestone Oct 22, 2019
@memsharded

This comment has been minimized.

Copy link
Member

memsharded commented Oct 22, 2019

I have added a mutex to avoid problems while loading hooks.

This PR is failing, not because of the PR itself but because of the ConanApiAuthManager layer, that is not designed for concurrency

I have opened #5957, which must be fixed first before this PR can be merged.

@lasote lasote added this to the 1.21 milestone Nov 19, 2019
@memsharded

This comment has been minimized.

Copy link
Member

memsharded commented Nov 19, 2019

I have merged develop and pushed. Then testing a little bit:

  • seems more robust, all seems working fine.
  • The final output messages become useless when parallel, it is impossible to know what has been uploaded. Lots of interleaved Uploading conan_package.tgz completed [10943.20k] messages. I guess they should be replaced with something like Uploaded pkg/0.1:ID conan_package.tgz [...]
  • I haven't been able to make it fail with Artifactory, but it is also true that the deduplication of Artifactory has made difficult to repeat the testing several times.
conans/util/progress_bar.py Outdated Show resolved Hide resolved
Copy link
Member

memsharded left a comment

Good!
Make sure to do a last round of manual testing (py27, py36, a couple of platforms) to check everything is good, and can be merged.
Create a UX issue to improve:

  • the text of the downloaded messages, something that is useful as a log of what exactly has been uploaded
  • In platforms like CI or cmake.conan, the output is a few lines of blank text, which reads weird. Try if that output can be improved too.
@memsharded memsharded removed their assignment Nov 21, 2019
@lasote

This comment has been minimized.

Copy link
Contributor

lasote commented Nov 22, 2019

@czoido create the issue and merge.

@czoido czoido mentioned this pull request Nov 25, 2019
1 of 1 task complete
@czoido czoido merged commit b9f82c3 into conan-io:develop Nov 26, 2019
2 checks passed
2 checks passed
continuous-integration/jenkins/pr-head This commit looks good
Details
license/cla Contributor License Agreement is signed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.