Add concurrent package validation #48
Conversation
- Use `multiprocessing` from the std lib to concurrently validate packages. Add a cli argument `num-threads` setting the number of concurrent processes to be used. - Add test for three concurrent scenarios: `num_threads=0` using all available cores, `num_threads=1` for serial validation, `num_threads=4` explicitly using four processes. - Adapt travis script properly measure test coverage with concurrent validation.
Codecov Report
@@ Coverage Diff @@
## master #48 +/- ##
==========================================
+ Coverage 94.06% 94.53% +0.47%
==========================================
Files 2 2
Lines 219 238 +19
==========================================
+ Hits 206 225 +19
Misses 13 13
Continue to review full report at Codecov.
|
| are not in `repodata` and also any packages that fail the package | ||
| validation | ||
| NOTE2: In concurrent mode (num_threads is not 1) this might be hard to kill | ||
| using CTRL-C. |
There was a problem hiding this comment.
it's actually pretty simple to kill with ctrl+c. I just hit ctrl+c a few times and everything died appropriately ¯_(ツ)_/¯
| @@ -456,7 +521,8 @@ def main(upstream_channel, target_directory, temp_directory, platform, | |||
| # 1. validate local repo | |||
| # validating all packages is taking many hours. | |||
| # _validate_packages(repodata=repodata, | |||
| # package_directory=local_directory) | |||
| # package_directory=local_directory, | |||
| # num_threads=num_threads) | |||
There was a problem hiding this comment.
Shame on me for leaving commented code in this project.
👍 to you for updating commented code with the new API 😀
| @@ -41,7 +41,8 @@ def test_version(): | |||
| @pytest.mark.parametrize( | |||
| 'channel,platform', | |||
| itertools.product([anaconda_channel, 'conda-forge'], ['linux-64'])) | |||
| def test_cli(tmpdir, channel, platform, repodata): | |||
| @pytest.mark.parametrize('num_threads', [0, 1, 4]) | |||
There was a problem hiding this comment.
Did not know you could combine multiple parametrize decorators. Neat!
|
works locally. Thanks @willirath ! Merging, tagging and deploying a new version to pypi/conda-forge |
|
Outstanding! 👍 |
|
should be able to pip install 0.7.0 which has the additional command line flag |
|
conda package for 0.7.0 should be available in the next week or so. |
Use
multiprocessingfrom the std lib to concurrently validatepackages. Add a cli argument
num-threadssetting the number ofconcurrent processes to be used.
Add test for three concurrent scenarios:
num_threads=0using allavailable cores,
num_threads=1for serial validation,num_threads=4explicitly using four processes.Adapt Travis script to properly measure test coverage with concurrent
validation.
This is a cleaned-up version of #45.