Run the pytest testsuite in parallel #1418

utkbansal · 2017-06-21T14:28:45Z

Fixes #

Changes made in this Pull Request:

use pytest-xdist to run in parallel on 2 cores

PR Checklist

Tests?
Docs?
CHANGELOG updated?
Issue raised/referenced?

kain88-de · 2017-06-21T19:24:30Z

.travis.yml

@@ -24,15 +24,15 @@ env:
    - COVERALLS=false
    - NOSE_FLAGS="--processes=2 --process-timeout=400 --no-open-files --with-timer --timer-top-n 50"
    - NOSE_TEST_LIST="analysis auxiliary coordinates core formats topology utils"
-    - PYTEST_FLAGS="--disable-pytest-warnings"
+    - PYTEST_FLAGS="--disable-pytest-warnings -n 2"


can you please use --numprocesses it's more explicit and I'm trying to get upstream ci-helpers to detect that to auto install pytest-xdist.

Sure. I'll just let this build complete and then push.

@kain88-de That doesn't work. I don't think there is anything like --numprocesses

utkbansal · 2017-06-21T19:44:57Z

@kain88-de Should be ready to merge. No drop in coverage.

richardjgowers

Looks like coverage isn't working in parallel

Coverage.py warning: Module MDAnalysis was never imported. (module-not-imported)

Coverage.py warning: No data was collected. (no-data-collected)

utkbansal · 2017-06-22T13:38:04Z

This is weird, I think I saw no drop in coverage last night. Let me cross check.

utkbansal · 2017-06-22T14:23:52Z

I wonder how pytest shows the percentage of coverage when it did not collect data.

utkbansal · 2017-06-22T16:44:30Z

@richardjgowers @jbarnoud @kain88-de @mnmelo I have a wild theory - the warnings might just be meaningless.

I ran pytest on the lib module with coverage, once serially and once in parallel - both times the coverage reported was the same. I assumed that the difference might be too small and it might be a round off error since the coverage was an integer.

So, I ran the coordinates module - because it is large in terms of the number of tests. The reasoning was that the difference in coverage would be larger and there should be a tangible difference even after rounding off to the nearest integer. I still got the same coverage in both cases - 41%

Commands I used -

Parallel: py.test --cov=MDAnalysis coordinates --cov-report=html -q --disable-pytest-warnings -n 2
Serial: py.test --cov=MDAnalysis coordinates --cov-report=html -q --disable-pytest-warnings

Can someone run these commands locally (off the port coordinates PR) and confirm my findings?

utkbansal · 2017-06-22T17:35:38Z

So the build passed. The coverage is still at 88.837% Was there a drop? The button on the README file shows the same number.

kain88-de · 2017-06-22T19:50:07Z

.travis.yml

    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats"
    - NOSE_COVERAGE_FILE="nose_coverage"
    - PYTEST_COVERAGE_FILE="pytest_coverage"
    - MAIN_CMD="pytest ${PYTEST_LIST} ${PYTEST_FLAGS}; python ./testsuite/MDAnalysisTests/mda_nosetests ${NOSE_TEST_LIST} ${NOSE_FLAGS}"
    - SETUP_CMD=""
    - BUILD_CMD="pip install -v package/ && pip install testsuite/"
-    - CONDA_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer pytest=3.1.2 pytest-cov=2.5.1"
-    - CONDA_ALL_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer matplotlib netcdf4 scikit-learn scipy seaborn coveralls clustalw=2.1 pytest=3.1.2 pytest-cov=2.5.1"
+    - CONDA_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer pytest=3.1.2 pytest-cov=2.5.1 pytest-xdist=1.17.1"


Could you please remove the version pinning. I'm trying to get everything updated to we can take use of ci-helpers magic to autoinstall things for us. You should be able to remove the pytest explicit dependency as well.

I'm hijacking this PR to clean up some installations of pytest.

utkbansal · 2017-06-23T15:04:18Z

@kain88-de @jbarnoud @richardjgowers I don't really like the way pytest displays its output (listing each file and the other stuff). I'd like it to be more like what nose does currently. Can I add the -q flag to minimize the noise?

richardjgowers

Yeah, looks like we are getting two coverage reports, despite being told for the pytest version that we didn't gather coverage. Maybe it's a screw up between the parallel processes in pytest not communicating with each other.

kain88-de · 2017-06-24T10:36:37Z

.travis.yml

    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats"
    - NOSE_COVERAGE_FILE="nose_coverage"
    - PYTEST_COVERAGE_FILE="pytest_coverage"
    - MAIN_CMD="pytest ${PYTEST_LIST} ${PYTEST_FLAGS}; python ./testsuite/MDAnalysisTests/mda_nosetests ${NOSE_TEST_LIST} ${NOSE_FLAGS}"
    - SETUP_CMD=""
    - BUILD_CMD="pip install -v package/ && pip install testsuite/"
-    - CONDA_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer matplotlib scipy griddataformats pytest=3.1.2 pytest-cov=2.5.1"
-    - CONDA_ALL_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer matplotlib netcdf4 scikit-learn scipy griddataformats seaborn coveralls clustalw=2.1 pytest=3.1.2 pytest-cov=2.5.1"
+    - CONDA_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer matplotlib scipy griddataformats pytest-cov pytest-xdist"


@utkbansal you should also be able to drop the pytest-cov package as an explicit dependency. See astropy/ci-helpers#216

So here do we need to list pytest-cov? And don't we need to list pytest?

richardjgowers · 2017-06-24T11:26:18Z

@utkbansal rebase this and we're good to go

utkbansal · 2017-06-24T11:55:26Z

@richardjgowers The rebase for this depends on what we want to do about #1414

It wasn't supposed to be merged yet.

Should I open a new PR for the coordinates module and fix coverage in that? Then I can drop coordinates from NOSE_TEST_LIST here
Or if we will revert #1414 then, I have to drop coordinates from PYTEST_TEST_LIST.

utkbansal · 2017-06-24T12:06:29Z

@richardjgowers @jbarnoud @kain88-de Something has gone wrong while merging my PRs.

All the builds on the develop branch and my PR are failing with the following error -

import MDAnalysis as mda
E     File "/Users/utkbansal/Code/mdanalysis/package/MDAnalysis/__init__.py", line 148
E       import logging
E            ^
E   SyntaxError: invalid syntax

jbarnoud · 2017-06-24T12:13:09Z

@kain88-de @richardjgowers This is the commit responsible for the error. Line 146 of __init__.py is broken.

jbarnoud · 2017-06-24T12:14:15Z

I forgot to paste the link to the commit: d99abba

richardjgowers · 2017-06-24T12:45:24Z

We won't revert 1414

utkbansal · 2017-06-25T10:15:53Z

This should be ready too once the build completes.

richardjgowers · 2017-06-25T10:36:34Z

@utkbansal needs a rebase thanks to utils

utkbansal · 2017-06-25T11:13:09Z

Now how did utils slip away!

richardjgowers

This reverts utils

utkbansal · 2017-06-25T11:19:10Z

@richardjgowers Got it back.

utkbansal · 2017-06-25T13:02:53Z

@richardjgowers Looks good to go. The full build fails due to the time limit though.

kain88-de reviewed Jun 21, 2017

View changes

utkbansal force-pushed the utkbansal:pytest-parallel branch from 2d5fe43 to ebfc117 Jun 21, 2017

richardjgowers requested changes Jun 22, 2017

View changes

orbeckst added testing Continuous Integration and removed Continuous Integration labels Jun 22, 2017

kain88-de reviewed Jun 22, 2017

View changes

utkbansal force-pushed the utkbansal:pytest-parallel branch from 0b8f1c3 to 1546462 Jun 23, 2017

richardjgowers approved these changes Jun 23, 2017

View changes

kain88-de reviewed Jun 24, 2017

View changes

utkbansal force-pushed the utkbansal:pytest-parallel branch from 1546462 to 575aabd Jun 24, 2017

jbarnoud mentioned this pull request Jun 24, 2017

Fix a syntax error in MDAnalysis.__init__ #1432

Merged

utkbansal force-pushed the utkbansal:pytest-parallel branch from f8f28a0 to d599baf Jun 25, 2017

utkbansal added 3 commits Jun 21, 2017

Use pytest-xdist to run test in parallel

Verified

This commit was signed with a verified signature.

utkbansal Utkarsh Bansal

GPG key ID: 359A4CFC0731150A Learn about signing commits

02bf97b

Adds pytest-xdist=1.17.1 to CONDA_ALL_DEPENDENCIES

Verified

This commit was signed with a verified signature.

utkbansal Utkarsh Bansal

GPG key ID: 359A4CFC0731150A Learn about signing commits

6d49bd1

Adds the explicit flag

Verified

This commit was signed with a verified signature.

utkbansal Utkarsh Bansal

GPG key ID: 359A4CFC0731150A Learn about signing commits

51cb646

utkbansal added 2 commits Jun 22, 2017

Revert back processes flag

Verified

This commit was signed with a verified signature.

utkbansal Utkarsh Bansal

GPG key ID: 359A4CFC0731150A Learn about signing commits

c3f7e2d

Remove verison pinning

Verified

This commit was signed with a verified signature.

utkbansal Utkarsh Bansal

GPG key ID: 359A4CFC0731150A Learn about signing commits

Loading status checks…

7f9cd3d

utkbansal force-pushed the utkbansal:pytest-parallel branch from d599baf to 7f9cd3d Jun 25, 2017

Add topology to pytest list

Verified

This commit was signed with a verified signature.

utkbansal Utkarsh Bansal

GPG key ID: 359A4CFC0731150A Learn about signing commits

Loading status checks…

f6654e0

Add utils back to pytest list

Verified

This commit was signed with a verified signature.

utkbansal Utkarsh Bansal

GPG key ID: 359A4CFC0731150A Learn about signing commits

Loading status checks…

a98924a

richardjgowers requested changes Jun 25, 2017

View changes

richardjgowers approved these changes Jun 25, 2017

View changes

richardjgowers merged commit 2d969b7 into MDAnalysis:develop Jun 25, 2017
1 of 2 checks passed

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details

QuantifiedCode No new issues introduced.
Details

MDAnalysis / mdanalysis

Run the pytest testsuite in parallel #1418

Run the pytest testsuite in parallel #1418

utkbansal commented Jun 21, 2017

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

utkbansal commented Jun 21, 2017

richardjgowers left a comment

utkbansal commented Jun 22, 2017

utkbansal commented Jun 22, 2017 •

edited

utkbansal commented Jun 22, 2017

utkbansal commented Jun 22, 2017

This comment has been minimized.

utkbansal commented Jun 23, 2017

richardjgowers left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

richardjgowers commented Jun 24, 2017

utkbansal commented Jun 24, 2017

utkbansal commented Jun 24, 2017

jbarnoud commented Jun 24, 2017

jbarnoud commented Jun 24, 2017

richardjgowers commented Jun 24, 2017

utkbansal commented Jun 25, 2017

richardjgowers commented Jun 25, 2017

utkbansal commented Jun 25, 2017 •

edited

richardjgowers left a comment

utkbansal commented Jun 25, 2017

utkbansal commented Jun 25, 2017

MDAnalysis / mdanalysis

Run the pytest testsuite in parallel #1418

Run the pytest testsuite in parallel #1418

Conversation

utkbansal commented Jun 21, 2017

PR Checklist

This comment has been minimized.

kain88-de Jun 21, 2017 Member

This comment has been minimized.

utkbansal Jun 21, 2017 Author Member

This comment has been minimized.

utkbansal Jun 22, 2017 Author Member

utkbansal commented Jun 21, 2017

richardjgowers left a comment

utkbansal commented Jun 22, 2017

utkbansal commented Jun 22, 2017 • edited

utkbansal commented Jun 22, 2017

utkbansal commented Jun 22, 2017

This comment has been minimized.

kain88-de Jun 22, 2017 Member

utkbansal commented Jun 23, 2017

richardjgowers left a comment

This comment has been minimized.

kain88-de Jun 24, 2017 Member

This comment has been minimized.

utkbansal Jun 24, 2017 Author Member

This comment has been minimized.

richardjgowers Jun 25, 2017 Member

richardjgowers commented Jun 24, 2017

utkbansal commented Jun 24, 2017

utkbansal commented Jun 24, 2017

jbarnoud commented Jun 24, 2017

jbarnoud commented Jun 24, 2017

richardjgowers commented Jun 24, 2017

utkbansal commented Jun 25, 2017

richardjgowers commented Jun 25, 2017

utkbansal commented Jun 25, 2017 • edited

richardjgowers left a comment

utkbansal commented Jun 25, 2017

utkbansal commented Jun 25, 2017

kain88-de Jun 21, 2017
Member

utkbansal Jun 21, 2017
Author Member

utkbansal Jun 22, 2017
Author Member

utkbansal commented Jun 22, 2017 •

edited

kain88-de Jun 22, 2017
Member

kain88-de Jun 24, 2017
Member

utkbansal Jun 24, 2017
Author Member

richardjgowers Jun 25, 2017
Member

utkbansal commented Jun 25, 2017 •

edited