Run the pytest testsuite in parallel #1418

utkbansal · 2017-06-21T14:28:45Z

Fixes #

Changes made in this Pull Request:

use pytest-xdist to run in parallel on 2 cores

PR Checklist

Tests?
Docs?
CHANGELOG updated?
Issue raised/referenced?

kain88-de · 2017-06-21T19:24:30Z

.travis.yml

@@ -24,15 +24,15 @@ env:
    - COVERALLS=false
    - NOSE_FLAGS="--processes=2 --process-timeout=400 --no-open-files --with-timer --timer-top-n 50"
    - NOSE_TEST_LIST="analysis auxiliary coordinates core formats topology utils"
-    - PYTEST_FLAGS="--disable-pytest-warnings"
+    - PYTEST_FLAGS="--disable-pytest-warnings -n 2"


can you please use --numprocesses it's more explicit and I'm trying to get upstream ci-helpers to detect that to auto install pytest-xdist.

Sure. I'll just let this build complete and then push.

@kain88-de That doesn't work. I don't think there is anything like --numprocesses

utkbansal · 2017-06-21T19:44:57Z

@kain88-de Should be ready to merge. No drop in coverage.

richardjgowers

Looks like coverage isn't working in parallel

Coverage.py warning: Module MDAnalysis was never imported. (module-not-imported)

Coverage.py warning: No data was collected. (no-data-collected)

utkbansal · 2017-06-22T13:38:04Z

This is weird, I think I saw no drop in coverage last night. Let me cross check.

utkbansal · 2017-06-22T14:23:52Z

I wonder how pytest shows the percentage of coverage when it did not collect data.

utkbansal · 2017-06-22T16:44:30Z

@richardjgowers @jbarnoud @kain88-de @mnmelo I have a wild theory - the warnings might just be meaningless.

I ran pytest on the lib module with coverage, once serially and once in parallel - both times the coverage reported was the same. I assumed that the difference might be too small and it might be a round off error since the coverage was an integer.

So, I ran the coordinates module - because it is large in terms of the number of tests. The reasoning was that the difference in coverage would be larger and there should be a tangible difference even after rounding off to the nearest integer. I still got the same coverage in both cases - 41%

Commands I used -

Parallel: py.test --cov=MDAnalysis coordinates --cov-report=html -q --disable-pytest-warnings -n 2
Serial: py.test --cov=MDAnalysis coordinates --cov-report=html -q --disable-pytest-warnings

Can someone run these commands locally (off the port coordinates PR) and confirm my findings?

utkbansal · 2017-06-22T17:35:38Z

So the build passed. The coverage is still at 88.837% Was there a drop? The button on the README file shows the same number.

kain88-de · 2017-06-22T19:50:07Z

.travis.yml

    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats"
    - NOSE_COVERAGE_FILE="nose_coverage"
    - PYTEST_COVERAGE_FILE="pytest_coverage"
    - MAIN_CMD="pytest ${PYTEST_LIST} ${PYTEST_FLAGS}; python ./testsuite/MDAnalysisTests/mda_nosetests ${NOSE_TEST_LIST} ${NOSE_FLAGS}"
    - SETUP_CMD=""
    - BUILD_CMD="pip install -v package/ && pip install testsuite/"
-    - CONDA_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer pytest=3.1.2 pytest-cov=2.5.1"
-    - CONDA_ALL_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer matplotlib netcdf4 scikit-learn scipy seaborn coveralls clustalw=2.1 pytest=3.1.2 pytest-cov=2.5.1"
+    - CONDA_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer pytest=3.1.2 pytest-cov=2.5.1 pytest-xdist=1.17.1"


Could you please remove the version pinning. I'm trying to get everything updated to we can take use of ci-helpers magic to autoinstall things for us. You should be able to remove the pytest explicit dependency as well.

I'm hijacking this PR to clean up some installations of pytest.

utkbansal · 2017-06-23T15:04:18Z

@kain88-de @jbarnoud @richardjgowers I don't really like the way pytest displays its output (listing each file and the other stuff). I'd like it to be more like what nose does currently. Can I add the -q flag to minimize the noise?

richardjgowers

Yeah, looks like we are getting two coverage reports, despite being told for the pytest version that we didn't gather coverage. Maybe it's a screw up between the parallel processes in pytest not communicating with each other.

kain88-de · 2017-06-24T10:36:37Z

.travis.yml

    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats"
    - NOSE_COVERAGE_FILE="nose_coverage"
    - PYTEST_COVERAGE_FILE="pytest_coverage"
    - MAIN_CMD="pytest ${PYTEST_LIST} ${PYTEST_FLAGS}; python ./testsuite/MDAnalysisTests/mda_nosetests ${NOSE_TEST_LIST} ${NOSE_FLAGS}"
    - SETUP_CMD=""
    - BUILD_CMD="pip install -v package/ && pip install testsuite/"
-    - CONDA_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer matplotlib scipy griddataformats pytest=3.1.2 pytest-cov=2.5.1"
-    - CONDA_ALL_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer matplotlib netcdf4 scikit-learn scipy griddataformats seaborn coveralls clustalw=2.1 pytest=3.1.2 pytest-cov=2.5.1"
+    - CONDA_DEPENDENCIES="mmtf-python nose=1.3.7 mock six biopython networkx cython joblib nose-timer matplotlib scipy griddataformats pytest-cov pytest-xdist"


@utkbansal you should also be able to drop the pytest-cov package as an explicit dependency. See astropy/ci-helpers#216

So here do we need to list pytest-cov? And don't we need to list pytest?

richardjgowers · 2017-06-24T11:26:18Z

@utkbansal rebase this and we're good to go

utkbansal · 2017-06-24T11:55:26Z

@richardjgowers The rebase for this depends on what we want to do about #1414

It wasn't supposed to be merged yet.

Should I open a new PR for the coordinates module and fix coverage in that? Then I can drop coordinates from NOSE_TEST_LIST here
Or if we will revert Port coordinates module #1414 then, I have to drop coordinates from PYTEST_TEST_LIST.

utkbansal · 2017-06-24T12:06:29Z

@richardjgowers @jbarnoud @kain88-de Something has gone wrong while merging my PRs.

All the builds on the develop branch and my PR are failing with the following error -

import MDAnalysis as mda
E     File "/Users/utkbansal/Code/mdanalysis/package/MDAnalysis/__init__.py", line 148
E       import logging
E            ^
E   SyntaxError: invalid syntax

jbarnoud · 2017-06-24T12:13:09Z

@kain88-de @richardjgowers This is the commit responsible for the error. Line 146 of __init__.py is broken.

jbarnoud · 2017-06-24T12:14:15Z

I forgot to paste the link to the commit: d99abba

richardjgowers · 2017-06-24T12:45:24Z

We won't revert 1414

utkbansal · 2017-06-25T10:15:53Z

This should be ready too once the build completes.

richardjgowers · 2017-06-25T10:36:34Z

@utkbansal needs a rebase thanks to utils

utkbansal · 2017-06-25T11:13:09Z

Now how did utils slip away!

richardjgowers

This reverts utils

utkbansal · 2017-06-25T11:19:10Z

@richardjgowers Got it back.

utkbansal · 2017-06-25T13:02:53Z

@richardjgowers Looks good to go. The full build fails due to the time limit though.

kain88-de reviewed Jun 21, 2017

View reviewed changes

utkbansal force-pushed the pytest-parallel branch from 2d5fe43 to ebfc117 Compare June 21, 2017 19:44

richardjgowers requested changes Jun 22, 2017

View reviewed changes

orbeckst added testing Continuous Integration and removed Continuous Integration labels Jun 22, 2017

kain88-de reviewed Jun 22, 2017

View reviewed changes

utkbansal force-pushed the pytest-parallel branch from 0b8f1c3 to 1546462 Compare June 23, 2017 13:53

richardjgowers approved these changes Jun 23, 2017

View reviewed changes

kain88-de reviewed Jun 24, 2017

View reviewed changes

utkbansal force-pushed the pytest-parallel branch from 1546462 to 575aabd Compare June 24, 2017 11:48

jbarnoud mentioned this pull request Jun 24, 2017

Fix a syntax error in MDAnalysis.__init__ #1432

Merged

utkbansal force-pushed the pytest-parallel branch from f8f28a0 to d599baf Compare June 25, 2017 10:08

utkbansal added 3 commits June 25, 2017 16:10

Use pytest-xdist to run test in parallel

02bf97b

Adds pytest-xdist=1.17.1 to CONDA_ALL_DEPENDENCIES

6d49bd1

Adds the explicit flag

51cb646

utkbansal added 2 commits June 25, 2017 16:12

Revert back processes flag

c3f7e2d

Remove verison pinning

7f9cd3d

utkbansal force-pushed the pytest-parallel branch from d599baf to 7f9cd3d Compare June 25, 2017 10:44

Add topology to pytest list

f6654e0

Add utils back to pytest list

a98924a

richardjgowers requested changes Jun 25, 2017

View reviewed changes

richardjgowers approved these changes Jun 25, 2017

View reviewed changes

richardjgowers merged commit 2d969b7 into MDAnalysis:develop Jun 25, 2017

Run the pytest testsuite in parallel #1418

Run the pytest testsuite in parallel #1418

utkbansal commented Jun 21, 2017

kain88-de Jun 21, 2017

utkbansal Jun 21, 2017

utkbansal Jun 22, 2017

utkbansal commented Jun 21, 2017

richardjgowers left a comment

utkbansal commented Jun 22, 2017

utkbansal commented Jun 22, 2017 •

edited

utkbansal commented Jun 22, 2017

utkbansal commented Jun 22, 2017

kain88-de Jun 22, 2017

utkbansal commented Jun 23, 2017

richardjgowers left a comment

kain88-de Jun 24, 2017

utkbansal Jun 24, 2017

richardjgowers Jun 25, 2017

richardjgowers commented Jun 24, 2017

utkbansal commented Jun 24, 2017

utkbansal commented Jun 24, 2017

jbarnoud commented Jun 24, 2017

jbarnoud commented Jun 24, 2017

richardjgowers commented Jun 24, 2017

utkbansal commented Jun 25, 2017

richardjgowers commented Jun 25, 2017

utkbansal commented Jun 25, 2017 •

edited

richardjgowers left a comment

utkbansal commented Jun 25, 2017

utkbansal commented Jun 25, 2017

Run the pytest testsuite in parallel #1418

Run the pytest testsuite in parallel #1418

Conversation

utkbansal commented Jun 21, 2017

PR Checklist

kain88-de Jun 21, 2017

Choose a reason for hiding this comment

utkbansal Jun 21, 2017

Choose a reason for hiding this comment

utkbansal Jun 22, 2017

Choose a reason for hiding this comment

utkbansal commented Jun 21, 2017

richardjgowers left a comment

Choose a reason for hiding this comment

utkbansal commented Jun 22, 2017

utkbansal commented Jun 22, 2017 • edited

utkbansal commented Jun 22, 2017

utkbansal commented Jun 22, 2017

kain88-de Jun 22, 2017

Choose a reason for hiding this comment

utkbansal commented Jun 23, 2017

richardjgowers left a comment

Choose a reason for hiding this comment

kain88-de Jun 24, 2017

Choose a reason for hiding this comment

utkbansal Jun 24, 2017

Choose a reason for hiding this comment

richardjgowers Jun 25, 2017

Choose a reason for hiding this comment

richardjgowers commented Jun 24, 2017

utkbansal commented Jun 24, 2017

utkbansal commented Jun 24, 2017

jbarnoud commented Jun 24, 2017

jbarnoud commented Jun 24, 2017

richardjgowers commented Jun 24, 2017

utkbansal commented Jun 25, 2017

richardjgowers commented Jun 25, 2017

utkbansal commented Jun 25, 2017 • edited

richardjgowers left a comment

Choose a reason for hiding this comment

utkbansal commented Jun 25, 2017

utkbansal commented Jun 25, 2017

utkbansal commented Jun 22, 2017 •

edited

utkbansal commented Jun 25, 2017 •

edited