Port analysis module to pytest #1450

utkbansal · 2017-07-02T22:08:18Z

Fixes #

Changes made in this Pull Request:

PR Checklist

Tests?
Docs?
CHANGELOG updated?
Issue raised/referenced?

utkbansal · 2017-07-02T22:34:45Z

There is some unusual stuff going on in the TestHydrogenBondAnalysisChecking class. Can someone explain?

richardjgowers · 2017-07-03T07:33:21Z

@utkbansal not sure, but it might be because of self.universe being shared? Try replacing _setUp with a fixture, and make the _run method not operate on self variables

kain88-de · 2017-07-03T11:03:51Z

@utkbansal looks like you have to replace assertRaises and assert_warns with pytest equivalents.

utkbansal · 2017-07-03T14:33:04Z

@richardjgowers @kain88-de What should I do with this setup?

class TestEncoreImportWarnings(object):
    def setUp(self):
        # clear cache of encore module
        for mod in list(sys.modules):  # list as we're changing as we iterate
            if 'encore' in mod:
                sys.modules.pop(mod, None)

    @block_import('sklearn')
    def _check_sklearn_import_warns(self, package):
        warnings.simplefilter('always')
        assert_warns(ImportWarning, importlib.import_module, package)

    def test_import_warnings(self):
        for pkg in (
                'MDAnalysis.analysis.encore.dimensionality_reduction.DimensionalityReductionMethod',
                'MDAnalysis.analysis.encore.clustering.ClusteringMethod',
        ):
            yield self._check_sklearn_import_warns, pkg

This isn't something that a fixture is used for. This literally is some setup stuff. One option is to name this method setup and call it in every method. Any other non-hacky solutions?

jbarnoud · 2017-07-03T16:49:20Z

I did not test, but you should be able to do something like:

@pytest.fixture()
def remove_encore_module():
    # clear cache of encore module
    for mod in list(sys.modules):  # list as we're changing as we iterate
        if 'encore' in mod:
            sys.modules.pop(mod, None)

@pytest.mark.usefixtures('remove_encore_module')
class TestEncoreImportWarnings(TestCase):
    ...

From what I understand, the @pytest.mark.usefixtures decorator make sure the fixture is called but do not gives you access to the result of the fixture directly. Since the fixture returns nothing (None actually), it does not matter that you cannot access it; but you do not polute your tests with a useless argument to call the fixture.

Once again, I did not test so I might be wrong.

utkbansal · 2017-07-08T17:10:41Z

TestHydrogenBondAnalysisChecking is ultra-confusing. Other than that, everything else works.

jbarnoud · 2017-07-08T21:32:25Z

testsuite/MDAnalysisTests/analysis/test_encore.py

        for pkg in (
                'MDAnalysis.analysis.encore.dimensionality_reduction.DimensionalityReductionMethod',
                'MDAnalysis.analysis.encore.clustering.ClusteringMethod',
        ):
-            yield self._check_sklearn_import_warns, pkg
+            self._check_sklearn_import_warns(pkg, recwarn)


This changes the behaviour of the test. Instead of having 2 tests that can fail individually, there is only one test that has two consecutive asserts. It is fine for now, but please add a comment as it may be forgotten when you will move to proper pytest idioms.

The comment is still missing here.

The comment is bellow. I missed it.

jbarnoud · 2017-07-09T06:45:36Z

TestHydrogenBondAnalysisChecking is ultra-confusing

It is confusing indeed, but what do you find confusing there?

jbarnoud · 2017-07-11T17:24:33Z

Please prioritize this PR so we can get rid of nose.

orbeckst

Smaller things in the comments.

For the problem with test_hbonds.py I suggest you run just those tests locally. Looking at https://travis-ci.org/MDAnalysis/mdanalysis/jobs/251523905#L1441 some failures seem to come from something that pytest does:

E           AttributeError: 'TestHydrogenBondAnalysisChecking' object has no attribute 'universe'

Puzzling, because I don't think you touched TestHydrogenBondAnalysisChecking.

For the cryptic failures such as https://travis-ci.org/MDAnalysis/mdanalysis/jobs/251523905#L1302 add more details to the test output

try:
    ...
except Exception as err:
   raise AssertionError("... blabla failed. Exception: {}".format(err))

or similar. Then we might get a better idea where the problem lies. The hbond tests are admittedly not very good and just test that the code runs somehow but at least that should work. It would be better if they checked more specific things but that's outside the scope of your PR.

orbeckst · 2017-07-11T18:31:04Z

.travis.yml

    - PYTEST_FLAGS="--disable-pytest-warnings -n 2"
-    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats testsuite/MDAnalysisTests/coordinates testsuite/MDAnalysisTests/utils testsuite/MDAnalysisTests/topology testsuite/MDAnalysisTests/auxiliary testsuite/MDAnalysisTests/core"
+    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats testsuite/MDAnalysisTests/coordinates testsuite/MDAnalysisTests/utils testsuite/MDAnalysisTests/topology testsuite/MDAnalysisTests/auxiliary testsuite/MDAnalysisTests/core testsuite/MDAnalysisTests/analysis"


also remove nose from package/setup.py and testsuite/setup.py — yay, major step forward in your project: getting rid of nose.

And remove testsuite/mda_nosetests.py.

@mnmelo @jbarnoud @kain88-de @richardjgowers : should the nose-specific plugins also be removed at this time? We have to recreate the plugins for pytest or replace with native pytest plugins anyway,

@orbeckst these changes should be in a separate PR. The plugins can pretty much all go. There is the openfile plugin but otherwise pytest comes with better alternatives

orbeckst · 2017-07-11T18:31:47Z

.travis.yml

    - PYTEST_FLAGS="--disable-pytest-warnings -n 2"
-    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats testsuite/MDAnalysisTests/coordinates testsuite/MDAnalysisTests/utils testsuite/MDAnalysisTests/topology testsuite/MDAnalysisTests/auxiliary testsuite/MDAnalysisTests/core"
+    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats testsuite/MDAnalysisTests/coordinates testsuite/MDAnalysisTests/utils testsuite/MDAnalysisTests/topology testsuite/MDAnalysisTests/auxiliary testsuite/MDAnalysisTests/core testsuite/MDAnalysisTests/analysis"
    - NOSE_COVERAGE_FILE="nose_coverage"
    - PYTEST_COVERAGE_FILE="pytest_coverage"
    - MAIN_CMD="pytest ${PYTEST_LIST} ${PYTEST_FLAGS}; python ./testsuite/MDAnalysisTests/mda_nosetests ${NOSE_TEST_LIST} ${NOSE_FLAGS}"


remove use of mda_nosetests

I agree with @orbeckst. With NOSE_TEST_LIST being empty, the call to mda_nosetests becomes buggy and should be removed. Removing the dependencies to nose is out of scope, though.

orbeckst · 2017-07-11T18:36:07Z

testsuite/MDAnalysisTests/analysis/test_align.py

-class TestRotationMatrix(object):
-    def __init__(self):
+class TestRotationMatrix(TestCase):
+    def setUp(self):


@utkbansal I assume the goal of this PR is to make it run under pytest and then you will do a second round adapting it to pytest style? Is this correct? (Would be fine with me.)

yes that idea is correct

orbeckst · 2017-07-11T18:38:24Z

testsuite/MDAnalysisTests/analysis/test_rms.py

@@ -45,10 +48,10 @@
 warnings.simplefilter('always')


-class Testrmsd(object):
+class Testrmsd(TestCase):
    @dec.skipif(parser_not_found('DCD'),


remove DCD skipif, we have full DCD support now

better in a second PR

orbeckst · 2017-07-11T18:39:39Z

testsuite/MDAnalysisTests/analysis/test_rms.py

        reference = MDAnalysis.Universe(PSF, DCD)
        reference.atoms = reference.atoms[:-1]
-        RMSD = MDAnalysis.analysis.rms.RMSD(self.universe,
-                                            reference=reference)
+        with pytest.raises(SelectionError):


orbeckst · 2017-07-11T18:40:32Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

@@ -239,7 +241,7 @@ def run_HBA_dynamic_selections(*args):
            self._tearDown()


-class TestHydrogenBondAnalysisTIP3P(object):
+class TestHydrogenBondAnalysisTIP3P(TestCase):
    @dec.skipif(parser_not_found('DCD'),


remove DCD skipif

separate PR, as @kain88-de says

kain88-de · 2017-07-13T07:30:49Z

@utkbansal You should prioritize this PR. The other PR's are nice but not as important as this one. You still haven't told us your exact problem.

jbarnoud · 2017-07-13T13:33:05Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

@@ -207,9 +210,11 @@ def runOK():
                            raise AssertionError("HydrogenBondAnalysis protein/protein failed")
                        else:
                            return True
-                    yield runOK
+                    runOK()


This change the behaviour and transforms a test generator into a succession of assertions in a single test. Please, add a comment so you can restore the initial behaviour latter.

I am still working on it. This isn't final.

jbarnoud · 2017-07-13T13:33:19Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

-                    yield assert_raises, SelectionError, run_HBA, s1, s2, s1type
+                    # yield assert_raises, SelectionError, run_HBA, s1, s2, s1type
+                    with pytest.raises(SelectionError):
+                        run_HBA(s1, s2, s1type)


Same here. Add a comment.

jbarnoud · 2017-07-13T13:33:29Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

@@ -234,12 +239,12 @@ def run_HBA_dynamic_selections(*args):
                        raise AssertionError("HydrogenBondAnalysis with update=True failed")
                    else:
                        return True
-                yield run_HBA_dynamic_selections, s1, s2, s1type
+                run_HBA_dynamic_selections(s1, s2, s1type)


Idem. Comment.

utkbansal · 2017-07-13T14:02:12Z

@kain88-de @richardjgowers @jbarnoud This is the best hack I could come up with. Review please.

mnmelo · 2017-07-13T14:16:37Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

+    for s1, s2, s1type in itertools.product((protein, nothing),
+                                            (protein, nothing),
+                                            ("donor", "acceptor", "both")):
+        params.append((s1, s2, s1type))


Must this become a list? Won't pytest.mark.parametrize accept the itertools generator object?

even if this has to be a list. Why not list(itertools.product(...))

The iterator still works just fine.

import pytest import itertools @pytest.mark.parametrize('a, b', itertools.product(range(3), range(3))) def test_d(a, b): assert True

mnmelo · 2017-07-13T14:17:59Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

+    return params
+
+
+def build_parameters2():


I'm not getting the rationale behind the two build_parameters. Is it to have the option of eventually supplying different lists of test args to the two tests?

mnmelo · 2017-07-13T14:20:32Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

@@ -162,6 +162,28 @@ def setUp(self):
        self.values['acceptor_resnm'] = np.array([], dtype="<U3")


+def build_parameters():


Could you instead define protein, nothing and the itertools.product generator as class attributes of TestHydrogenBondAnalysisChecking? This way the the itertools.product will be available when the decorator is used, and there won't be any need for the build_parameters functions.

Can't access self in @pytest.mark.parametrize

I've already removed build_parameters()

Class attrs don't need to be accessed via self when in the class body. They're already in the namespace. Anyway, I think the solution of having the iterator call within the decorator is the best, as it clearer what's happening and allows customization in place.

So is the latest change ok?

I've done this

@pytest.mark.parametrize('s1, s2, s1type', itertools.product( ("protein", "resname ALA and not backbone"), ("protein", "resname ALA and not backbone"), ("donor", "acceptor", "both"), ))

utkbansal · 2017-07-13T15:29:56Z

nose is messing up now. It's trying to run everything.

utkbansal · 2017-07-13T15:53:26Z

@kain88-de @richardjgowers @jbarnoud The build technically passes because there are no errors in the pytest part.

richardjgowers

Yeah tests look ok now, the pytest half of things works. I'll let @orbeckst and @jbarnoud check their concerns

utkbansal · 2017-07-13T16:01:25Z

@richardjgowers So would be able to merge this as is(once everyone confirms the changes)? I'm already working on #1480 to remove nose specific stuff.

jbarnoud

Still a few comments, mostly about hbonds.

Also, I would like you to remove the calls to mda_nosetest in this PR to avoid breaking develop. Please do so on a different commit.

Finally, you could squash your commits from today (starting from "Quick fix"). Keep the commit that remove the calls to mda_nosetest separate.

jbarnoud · 2017-07-13T16:08:03Z

testsuite/MDAnalysisTests/analysis/test_encore.py

        for pkg in (
                'MDAnalysis.analysis.encore.dimensionality_reduction.DimensionalityReductionMethod',
                'MDAnalysis.analysis.encore.clustering.ClusteringMethod',
        ):
-            yield self._check_sklearn_import_warns, pkg
+            self._check_sklearn_import_warns(pkg, recwarn)


The comment is still missing here.

jbarnoud · 2017-07-13T16:20:32Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

+        protein = "protein"
+
+        if s1 == s2 == protein:
+            def runOK():


The way the test is written now, what is the point of having the nested function?

Nothing really. I can pull that out.

jbarnoud · 2017-07-13T16:23:20Z

testsuite/MDAnalysisTests/analysis/test_hbonds.py

+
+        self._tearDown()
+
+    def run_HBA2(self, s1, s2, s1type):


I do not like having functions named run_HBA and run_HBA2. It was OK that they had the same name when they were nested because the parent function was giving the context. Now the names do not tell how the two functions differ from each other. run_HBA_no_update and run_HBA_update would be better names, other names may work as well.

jbarnoud · 2017-07-13T16:34:43Z

.travis.yml

    - PYTEST_FLAGS="--disable-pytest-warnings -n 2"
-    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats testsuite/MDAnalysisTests/coordinates testsuite/MDAnalysisTests/utils testsuite/MDAnalysisTests/topology testsuite/MDAnalysisTests/auxiliary testsuite/MDAnalysisTests/core"
+    - PYTEST_LIST="testsuite/MDAnalysisTests/lib testsuite/MDAnalysisTests/formats testsuite/MDAnalysisTests/coordinates testsuite/MDAnalysisTests/utils testsuite/MDAnalysisTests/topology testsuite/MDAnalysisTests/auxiliary testsuite/MDAnalysisTests/core testsuite/MDAnalysisTests/analysis"
    - NOSE_COVERAGE_FILE="nose_coverage"
    - PYTEST_COVERAGE_FILE="pytest_coverage"
    - MAIN_CMD="pytest ${PYTEST_LIST} ${PYTEST_FLAGS}; python ./testsuite/MDAnalysisTests/mda_nosetests ${NOSE_TEST_LIST} ${NOSE_FLAGS}"


I agree with @orbeckst. With NOSE_TEST_LIST being empty, the call to mda_nosetests becomes buggy and should be removed. Removing the dependencies to nose is out of scope, though.

utkbansal · 2017-07-13T17:00:13Z

@jbarnoud If I understand correctly, I have to squash all my commits now and then have one more commit that removes mda_nosetests . In the end, one commit for porting analysis module and one for removing mda_nosetests

jbarnoud · 2017-07-13T17:10:14Z

All your commits until the one titled "Quick fix" are logical units. I would squash them but I do not mind having them separate. From "Quick fix" and latter, the commits are iterative fixes that should be squashed. The removal of mda_nosetest must be on a separate commit.

Hack in parameterize Cleanup Use itertools Cleanup

utkbansal · 2017-07-13T18:23:12Z

It's hard to believe, but the full build completed in 16.5 mins.

utkbansal · 2017-07-13T18:34:30Z

@jbarnoud All good?

utkbansal · 2017-07-13T19:16:06Z

The build passed! Plus a minor increase in coverage.

kain88-de · 2017-07-14T07:53:04Z

@orbeckst do you have any more comments? I would like to merge this.

utkbansal added 6 commits July 3, 2017 03:37

Fix test_align.py

f0a8204

Fix test_base.py

3b4841b

Fix test_contacts.py

60cb030

Fix test_density.py

c773c0f

Fix test_diffusionmap.py

ed4c80a

Fix test_gnm.py

c1e7cf2

utkbansal added 9 commits July 3, 2017 04:05

Partially fix test_hbonds.py

2deec4e

Fix test_hydrogenbondautocorrel.py

c9f2038

Fix test_nuclinfo.py

a7f050e

Fix test_pca.py

f3946f3

Fix test_persistencelength.py

80c4cba

Fix test_psa.py

0b5aced

Fix test_rdf.py

e025e63

Partially fix test_rms.py

1f0d2f9

Shift analysis to PYTEST_LIST

123139f

utkbansal changed the title ~~Port analysis module to pytest~~ [WIP]Port analysis module to pytest Jul 2, 2017

Delete useless file

17a9a47

orbeckst added testing Component-Analysis labels Jul 7, 2017

utkbansal added 2 commits July 8, 2017 14:58

Fix test_rms.py

ebc342e

Fix test_encore.py

fd129d6

jbarnoud reviewed Jul 8, 2017

View reviewed changes

orbeckst added the Work in progress label Jul 11, 2017

orbeckst requested changes Jul 11, 2017

View reviewed changes

jbarnoud requested changes Jul 13, 2017

View reviewed changes

mnmelo reviewed Jul 13, 2017

View reviewed changes

utkbansal changed the title ~~[WIP]Port analysis module to pytest~~ Port analysis module to pytest Jul 13, 2017

richardjgowers approved these changes Jul 13, 2017

View reviewed changes

jbarnoud requested changes Jul 13, 2017

View reviewed changes

utkbansal force-pushed the pytest-analysis branch from cc80041 to 3c9151d Compare July 13, 2017 17:27

utkbansal added 2 commits July 13, 2017 23:12

Fix test_hbonds.py

5f6de0e

Hack in parameterize Cleanup Use itertools Cleanup

Removes mda_nosetests

f51f49a

utkbansal force-pushed the pytest-analysis branch from 75e623d to f51f49a Compare July 13, 2017 18:02

jbarnoud approved these changes Jul 13, 2017

View reviewed changes

kain88-de removed the Work in progress label Jul 13, 2017

orbeckst approved these changes Jul 14, 2017

View reviewed changes

jbarnoud merged commit 277127c into MDAnalysis:develop Jul 14, 2017

jbarnoud mentioned this pull request Jul 15, 2017

test_progress_meter, in test_rms.py, errors when output isn't captured. #1412

Closed

		@@ -162,6 +162,28 @@ def setUp(self):
		self.values['acceptor_resnm'] = np.array([], dtype="<U3")


		def build_parameters():

Port analysis module to pytest #1450

Port analysis module to pytest #1450

Uh oh!

Conversation

utkbansal commented Jul 2, 2017

Changes made in this Pull Request:

PR Checklist

Uh oh!

utkbansal commented Jul 2, 2017

Uh oh!

richardjgowers commented Jul 3, 2017

Uh oh!

kain88-de commented Jul 3, 2017

Uh oh!

utkbansal commented Jul 3, 2017

Uh oh!

jbarnoud commented Jul 3, 2017

Uh oh!

utkbansal commented Jul 8, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbarnoud commented Jul 9, 2017

Uh oh!

jbarnoud commented Jul 11, 2017

Uh oh!

orbeckst left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kain88-de commented Jul 13, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

utkbansal commented Jul 13, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment