Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spurious test failures #556

Closed
orbeckst opened this issue Dec 2, 2015 · 10 comments

Comments

Projects
None yet
4 participants
@orbeckst
Copy link
Member

commented Dec 2, 2015

Tests started failing somewhat randomly as mentioned in the post mdnalysis-devel: spurious test failures

Without reliable unit tests, our development workflow is severely hampered. This needs to be fixed but I have no idea how.

Details

See https://travis-ci.org/MDAnalysis/mdanalysis/builds/94270229 for an example. All failures are in the coordinate tests and are of the form

  File "/home/travis/miniconda/envs/pyenv/lib/python2.7/site-packages/MDAnalysisTests/coordinates/base.py", line 198, in test_get_writer_container 
    with tempdir.in_tempdir(): 
  File "/home/travis/miniconda/envs/pyenv/lib/python2.7/contextlib.py", line 17, in __enter__ 
    return self.gen.next() 
  File "/home/travis/miniconda/envs/pyenv/lib/python2.7/site-packages/tempdir.py", line 44, in in_tempdir 
    old_path = os.getcwd() 

OSError: [Errno 2] No such file or directory 

and the common theme appears to be tempdir.in_tempdir().

Initial discussion

Initial reports/discussion about these test failures started at #551 (comment) and should continue here.

On 1 Dec, 2015, at 17:10, @hainm wrote:

it's better to run the test in single core to debug.

Maybe we should try --processes=1 just to avoid any opportunity for race conditions. It is suspicious that tempdir.in_tempdir() appears to be always involved.

How's about

cd testsuite/MDAnalysisTests/coordinates
python test_netcdf.py

python test_netcdf.py does not do anything.

./mda_nosetests coordinates/test_netcdf.py works fine in serial or parallel. The whole 3300+ tests pass locally, with up to 24 threads... most of the time. When it fails for a few tests then these failures are not necessarily the same. See #551 (comment) .

@orbeckst

This comment has been minimized.

Copy link
Member Author

commented Dec 2, 2015

@hainm reported that a potentially similar error was reported in frescobaldi/frescobaldi#554 and frescobaldi/frescobaldi#776 although no solution was presented apart from the hint of "problems when creating a tempdir inside a tempdir".

@richardjgowers

This comment has been minimized.

Copy link
Member

commented Dec 2, 2015

If we've accidentally nested tempdirs (calling Tempdir while inside a Tempdir decorated class?) that might be a way for that to happeb

@kain88-de

This comment has been minimized.

Copy link
Member

commented Dec 2, 2015

But the errors seem to be random and I can't find such a nested pattern in the code where it fails.

@hainm

This comment has been minimized.

Copy link
Contributor

commented Dec 2, 2015

@orbeckst should buy me a beer.

nosetests -vs testsuite/MDAnalysisTests/analysis/test_helanal.py testsuite/MDAnalysisTests/analysis/test_psa.py

will cause error because someone went to temp folder in test_helanal.py but has never come back to working dir.

MDAnalysis.analysis.helanal: Check for resolution of Issue 188. ... ok test_hausdorff_bound (MDAnalysisTests.analysis.test_psa.TestPSAnalysis) ... ERROR test_reversal_frechet (MDAnalysisTests.analysis.test_psa.TestPSAnalysis) ... ERROR test_reversal_hausdorff (MDAnalysisTests.analysis.test_psa.TestPSAnalysis) ... ERROR

@hainm

This comment has been minimized.

Copy link
Contributor

commented Dec 2, 2015

it's better to always use contextmanager.

@orbeckst

This comment has been minimized.

Copy link
Member Author

commented Dec 3, 2015

On 2 Dec, 2015, at 16:53, Hai Nguyen wrote:

@orbeckst should give beer to me.

nosetests -vs testsuite/MDAnalysisTests/analysis/test_helanal.py testsuite/MDAnalysisTests/analysis/
test_psa.py

will cause error because someone go to temp folder in test_helanal.py but has never come back to working dir.

I just heard from @dldotson and @richardjgowers that this fixes the problem: I am very happy, so yes: I will gladly buy you a 🍺 - find me at ACS (or find @dldotson or @slseyler at BPS who can buy you a beer in my name ;-) ).

orbeckst referenced this issue in hainm/mdanalysis Dec 3, 2015

@orbeckst

This comment has been minimized.

Copy link
Member Author

commented Dec 3, 2015

I added a note to ban os.chdir() when writing testcases.

(And yes, the wiki page on testing needs a major clean-up.)

@orbeckst

This comment has been minimized.

Copy link
Member Author

commented Dec 3, 2015

$ git grep chdir
testsuite/MDAnalysisTests/analysis/test_helanal.py:        os.chdir(self.tempdir)
testsuite/MDAnalysisTests/analysis/test_hole.py:        os.chdir(self.dir_name)
testsuite/ez_setup.py:        os.chdir(tmpdir)
testsuite/ez_setup.py:        os.chdir(subdir)
testsuite/ez_setup.py:        os.chdir(old_wd)
testsuite/ez_setup.py:        os.chdir(tmpdir)
testsuite/ez_setup.py:        os.chdir(subdir)
testsuite/ez_setup.py:        os.chdir(old_wd)

#558 should take care of all of this (and of course we should eliminate ez_setup.py etc from the testsuite....)

@orbeckst

This comment has been minimized.

Copy link
Member Author

commented Dec 3, 2015

@hainm do you also want to fix test_hole.py with #558? Otherwise I can clean up my own mess in test_hole.py (which, by the way, is only run when you have hole available so it won't trigger the bug on travis).

@hainm

This comment has been minimized.

Copy link
Contributor

commented Dec 3, 2015

@orbeckst I am leaving test_hole.py to you since I don't have HOLE installed (so not sure if the change is good or not).

@orbeckst orbeckst closed this in d424ccf Dec 3, 2015

orbeckst added a commit that referenced this issue Dec 3, 2015

finalized fix of #556 (test failure due os.chdir/tempdir)
- use tempdir.in_tempdir() in test_hole.py
- added Hai Nguyen to AUTHORS and sphinx/source/conf.py
  (for #558)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.