Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate from nose to py.test #884

Closed
kain88-de opened this issue Jun 30, 2016 · 27 comments

Comments

Projects
None yet
8 participants
@kain88-de
Copy link
Member

commented Jun 30, 2016

See: https://nose.readthedocs.io/en/latest/

Alternatives are py.test and nose2

http://nose2.readthedocs.io/en/latest/

http://pytest.org/latest/

I've only have additional experience with py.test and actually like it very much.

This should be done in one go as test-fixtures are handled differently in py.test for example.

@richardjgowers

This comment has been minimized.

Copy link
Member

commented Jul 1, 2016

Yeah I used py.test a little too and it seems nice, I think @dotsdl is also a fan.

As a little test, I ran test_atomgroup with py.test and got 16 fails and 328 passes, so it looks like there will be a little effort required, but not a crazy amount.

@orbeckst

This comment has been minimized.

Copy link
Member

commented Jul 1, 2016

What is numpy going to do?

Changing the whole test suite over to something else is going to be a nightmare. And there are all the plugins that we have for our custom checks that we would need to rewrite. What do we gain by doing this? What is the opportunity cost?

Parenthetically, I admit that I found py.test confusing and never figured out how to do generators (so I use the nose syntax that is somewhat supported) but I am sure that someone who spent a bit more time with py.test will be able to convince me that this is a fine piece of software.

@kain88-de

This comment has been minimized.

Copy link
Member Author

commented Jul 1, 2016

This is nothing we have to do quick. We can also wait to see what others are doing. Currently everything works so no hurry.

@mnmelo

This comment has been minimized.

Copy link
Member

commented Jul 2, 2016

@dotsdl also commented (personal communication) that he had good experience with py.test (as it allows for linter plugins and the like).

Personally, I'm happy to change, but agree it'll be a hassle. As for plugins, maybe some of the functionality already exists and saves us some trouble.

@orbeckst

This comment has been minimized.

Copy link
Member

commented Jul 10, 2016

With all the hoozah! for py.test, I'd say we start thinking about transitioning to py.test but monitor what numpy is doing, given that we extensively rely on their array comparisons. Or does py.test already include equivalent tests?

@kain88-de

This comment has been minimized.

Copy link
Member Author

commented Jul 10, 2016

Nope I can ask on the numpy dev list

@dotsdl

This comment has been minimized.

Copy link
Member

commented Jul 10, 2016

+1 for py.test, and most things we currently do should "just work" without changing anything. To eliminate nosetests as a dependency we will mainly need to replace decorators with py.test equivalents.

@jbarnoud

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2016

Just for the record, I ran py.test with the xdist plug-in for parallelization and I got:

651 failed, 2781 passed, 5 skipped, 46 error in 127.21 seconds

I find the error messages very verbose, and I do not like that the traceback is not displayed. Yet, I guess it can be configures.

I do not like either how the tests coming from test generators are displayed in verbose mode. Instead of the very informative list of arguments passed to the test, py.test only displays the index of the test in the generator. Once again, it is hopefully configurable.

@kain88-de

This comment has been minimized.

Copy link
Member Author

commented Jul 11, 2016

test generators work different in py.test, you decorate the function with @pytest.mark.parametrize This is all stuff that would need to be ported. For example to test multiple inputs for a function even you write:

@pytest.mark.parametrize('input, expected', [(1, False) , (2, True), (3, False)])
def test_even(input, expected):
     assert even(input) == expected

That is the most basic approach. With fixtures you can do even more advanced things and even completely change the default behavior.

Tracebacks are configurable, they are shown in single-process mode and by default more verbose then nosetests.

@dotsdl dotsdl referenced this issue Jul 22, 2016

Merged

Issue 363 - New Topology System #815

6 of 6 tasks complete
@kain88-de

This comment has been minimized.

Copy link
Member Author

commented Aug 8, 2016

numpy is doing, given that we extensively rely on their array comparisons.

I just had a look into the array comparison code from numpy. They only use numpy internal functions for that. No need for us to pay any attention to them since it would affect our own choice of testing frame work. They do use nose in assert_raise and raises but these would be replaced with pytest equivalents.

I also have the feeling that @dotsdl is anxious to move away from nose.

@orbeckst

This comment has been minimized.

Copy link
Member

commented Aug 8, 2016

On 8 Aug, 2016, at 14:25, Max Linke notifications@github.com wrote:

numpy is doing, given that we extensively rely on their array comparisons.

I just had a look into the array comparison code from numpy. They only use numpy internal functions for that. No need for us to pay any attention to them since it would affect our own choice of testing frame work.

So we would write our own array_equal, array_almost_equal etc?

That would make the transition easier… just import from our own testing helper module instead of numpy.testing

They do use nose in assert_raise and raises but these would be replaced with pytest equivalents.

Seems easy enough (famous last words).

@dotsdl

This comment has been minimized.

Copy link
Member

commented Aug 8, 2016

So we would write our own array_equal, array_almost_equal etc?

@orbeckst, no, we could continue using the numpy functions as we already are. They work just fine with py.test as written as far as I can tell. The only things that change are the nose things we're using.

@orbeckst

This comment has been minimized.

Copy link
Member

commented Aug 9, 2016

So this looks feasible, doesn't it?

So I would suggest, in preparation for the eventual move to py.tests, do not use nose-test specific constructs in tests anymore. Which ones are the ones to avoid?

@kain88-de

This comment has been minimized.

Copy link
Member Author

commented Aug 9, 2016

yes it's feasible. One construct we should avoid are test-generators that work with yield. There are much better alternatives available in pytest.

A short list at the top post of test refactors that will make the transition easier would be helpful. @dotsdl do you where this could be done?

On the side of actually doing this transition. This is going to be a huge change. Does anyone have a plan how to best do this? At the top of my head there are 3 things we should check/do.

  • port our plugins if necessary
  • do small refactorings that makes tests nose/pytest compatible
  • make a list of tests that won't run under pytest right now.
@orbeckst

This comment has been minimized.

Copy link
Member

commented Aug 9, 2016

On 9 Aug, 2016, at 02:11, Max Linke notifications@github.com wrote:

yes it's feasible. One construct we should avoid are test-generators that work with yield. There are much better alternatives available in pytest.

Can you give an example for how to transform one of the tests that is currently written with yield? That would make it easier.

(Also, doesn't py.test happily work with generators? – http://doc.pytest.org/en/latest/nose.html#supported-nose-idioms)

@kain88-de

This comment has been minimized.

Copy link
Member Author

commented Aug 10, 2016

As you link generators are supported nose idioms and pytest has much better alternatives with pytest.mark.parametrize.

But your link already gives one item. All tests classes should inherit from unittest.TestCase for setup to work.

@richardjgowers

This comment has been minimized.

Copy link
Member

commented Aug 10, 2016

@kain88-de the generators don't work inside a TestCase class, see the last line here:

http://nose.readthedocs.io/en/latest/writing_tests.html#test-generators

But yeah, rewriting the test generators will be required for pytest

@richardjgowers richardjgowers changed the title Nosetest isn't maintained any longer Migrate from nose to py.test Jan 21, 2017

@richardjgowers

This comment has been minimized.

Copy link
Member

commented Jan 21, 2017

Ok I've changed this to a migrate to py.test issue, nobody seemed to want/know about nose2 and py.test seems to do everything we need.

@vedantrathore

This comment has been minimized.

Copy link
Contributor

commented Feb 4, 2017

I would like to work on this, I have some experience in writing tests with py.test and would love to contribute to mdanalysis repository. Can anyone guide me on where to start with this?

@richardjgowers

This comment has been minimized.

Copy link
Member

commented Feb 4, 2017

@utkbansal

This comment has been minimized.

Copy link
Member

commented Feb 5, 2017

@kain88-de

This comment has been minimized.

Copy link
Member Author

commented Feb 6, 2017

I think the best start here is to create a wiki-page where we document several things

  • the plugins we have and how they could be ported to py.test
  • how many tests pass now with pytests
  • what fails
  • what idoms with examples in our testsuite fail
  • where can we replace nose with pytest idoms

After this initial research is complete we can look into further work.

@vedantrathore

This comment has been minimized.

Copy link
Contributor

commented Feb 7, 2017

@kain88-de @richardjgowers I have created a wiki page. If there is anything you'd like to add or edit do tell me.

Link to wiki page : here
Link to google group discussion : here

@jbarnoud

This comment has been minimized.

Copy link
Contributor

commented May 11, 2017

@utkbansal Any update on the performances of the memory plugin?

@utkbansal

This comment has been minimized.

Copy link
Member

commented May 11, 2017

@jbarnoud Yes, I'm working on one alternative. Should I open a separate issue for the same and continue discussions on it?

@jbarnoud

This comment has been minimized.

Copy link
Contributor

commented May 11, 2017

@orbeckst

This comment has been minimized.

Copy link
Member

commented Jul 18, 2017

Awesome, congratulations @utkbansal to this milestone! Well done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.