Skip to content

Loading…

setup.py nosetests issue #556

Closed
elemoine opened this Issue · 18 comments

6 participants

@elemoine

I'm not sure what's the proper place to ask questions about nose, so I'll ask my question here.

I very often use python setup.py nosetests to run tests for my Python apps. And I use python setup.py nosetests --tests ... to run specific tests. That worked well with nose 1.1.2, but it no longer works with 1.2.

With 1.2 all the tests are executed first, and then the ones specified using --tests. When I use --tests I'd like that only the specified tests get executed.

Is this a known issue? Are there any workarounds?

I tried to do python setup.py nosetests my/specific/test.py, but that does not work either. I get invalid command name 'my/specific/test.py', from distutils/setuptools I think.

@jpellerin
nose-devs member
@elemoine elemoine referenced this issue in camptocamp/c2cgeoportal
Merged

functional tests fail when using nose 1.2 #333

@elemoine

FWIW I think the issue is somewhere around https://github.com/nose-devs/nose/blob/master/nose/commands.py#L146. See also e4cd723.

@dcc635

This issue is still not fixed. When using python setup.py nosetests --tests test/file.py:TestClass.test, all tests are still run. Using --where only allows for specifying the directory. This is not sufficient for zooming in on a single test. (FYI I'm using Python 2.6, nosetests version 1.2.1)

A workaround is to just use the nosetests command.

@jszakmeister

It wouldn't be fixed in 1.2.1. The fix is on master, and hasn't been released yet.

@ajdavis ajdavis added a commit to mongodb/motor that referenced this issue
@ajdavis ajdavis Undo my hack to work around nose-devs/nose#556. a559af0
@MRigal

Why is this ticket actually closed ? It is still not working with nose 1.3.0 or 1.3.4...

@jszakmeister

@MRigal See #604--which is linked to this issue.

@MRigal

The PR was nice and I am using this -where option, but it is still not the same. You can have potentially hundreds of tests in a directory and it would be much better to be able to run only one file and even better to be able to spot one class and one test, normal behaviour when nosetests is run standalone

@jszakmeister

@MRigal It should not have prevented that. If it did, please open a new issue and provide a small test case.

@MRigal

@jszakmeister sorry to insist but I don't understand. This ticket describes perfectly what is not working and just has not been fixed. @dcc635 comment in 2013 was perfectly right and is still actual !

The PR #604 brought an addition and helps, but does not fix the problem described in this ticket. I don't know why I should open a new one as this one has NOT been fixed...

Towards a test case, I wouldn't know how to write one for this. There was also none in the PR #604. But you could reproduce it easily. For example, I am running tests for mongoengine (https://github.com/MongoEngine/mongoengine), and I would like to run only one specific test, like following:

python setup.py nosetests --tests tests.queryset.queryset:QuerySetTest.test_distinct_ListField_EmbeddedDocumentField

And instead of running only this test, it runs all tests PLUS this test (a second time).

And running the following should IMHO also work:

python setup.py nosetests tests.queryset.queryset:QuerySetTest.test_distinct_ListField_EmbeddedDocumentField

But instead returns 'invalid command name'

@jszakmeister

@jszakmeister sorry to insist but I don't understand. This ticket describes perfectly what is not working and just has not been fixed. @dcc635 comment in 2013 was perfectly right and is still actual !

The PR #604 brought an addition and helps, but does not fix the problem described in this ticket. I don't know why I should open a new one as this one has NOT been fixed...

sigh This was wholly unhelpful.

Your question was "why was this ticket closed", and my answer is because "it was fixed". If it's still a problem then great. But the ticket was closed before @dcc635's comment was made because it was believed to be fixed. In the mean time, you provided only a few details and nothing to indicate that the problem you were seeing is at all related to this one.

Towards a test case, I wouldn't know how to write one for this. There was also none in the PR #604. But you could reproduce it easily. For example, I am running tests for mongoengine (https://github.com/MongoEngine/mongoengine), and I would like to run only one specific test, like following:

I was asking for a way to reproduce the problem, not to write an actual test case.

python setup.py nosetests --tests tests.queryset.queryset:QuerySetTest.test_distinct_ListField_EmbeddedDocumentField

And instead of running only this test, it runs all tests PLUS this test (a second time).

I don't see this problem:

{jszakmeister@sidious.home} [~/tmp/nose-test] [nose-2.7]
:: python setup.py nosetests --tests simple/tests/test_one.py
running nosetests
running egg_info
writing dependency_links to simple.egg-info/dependency_links.txt
writing simple.egg-info/PKG-INFO
writing top-level names to simple.egg-info/top_level.txt
reading manifest file 'simple.egg-info/SOURCES.txt'
writing manifest file 'simple.egg-info/SOURCES.txt'
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
{jszakmeister@sidious.home} [~/tmp/nose-test] [nose-2.7]
:: python setup.py nosetests --tests simple/tests/test_one.py:test_one
running nosetests
running egg_info
writing dependency_links to simple.egg-info/dependency_links.txt
writing simple.egg-info/PKG-INFO
writing top-level names to simple.egg-info/top_level.txt
reading manifest file 'simple.egg-info/SOURCES.txt'
writing manifest file 'simple.egg-info/SOURCES.txt'
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
{jszakmeister@sidious.home} [~/tmp/nose-test] [nose-2.7]
:: python setup.py nosetests
running nosetests
running egg_info
writing dependency_links to simple.egg-info/dependency_links.txt
writing simple.egg-info/PKG-INFO
writing top-level names to simple.egg-info/top_level.txt
reading manifest file 'simple.egg-info/SOURCES.txt'
writing manifest file 'simple.egg-info/SOURCES.txt'
...
----------------------------------------------------------------------
Ran 3 tests in 0.004s

OK

As you can see from the above snippet, I have three test in total, two of which are in a file called test_one.py. I was able to run both test in there, without running the others. And I was able to select an individual test and run it too.

So, the way I see it, the bug still does not exist. Care to come up with a counter example?

And running the following should IMHO also work:

python setup.py nosetests tests.queryset.queryset:QuerySetTest.test_distinct_ListField_EmbeddedDocumentField

That will never work. It's a function of distutils behavior and the fact that you can string multiple commands in a single command line. To be honest, this is part of the reason why the documentation discourages integration with distutils--it'll never be as flexible as running nosetests directly.

@ajdavis

I've been lurking, and I want to point out that the syntax that @jszakmeister has demonstrated working is one that refers to test files like "foo/bar.py", and the syntax that @MRigal demonstrates not working uses test modules like "foo.bar".

@MRigal's syntax is the one I would've expected, since it matches the "nosetests" binary's argument syntax.

@jszakmeister

I've been lurking, and I want to point out that the syntax that @jszakmeister has demonstrated working is one that refers to test files like "foo/bar.py", and the syntax that @MRigal demonstrates not working uses test modules like "foo.bar".

@MRigal's syntax is the one I would've expected, since it matches the "nosetests" binary's argument syntax.

Not entirely true, they're both acceptable:

:: python setup.py nosetests --tests simple.tests.test_one:test_one
running nosetests
running egg_info
writing dependency_links to simple.egg-info/dependency_links.txt
writing simple.egg-info/PKG-INFO
writing top-level names to simple.egg-info/top_level.txt
reading manifest file 'simple.egg-info/SOURCES.txt'
writing manifest file 'simple.egg-info/SOURCES.txt'
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
@MRigal

Hi @jszakmeister nice to see it works for you. Indeed the problem may not be in nose but in how it was intregrated to mongoengine.

Could you eventually share this 'simple' app you have ? (gist of whatever, just to try something else on my side)

In my case, and I tried to describe it in a reproducable manner, if you checkout the mongoengine source code (https://github.com/MongoEngine/mongoengine) and try to run only selected tests, if I use --where it does work as expected, but if I use --tests, it runs all tests PLUS the one(s) requested a second time.

You may just try this to see if it has something to do with my configuration. In that case, sorry for the trouble, the ticket should stay closed and I'll look to get around differently. If you encounter the same behaviour, maybe there is a bug in the integration of nosetests into mongoengine and potentially a bug worth another ticket somewhere...

I hope this comment helps to go forward

@jszakmeister

Hi @jszakmeister nice to see it works for you. Indeed the problem may not be in nose but in how it was intregrated to mongoengine.

Could you eventually share this 'simple' app you have ? (gist of whatever, just to try something else on my side)

I won't promise to keep it up there forever, but here it is: http://www.szakmeister.net/downloads/nose-test.tar.gz

In my case, and I tried to describe it in a reproducable manner, if you checkout the mongoengine source code (https://github.com/MongoEngine/mongoengine) and try to run only selected tests, if I use --where it does work as expected, but if I use --tests, it runs all tests PLUS the one(s) requested a second time.

I'm sorry, but I don't checkout other projects and attempt things any more. I've spent hours in the past trying to get environments figured out and making things work, and it's simply too much time--I cannot afford to spend the little time I have that way. That's why I asked for a smaller, reproducible test case.

It looks like from the above that I'd probably need Mongo installed--and I don't want it, or need it--along with a bunch of other stuff. I did try creating a virtualenv and setting it up, but it failed to build a dependency. Unfortunately, I cannot invest any more time than that. Sorry.

You may just try this to see if it has something to do with my configuration. In that case, sorry for the trouble, the ticket should stay closed and I'll look to get around differently. If you encounter the same behaviour, maybe there is a bug in the integration of nosetests into mongoengine and potentially a bug worth another ticket somewhere...

I suspect the latter is true. There's probably some configuration that saying "run all tests from here" and your --tests argument is just tacking onto the end of it. If you find it, and it looks like buggy behavior, the yes, open a ticket. It may be user error though: it's doing exactly what you told it too. :-)

I hope this comment helps to go forward

No worries. I've been doing open source for nearly 15 years now. My skin is quite thick. ;-)

@MRigal

Hi @jszakmeister.
Thanks for your testcase, it helped me to find out what was wrong. Your one is indeed doing the expected thing.
Now I'll expose my problem and you shall tell me if it is worth opening a new ticket on the nose side or fixing the packaging on the mongoengine side.

Take your test package and add at the root level a file 'setup.cfg'
Simply add this to the file:

[nosetests]
where = tests

And observe the same behaviour that I was having when specifying --tests

Thanks again and don't worry about not installing mongo, I was not asking to be feed like a children, just for a bit more to help also.

@jszakmeister
[nosetests]
where = tests

I don't think you've actually tried that, otherwise you would have noticed that it results in a error:

======================================================================
ERROR: Failure: ImportError (No module named tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jszakmeister/projects/nose/nose/loader.py", line 409, in loadTestsFromName
    module = resolve_name(addr.module)
  File "/Users/jszakmeister/projects/nose/nose/util.py", line 311, in resolve_name
    module = __import__('.'.join(parts_copy))
ImportError: No module named tests

----------------------------------------------------------------------

You should also note this:

/Users/jszakmeister/projects/nose/nose/config.py:435: DeprecationWarning: Use of multiple -w arguments is deprecated and support may be removed in a future release. You can get the same behavior by passing directories without the -w argument on the command line, or by using the --tests argument in a configuration file.

So it may be that your --where argument is colliding with a nosetests distutils command.

I'm going to assume that what you really meant was to put where = simple/tests. In this case I do see the problem. A workaround--and not even that really, it's just a good idea--is not to use the --where command, but tests instead:

[nosetests]
tests = simple/tests

The things work as expected:

:: python setup.py nosetests
running nosetests
running egg_info
writing simple.egg-info/PKG-INFO
writing top-level names to simple.egg-info/top_level.txt
writing dependency_links to simple.egg-info/dependency_links.txt
reading manifest file 'simple.egg-info/SOURCES.txt'
writing manifest file 'simple.egg-info/SOURCES.txt'
...
----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK

:: python setup.py nosetests --tests simple/tests/test_one.py:test_one
running nosetests
running egg_info
writing simple.egg-info/PKG-INFO
writing top-level names to simple.egg-info/top_level.txt
writing dependency_links to simple.egg-info/dependency_links.txt
reading manifest file 'simple.egg-info/SOURCES.txt'
writing manifest file 'simple.egg-info/SOURCES.txt'
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
@MRigal

Oh yes I did test that, and it just depend to from which level you run the tests.

Now I can change the nosetests configuration parameter from where to test and do the opposite, call it afterwards with --where.

# python setup.py nosetests --where simple/tests/test_one.py:test_one
running nosetests
running egg_info
writing simple.egg-info/PKG-INFO
writing top-level names to simple.egg-info/top_level.txt
writing dependency_links to simple.egg-info/dependency_links.txt
reading manifest file 'simple.egg-info/SOURCES.txt'
writing manifest file 'simple.egg-info/SOURCES.txt'
/Users/mrigal/dev/_virtualenvs/myorderbird/lib/python2.7/site-packages/nose/config.py:435: DeprecationWarning: Use of multiple -w arguments is deprecated and support may be removed in a future release. You can get the same behavior by passing directories without the -w argument on the command line, or by using the --tests argument in a configuration file.
  DeprecationWarning)
....
----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK

And here we have the same problem in the other direction.

In one direction or another, we have the problem that the --where or the --tests passed in the command line is interfering with the (opposite) tests XOR where given in setup.cfg

Is this expected behaviour or should I create a new ticket for that ?

@jszakmeister

Oh yes I did test that, and it just depend to from which level you run the tests.

I'm confused. Do you mean you did not put your setup.cfg alongside the setup.py? Did you not run it through python setup.py nosetests? Where you in a different directory? How are you executing it differently than what I did that having where = tests was correct?

# python setup.py nosetests --where simple/tests/test_one.py:test_one

I don't think --where was ever really meant to be used that way. --tests is how you want to spell it. :-)

In one direction or another, we have the problem that the --where or the --tests passed in the command line is interfering with the (opposite) tests XOR where given in setup.cfg

Is this expected behaviour or should I create a new ticket for that ?

Honestly, I'd call it expected behavior, but if it could be improved upon (without busting folks setup badly), then I'd take a patch to improve it too. But I'm still not convinced we're on the same page just yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.