New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests run twice with setup.py test in python3.5 #701

Closed
wtayyeb opened this Issue Jul 28, 2016 · 10 comments

Comments

Projects
None yet
5 participants
@wtayyeb

wtayyeb commented Jul 28, 2016

Hi.
The setup.py test in python3.5 result to twice test run.

the very simple project to reproduce the problem is located at https://github.com/wtayyeb/setuptools-twice-test which have three tests

by running it via tox you could see that for py27 no test found (legacy behaviour, need discover to find them) and for py34 the behaviour is fixed and three tests were run. but in py35 it run 6 tests, as they are doublicated.

if we use setup.py test -s dummy all three environment will report 3 test run. but it is not the general way. also when run python -m unittest discover it also result correctly 3 test for each python version.

In case of using legacy woraround that create test_command that add discover to unittest args, the problem still exists for py35.

the result of tox is like below

**py27 installed:**
py27 runtests: PYTHONHASHSEED='545'
py27 runtests: commands[0] | python setup.py test
running test
running egg_info
writing top-level names to dummy.egg-info\top_level.txt
writing dependency_links to dummy.egg-info\dependency_links.txt
writing dummy.egg-info\PKG-INFO
reading manifest file 'dummy.egg-info\SOURCES.txt'
writing manifest file 'dummy.egg-info\SOURCES.txt'
running build_ext

----------------------------------------------------------------------
**Ran 0 tests in 0.000s**

OK
**py34 installed:**
py34 runtests: PYTHONHASHSEED='545'
py34 runtests: commands[0] | python setup.py test
running test
running egg_info
writing dummy.egg-info\PKG-INFO
writing top-level names to dummy.egg-info\top_level.txt
writing dependency_links to dummy.egg-info\dependency_links.txt
reading manifest file 'dummy.egg-info\SOURCES.txt'
writing manifest file 'dummy.egg-info\SOURCES.txt'
running build_ext
test_one (dummy.test_dummy.Test) ... ok
test_three (dummy.test_dummy.Test) ... ok
test_two (dummy.test_dummy.Test) ... ok

----------------------------------------------------------------------
**Ran 3 tests in 0.000s**

OK
**py35 installed:**
py35 runtests: PYTHONHASHSEED='545'
py35 runtests: commands[0] | python setup.py test
running test
running egg_info
writing dummy.egg-info\PKG-INFO
writing top-level names to dummy.egg-info\top_level.txt
writing dependency_links to dummy.egg-info\dependency_links.txt
reading manifest file 'dummy.egg-info\SOURCES.txt'
writing manifest file 'dummy.egg-info\SOURCES.txt'
running build_ext
test_one (dummy.test_dummy.Test) ... ok
test_three (dummy.test_dummy.Test) ... ok
test_two (dummy.test_dummy.Test) ... ok
test_one (dummy.test_dummy.Test) ... ok
test_three (dummy.test_dummy.Test) ... ok
test_two (dummy.test_dummy.Test) ... ok

----------------------------------------------------------------------
**Ran 6 tests in 0.000s**

OK
___________________________________ summary __________________________
  py27: commands succeeded
  py34: commands succeeded
  py35: commands succeeded
  congratulations :)
@wtayyeb

This comment has been minimized.

wtayyeb commented Jul 28, 2016

i currently use latest version of setuptools 25.1.0
but the problem appear as of setuptools==18.4 and up

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 30, 2016

I'm not sure why this happens.

the problem appear as of setuptools==18.4 and up

Are you saying that the problem does not occur with setuptools 18.3.x and earlier?

Would you be willing to investigate deeper and track down the cause?

@wtayyeb

This comment has been minimized.

wtayyeb commented Jul 30, 2016

I do something like git bisect on it and found the exact version!
to find this 18.4 i do as follow
first I install setuptools latest(25.1.1) the problem exists.
then downgrage the setuptools to v15, problem gone ( setuptools not mentioned that work with py3.5 at that version, but it works)
then go up and down and lastly I found 18.3.3 have no problem, but 18.4 have it.

It is first version that implement a new way to find tests, before it (18.3.3 and down) the test model must explicitly defined as argument (with -s key).

@jaraco

This comment has been minimized.

Member

jaraco commented Jul 30, 2016

That's excellent work! Now we can go to the changelog for 18.4 and see that the change originated with #446.

So that at least explains why things changed with 18.4.

@wtayyeb

This comment has been minimized.

wtayyeb commented Jul 30, 2016

I know, I read the source code and try to fix it locally and if can do anything I would submit a pull request to fix it. but I fail! I read the changelog after I have found that the problem is in 18.4! :)

the @bb-migration mark #446 as major bug, so I dont alone with this problem!

setuptools will install the same on ether py3.4 or py3.5.
I will check python3.5 unittest againest python3.4 ones and try to find anything there that cause the problem! if there any, I comment it here!

@sigmavirus24

This comment has been minimized.

Member

sigmavirus24 commented Dec 6, 2016

@wtayyeb any updates?

@wtayyeb

This comment has been minimized.

wtayyeb commented Dec 8, 2016

@jaraco

This comment has been minimized.

Member

jaraco commented Dec 8, 2016

You might also test again with v30.2.1 based on the changes introduced in #850.

@mamoit

This comment has been minimized.

mamoit commented Jun 16, 2017

With setuptools-36.0.1 and Python 3.6.1 I still get this issue.

@benoit-pierre

This comment has been minimized.

Member

benoit-pierre commented Jun 26, 2017

Looks like with Python 3 (or at least 3.4, 3.5, 3.6, I don't have 3.3 to test), TestLoader.loadTestsFromModule already automatically recurse into sub-modules, so tests are duplicated. I guess one way to fix it would be to keep track of visited modules.

To fix the fact that tests are not run with Python 2, adding discover to unittest arguments when no testsuite is specified fix the issue, except for Python 2.6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment