tests run twice with setup.py test in python3.5

[wtayyeb]

Hi.
The setup.py test in python3.5 result to twice test run.

the very simple project to reproduce the problem is located at https://github.com/wtayyeb/setuptools-twice-test which have three tests

by running it via tox you could see that for py27 no test found (legacy behaviour, need discover to find them) and for py34 the behaviour is fixed and three tests were run. but in py35 it run 6 tests, as they are doublicated.

if we use setup.py test -s dummy all three environment will report 3 test run. but it is not the general way. also when run python -m unittest discover it also result correctly 3 test for each python version.

In case of using legacy woraround that create test_command that add discover to unittest args, the problem still exists for py35.

the result of tox is like below

**py27 installed:**
py27 runtests: PYTHONHASHSEED='545'
py27 runtests: commands[0] | python setup.py test
running test
running egg_info
writing top-level names to dummy.egg-info\top_level.txt
writing dependency_links to dummy.egg-info\dependency_links.txt
writing dummy.egg-info\PKG-INFO
reading manifest file 'dummy.egg-info\SOURCES.txt'
writing manifest file 'dummy.egg-info\SOURCES.txt'
running build_ext

----------------------------------------------------------------------
**Ran 0 tests in 0.000s**

OK
**py34 installed:**
py34 runtests: PYTHONHASHSEED='545'
py34 runtests: commands[0] | python setup.py test
running test
running egg_info
writing dummy.egg-info\PKG-INFO
writing top-level names to dummy.egg-info\top_level.txt
writing dependency_links to dummy.egg-info\dependency_links.txt
reading manifest file 'dummy.egg-info\SOURCES.txt'
writing manifest file 'dummy.egg-info\SOURCES.txt'
running build_ext
test_one (dummy.test_dummy.Test) ... ok
test_three (dummy.test_dummy.Test) ... ok
test_two (dummy.test_dummy.Test) ... ok

----------------------------------------------------------------------
**Ran 3 tests in 0.000s**

OK
**py35 installed:**
py35 runtests: PYTHONHASHSEED='545'
py35 runtests: commands[0] | python setup.py test
running test
running egg_info
writing dummy.egg-info\PKG-INFO
writing top-level names to dummy.egg-info\top_level.txt
writing dependency_links to dummy.egg-info\dependency_links.txt
reading manifest file 'dummy.egg-info\SOURCES.txt'
writing manifest file 'dummy.egg-info\SOURCES.txt'
running build_ext
test_one (dummy.test_dummy.Test) ... ok
test_three (dummy.test_dummy.Test) ... ok
test_two (dummy.test_dummy.Test) ... ok
test_one (dummy.test_dummy.Test) ... ok
test_three (dummy.test_dummy.Test) ... ok
test_two (dummy.test_dummy.Test) ... ok

----------------------------------------------------------------------
**Ran 6 tests in 0.000s**

OK
___________________________________ summary __________________________
  py27: commands succeeded
  py34: commands succeeded
  py35: commands succeeded
  congratulations :)

[wtayyeb]

i currently use latest version of setuptools 25.1.0
but the problem appear as of setuptools==18.4 and up

[jaraco]

I'm not sure why this happens.

the problem appear as of setuptools==18.4 and up

Are you saying that the problem does not occur with setuptools 18.3.x and earlier?

Would you be willing to investigate deeper and track down the cause?

[wtayyeb]

I do something like git bisect on it and found the exact version!
to find this 18.4 i do as follow
first I install setuptools latest(25.1.1) the problem exists.
then downgrage the setuptools to v15, problem gone ( setuptools not mentioned that work with py3.5 at that version, but it works)
then go up and down and lastly I found 18.3.3 have no problem, but 18.4 have it.

It is first version that implement a new way to find tests, before it (18.3.3 and down) the test model must explicitly defined as argument (with -s key).

[jaraco]

That's excellent work! Now we can go to the changelog for 18.4 and see that the change originated with #446.

So that at least explains why things changed with 18.4.

[wtayyeb]

I know, I read the source code and try to fix it locally and if can do anything I would submit a pull request to fix it. but I fail! I read the changelog after I have found that the problem is in 18.4! :)

the @bb-migration mark #446 as major bug, so I dont alone with this problem!

setuptools will install the same on ether py3.4 or py3.5.
I will check python3.5 unittest againest python3.4 ones and try to find anything there that cause the problem! if there any, I comment it here!

[sigmavirus24]

@wtayyeb any updates?

[wtayyeb]

@sigmavirus24 nope!

[jaraco]

You might also test again with v30.2.1 based on the changes introduced in #850.

[mamoit]

With setuptools-36.0.1 and Python 3.6.1 I still get this issue.

[benoit-pierre]

Looks like with Python 3 (or at least 3.4, 3.5, 3.6, I don't have 3.3 to test), TestLoader.loadTestsFromModule already automatically recurse into sub-modules, so tests are duplicated. I guess one way to fix it would be to keep track of visited modules.

To fix the fact that tests are not run with Python 2, adding discover to unittest arguments when no testsuite is specified fix the issue, except for Python 2.6.