unable to avoid duplicate tests when combining addopts and positional arguments using setup.cfg

[foxx]

This is extended from #1187.

In my case there is some surprising behaviour. If I run the test suite using addopts from setup.cfg then the tests are duplicated and detected twice. However if I run it using addopts from pytest.ini, then the problem goes away, which is very strange.

After using --debug, I can see that the arguments are actually duplicated when using setup.cfg;

args=['-rw', '-vv', '--maxfail=1', '--cov=bottlecap.py', '--showlocals', '--doctest-modules',
 '--cov-report=term-missing', '--trace-config', '--debug', 'tests', '-rw', '-vv', '--maxfail=1', 
'--cov=bottlecap.py', '--showlocals', '--doctest-modules', '--cov-report=term-missing',
 '--trace-config', '--debug', 'tests']

There are also issues with using setup.cfg, for example passing in testpaths results in an error;

$ python setup.py test
running pytest
error: error in setup.cfg: command 'PyTest' has no such option 'testpaths'

There is a comment by a maintainer usage of setup.cfg should be deprecated. All of the problems mentioned above seem to go away if I use pytest.ini, and there doesn't appear to be any clear advantage of trying to use setup.cfg for this.

Another option is to make [pytest] section of setup.cfg work exactly the same as if it was in pytest.ini, but I'm not sure how feasible that is. Alternatively, dropping support for that section in setup.cfg might be another option.

Thoughts @RonnyPfannschmidt / @nicoddemus ?

[RonnyPfannschmidt]

we should either use a prefixed section name in setup.cfg or remove the functionality,
since it is completely broken (due to how setuptools interprets the config file differently than pytest)

the error comes from setuptools, not from py.test

@hpk42 thoughts?

[foxx]

Removing support from setup.cfg would be ideal, but naturally would cause backwards compatibility issues. Perhaps we could add a deprecation warning, so that anyone attempting to use [pytest] in setup.cfg would know that it's no longer supported, along with a little hint instructing them to move it into pytest.ini ?

[nicoddemus]

IMHO if we can't fix the current behavior and it is considered problematic, we should issue a pytest-warning in 2.9 saying that setup.cfg support is deprecated and drop it for good in 2.10. This would give plenty of time for people to just move the options over to pytest.ini or tox.ini.

[foxx]

Agreed, +1

[nicoddemus]

Using [pytest] session in setup.cfg has been deprecated and will be removed in #3086