Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove test command and tests_require #931

Open
jaraco opened this issue Jan 22, 2017 · 36 comments
Open

Remove test command and tests_require #931

jaraco opened this issue Jan 22, 2017 · 36 comments
Labels
Needs Discussion Issues where the implementation still needs to be discussed.

Comments

@jaraco
Copy link
Member

jaraco commented Jan 22, 2017

In this comment, @graingert proposes that we may be able to completely remove support for tests_require instead of transitioning that tooling from easy_install to pip install.

While he didn't directly propose removal, the effort would only benefit that ticket if the install functionality were completely removed, so let's explore what that will entail.

While I agree that tox is an excellent, powerful, robust solution, it's more heavy than tests_require, requiring that the user have tox installed in advance. As a more thorough solution, it also is subject to bugs and constraints that a simpler test runner is not. There are several advantages to setup.py test and tests_require over tox:

  • Doesn't have any prerequisites (except the implicit one on Setuptools, which is generally assumed).
  • As a result of not having any prerequisites, invocation is often a simple invocation (setup.py test) instead of multiple steps (e.g. pip install --user tox; python -m tox).
  • setup.py test allows for invocation under a number of different Python versions naturally (i.e. python3.3 setup.py test or py -3.3 setup.py test) whereas tox offers "run under the python in which tox is installed" or "run for explicitly-declared python versions".
  • As setup.py test doesn't use virtualenv, it's not subject to virtualenv bugs or other constraints imposed by virtualenv (such as version 14 dropping support for Python 3.2.).
  • As setup.py test doesn't rely on pip, it's not subject to the bugs of pip (such as issues with --editable installs or namespace packages) or other constraints imposed by pip (such as dropping support for Python 3.2).

I consider these advantages small and easy enough to overcome, especially now that many of these issues have been resolved in setuptools, pip, and virtualenv. If we can get to a place that tox can broadly supplant the uses cases of setup.py test and pytest-runner (and thus tests_require) in practice, then yes, deprecating and removing it would be in order. Given the amount of activity and bugs I see around these tools, I'd asses they're still in active use.

Before flagging this functionality as deprecated, I'd like to survey the community about the possibility to see if there are use cases that would prove difficult to support with tox.

@graingert, would you be interested in being the champion for this effort (removing tests_require), starting with the outreach on distutils-sig and then implementing the deprecation/removal changes?

@FRidh
Copy link

FRidh commented Feb 11, 2017

Perhaps its relevant here. After discussing with @RonnyPfannschmidt we added the following to the Tox docs.

Integrating tox with setup.py test is as of October 2016 discouraged as it breaks packaging/testing approaches as used by downstream distributions which expect setup.py test to run tests with the invocation interpreter rather than setting up many virtualenvs and installing packages. If you need to define setup.py test you can better see about integrating your eventual test runner with it, here is an example of setup.py test integration with pytest. As the python eco-system rather moves away from using setup.py as a tool entry point it’s maybe best to not go for any setup.py test integration.

As a downstream we prefer to have one command that tests the package for the interpreter that is available and not any other versions.

@jaraco
Copy link
Member Author

jaraco commented Feb 12, 2017

I've discovered that tox -e py or tox -e python will test against the python in which tox is installed, and in fact that's the default behavior if no envlist is provided. I use this usage by default across dozens of projects and find it quite useful.

@FRidh
Copy link

FRidh commented Feb 12, 2017

Yes, I agree, as long as no other envs are defined, then there's no problem.

@techtonik
Copy link
Contributor

tox is good for CI, but too unwieldy for development. There should be an easy and intuitive way to run single test. Then this way can be combined with filesystem watcher of your choice.

@graingert
Copy link
Contributor

graingert commented May 16, 2017

@techtonik tox works totally fine for dev, you can use {posargs} to pass test flags to pytest: tox -e py36 -- --cov -s etc

@techtonik
Copy link
Contributor

techtonik commented May 16, 2017

@graingert how much time does it take on you system to execute a single test with tox. On my it takes more than 5 seconds, which is not acceptable. Also, tox doesn't plan to add output redirection, which makes debugging hard.

tox-dev/tox#405
tox-dev/tox#73

@RonnyPfannschmidt
Copy link
Contributor

@jaraco instead of test_requires, i'd like to see something that allows pip install -e .[test]

@pganssle pganssle added the Needs Triage Issues that need to be evaluated for severity and status. label Oct 19, 2018
@pganssle pganssle added Needs Discussion Issues where the implementation still needs to be discussed. and removed Needs Triage Issues that need to be evaluated for severity and status. labels Feb 12, 2019
@mcepl
Copy link
Contributor

mcepl commented Mar 7, 2019

I don't like the idea of replacing simple test calling with tox with my openSUSE Python maintenance hat on. We strive hard to achieve reproducible builds in openSUSE and thus we strongly prefer packages with small dependency trees. osc dependson openSUSE:Factory python-tox standard x86_64|grep python-|wc -l gives me 33 dependent packages (there are some false positives) and that seems too many for tool which should end up as a dependency of setuptools (or be on the same level in the dependency tree). We would probably have to eliminate all python setup.py test calls with something more primitive (e.g., python -munittest discover if you don't plan to eliminate that as well), but that seems counterproductive.

@pganssle
Copy link
Member

pganssle commented Jul 2, 2019

I don't like the idea of replacing simple test calling with tox with my openSUSE Python maintenance hat on. We strive hard to achieve reproducible builds in openSUSE and thus we strongly prefer packages with small dependency trees.

To be clear, tox is not installed in the environment that actually runs the tests (unless you specifically want that). It manages virtual environments that have exactly the dependencies you want installed and nothing more. I suspect that this will give you fewer dependencies during the run, not more.

We would probably have to eliminate all python setup.py test calls with something more primitive (e.g., python -munittest discover if you don't plan to eliminate that as well), but that seems counterproductive.

Eliminate where or how? For your own projects or for projects you are packaging? tox is similar to make test in that each project can specify its own test commands. python -munittest discover will not work for a large number of packages, which intend for you to use pytest, and presumably also intend to have the dependencies that tox installs.

@mcepl
Copy link
Contributor

mcepl commented Jul 3, 2019

Eliminate where or how? For your own projects or for projects you are packaging? tox is similar to make test in that each project can specify its own test commands. python -munittest discover will not work for a large number of packages, which intend for you to use pytest, and presumably also intend to have the dependencies that tox installs.

Well, the problem with whole virtual environments tox promotes is that it is completely useless in the Linux packaging world: we have all builds working in their own chrooted completely isolated environments so whatever issues virtenv is trying to solve are completely irrelevant for us. Also, we have different mechanisms (full blown isolated builds in separate virtual machines or something of that calibre) for testing with different versions of Python or something.

So, we usually end up with digging into tox.ini and ripping out only unittest (or pytest, it doesn't matter) line and using only that.

@pganssle
Copy link
Member

pganssle commented Jul 3, 2019

@mcepl Well, in that case I think then tox is not actually necessary, since tox is just doing the part that you are doing manually anyway (e.g. setting up the dependencies and environment, then running the tests).

One nice thing about the very declarative nature of tox is that it would probably not be terribly difficult for you to write a script that parses tox.ini and creates your preferred type of build file from it (subject to adjustment since you'd need to map the names of PyPI dependencies to your own system).

In any case, fewer and fewer projects are actually using setup.py test because of the problem of dependency management, so likely deprecating setup.py will create less work for you, since you wouldn't need to support both projects using tox and projects using setup.py test (and extracting tests_require from a non-declarative setup.py file, etc).

@mcepl
Copy link
Contributor

mcepl commented Jul 3, 2019

@mcepl Well, in that case I think then tox is not actually necessary, since tox is just doing the part that you are doing manually anyway (e.g. setting up the dependencies and environment, then running the tests).

That's exactly what I’ve meant, so general feeling that more and more projects rely on tox more heavily makes me a bit uneasy. And yes, setup.py test is probably just more a calling convention than anything useful. We usually rather use pytest test runner directly (even for nose- and unittest- based test suites).

@native-api
Copy link

tox is designed to build and test the package in all Python versions that it officially supports. While setup.py test is supposed to run tests, in the running Python version, assuming the build has already been done.

So, they are doing massively different things. To get comparable experience, a user needs to somehow figure out which tox environment specified ini tox.ini corresponds to the running Python (if there even is such an environment -- tox only supports official CPython releases AFAICS) and pass it to tox -e.

@pganssle
Copy link
Member

pganssle commented Jul 4, 2019

@native-api None of these things are true. While tox is very useful for parametrizing over different Python versions, it is trivial to use the current interpreter, it's even mentioned in this issue: tox -e py.

It certainly supports pypy in the normal parametrized list of environments.

Yes there are differences between tox and setup.py test, because tox does what setup.py test should do. It installs your dependencies in an isolated environment and then runs them.

@native-api
Copy link

native-api commented Jul 4, 2019

@pganssle
None of these things are true.

+1 for the explanations. It's not in the reference documentation so I had no way to know this. (The whole purpose of a reference is to be a complete list of the project's official guarantees.)

@FRidh
Copy link

FRidh commented Jul 4, 2019

In Nixpkgs the default Python builder (that expects setuptools-based projects) runs python setup.py test. Fewer projects implement it nowadays so we typically have to override it. Hopefully soon we can switch the default builder to use pyproject. In that case, we don't have any default command for the test phase, because nothing has been defined and nothing common exists, although it is tempting to just run pytest by default (bootstrapping is a bit more difficult then though).

I can understand the motivation for using Tox, but I suggest that at this point it makes more sense to standardize by going through a PEP. PEP 517/518 is a big improvement, and getting testing included there as well is where we should be going.

@techtonik
Copy link
Contributor

Looks like tox is toxic, because it can not be used without adding a vendor lockin.

Quick <1s test runs for TTD in IDEs that support test-on-edit and using tox for a slow, heavy and thorough CI testing should not be mutually exclusive options.

@RonnyPfannschmidt
Copy link
Contributor

its not ..

@pganssle
Copy link
Member

pganssle commented Jul 4, 2019

I can understand the motivation for using Tox, but I suggest that at this point it makes more sense to standardize by going through a PEP. PEP 517/518 is a big improvement, and getting testing included there as well is where we should be going.

You are welcome to write a draft PEP for this. I suspect that such a thing would be welcomed. I think the reason there has been no clear spec for this is that it's not very common for end users to run a project's tests - usually the people who run the tests are the developers of the package themselves and certain downstream redistributors. The package developers usually have to manually configure CI to run their tests and they don't have to do this at scale, so a standard is less necessary. The redistributors tend to be testing their whole package ecosystem, which means that something defined in terms of the PyPI package ecosystem will still need manual intervention, reducing the need for any sort of standard.

Still, if you and other downstream redistributors of Python packages would like to propose such a standard, I recommend doing so. I think the packaging category or the users category on the Python discourse would be a good place to start. I think you will get a better response if you come in with a concrete proposal rather than creating a general "brainstorming" topic, but that is just my experience with posting such things.

@FRidh
Copy link

FRidh commented Aug 3, 2019

Discussion on Python Packaging discourse about a section in pyproject.toml.
https://discuss.python.org/t/proposal-for-tests-entry-point-in-pyproject-toml/2077

@FRidh FRidh mentioned this issue Oct 24, 2019
2 tasks
kojiromike added a commit to kojiromike/avro that referenced this issue Dec 10, 2019
Python packaging is [moving away from embedding commands in
setup.py][1]. It is difficult to maintain external commands in Python
this way. Managing the dependencies needed to run commands from within
setup.py is gnarly, because dependencies cannot be resolved so early.
Furthermore, it's difficult to test code that happens at the point at
which tests themselves are triggered, so that code needs to be very
simple. A shell script is simple and more appropriate for this use case.

At another time we should look to extract the lint command as well.

[1][pypa/setuptools#931]
@RonaldAJ
Copy link

Wasting my time trying to overcome this change. I don't understand why you can't remove the dependency part while keeping the test runners. What is their relation? To me they are two entirely different things.

@RonaldAJ
Copy link

RonaldAJ commented Jan 27, 2021

Workaround for nose2 using the ability to define your own commands. It can probably be adapted for other testsuites.

The OnSuccess nose2 plugin can be edited out, but it provides me with the ability to remove log-files generated during testing. Only imports specific to this solution are shown.

from setuptools import Command
import nose2
from nose2.events import Plugin

class OnSuccess(Plugin): 
    def wasSuccessful(self, event):
        if event.success:
            print('Success!')
        else:
            print('Failure!')

class nose2Testing(Command):
    """ Run my command.
    """
    description = 'nose2testing'

    user_options = []  # obligatory

    def initialize_options(self):
        self.onsuccesshook = OnSuccess()
        print('initialize_options')

    def finalize_options(self):
        print('finalize_options')

    def run(self):
        nose2.discover(argv=['.'], exit=False, extraHooks=[('wasSuccessful',self.onsuccesshook)])


if __name__ == "__main__":
    setup(
        ...
        cmdclass={
                'nose2': nose2Testing,
            },
    )

Now I can call setup.py:

python setup.py clean build install nose2

@jaraco
Copy link
Member Author

jaraco commented Jan 27, 2021

@RonaldAJ Glad you found a workaround. Note that Setuptools also supports defining that command as a plugin, in a third-party package the way pytest runner does. You could use that to implement the nose2 command.

Still, if it were up to me, I'd try to decouple test running from building, as you're only adding constraints that may later become deprecated as well.

why you can't remove the dependency part while keeping the test runners

Setuptools is aiming to get out of the business of being a swiss-army knife of project management (as distutils was envisioned) and instead focus on providing a best-in-class build implementation (mainly the operations defined by PEP 517). The goal is to separate concerns (SCM tooling, testing, environments, installation, package indexes, distribution, building) into different, largely independent tools, coordinated by standards.

@RonaldAJ
Copy link

@jaraco I guess that good working examples of migration would help. Setuptools is hard to use for people like me who need to make changes very occasionally.

@jaraco
Copy link
Member Author

jaraco commented Jan 27, 2021

That would be nice. Unfortunately, because there are such a diverse array of workflows, there's no deterministic migration. However, for the use-case you indicate above, why not simply pip install .; nose2? That should be roughly equivalent to setup.py clean build install nose2. If you need something more elaborate, then consider using a tool like tox or nox or make to facilitate the setup.

kousu added a commit to shimming-toolbox/shimming-toolbox that referenced this issue Mar 20, 2021
This prevents an unnecessary weight installed by users who
probably don't want to run our unit tests; though it allows
them to, if they read the (as yet unwritten) dev docs or get
a tip from a developer helping them debug their install:
just `pip install shimmingtoolbox[testing]`.

This is the blessed method from the python packaging developers:
pypa/setuptools#1684 (comment)

The other(?) blessed method is to use tox's `deps` option:
pypa/setuptools#931
sloretz added a commit to colcon/colcon-core that referenced this issue Sep 17, 2021
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.

Example installing test dependencies and running pytest:

```
python3 -m venv env3
. env3/bin/activate
cd colcon-core/
pip install -e .[test]
pytest
```

I chose `test`, but the naming is inconsistent. I've found other
examples using `tests` or `testing` as well.

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
sloretz added a commit to colcon/colcon-core that referenced this issue Oct 7, 2021
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.

Example installing test dependencies and running pytest:

```
python3 -m venv env3
. env3/bin/activate
cd colcon-core/
pip install -e .[test]
pytest
```

I chose `test`, but the naming is inconsistent. I've found other
examples using `tests` or `testing` as well.

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
sloretz added a commit to colcon/colcon-core that referenced this issue Oct 8, 2021
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.

Example installing test dependencies and running pytest:

```
python3 -m venv env3
. env3/bin/activate
cd colcon-core/
pip install -e .[test]
pytest
```

I chose `test`, but the naming is inconsistent. I've found other
examples using `tests` or `testing` as well.

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
cottsay added a commit to colcon/colcon-lcov-result that referenced this issue Feb 7, 2024
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.
cottsay added a commit to colcon/colcon-lcov-result that referenced this issue Feb 7, 2024
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.
cottsay added a commit to colcon/colcon-coveragepy-result that referenced this issue Feb 8, 2024
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.
cottsay added a commit to colcon/colcon-coveragepy-result that referenced this issue Feb 9, 2024
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs Discussion Issues where the implementation still needs to be discussed.
Projects
None yet
Development

No branches or pull requests