Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate test command #1684

Closed
pganssle opened this issue Feb 12, 2019 · 34 comments
Closed

Deprecate test command #1684

pganssle opened this issue Feb 12, 2019 · 34 comments

Comments

@pganssle
Copy link
Member

@pganssle pganssle commented Feb 12, 2019

I think there's at least some agreement in #931 that we want to remove the test command. I think we should start by raising deprecation warnings pointing people at tox, the same way we've done for the upload and register commands.

The most likely stumbling block that I see is that I think a huge number of people have created their own test command that invokes their preferred test runner, pytest or whatever. Ideally we'd want to get the deprecation warning to them as well. Hopefully most of them are using TestCommand as their base class, though if we want to get really aggressive about it we could try parsing sys.argv directly.

I think we need to warn in the following situations:

  1. If the setup.py test command is executed
  2. If tests_require is specified
  3. If aliases.test is specified in setup.cfg

It's likely that at least two of these will be specified, but I think two separate warnings would be useful.

CC: @gaborbernat @RonnyPfannschmidt @nicoddemus

@jaraco
Copy link
Member

@jaraco jaraco commented Feb 14, 2019

The pytest-runner project does subclass the test command, and there are projects actively depending on it. I do agree we want to deprecate each, and probably both together.

Loading

@di
Copy link
Member

@di di commented Feb 17, 2019

A bit off-topic, but do we have an overarching issue for deprecating/replacing the various incantations of setup.py? If not, would probably be good to have this, as well as some documentation about migrating on https://packaging.python.org/.

(I can create these issues if they don't already exist)

Loading

@jaraco
Copy link
Member

@jaraco jaraco commented Mar 9, 2019

do we have an overarching issue for deprecating/replacing the various incantations of setup.py?

Not to my knowledge.

Loading

@native-api
Copy link

@native-api native-api commented Jul 2, 2019

I think there's at least some agreement in #931 that we want to remove the test command. I think we should start by raising deprecation warnings pointing people at tox

Is there, really? The feeedback there is basically that tox is a large dependency and is designed for a different use case than setup.py test.

Loading

@jayvdb
Copy link
Contributor

@jayvdb jayvdb commented Jul 3, 2019

This is a great way to kill sdists. (why have sdists if they have the same contents as wheels) How quickly the lesson of sourceforge.net has been forgotten. Or more recently Oracle changing the Term of Use of the Java downloads. IMO setuptools should be moving in the opposite direction, automatically including in sdists the test modules found using test_require or identified using pytest dry-run.

Anyways, unless setuptools also kills its plugin mechanism, someone will recreate the test command as soon as it is removed from setuptools. Maybe that is a good thing. (But if the goal is only to get that stuff out of the setuptools repo because it is hindering setuptools, why not split it off to a separate project now, rather than pretend that tox is a drop in solution)

Loading

@jaraco
Copy link
Member

@jaraco jaraco commented Jul 3, 2019

Is there, really? The feeedback there is basically that tox is a large dependency and is designed for a different use case than setup.py test.

Let me see if I can clarify/illustrate. There are two main concerns of setup.py test:

  • install dependencies in an environment (the current one)
  • invoke a test runner

Tox happens to do both of these in a user-friendly, declarative way, which is what makes it a recommended replacement. The user-experience of "run tox" is comparable to "run setup.py test".

tox does bring more sophistication (and thus friction) to the process, which is why it's somewhat not an exact match.

Users are welcome to seek out other solutions. I wrote pip-run as one such attempt. It has a different behavior in that it always installs dependencies, so doesn't have the second-run speed improvements seen with setup.py test.

why not split it off to a separate project now

You're welcome to consider that. The biggest issue with setup.py test now is that it relies on easy_install and setup_requires, both of which are superseded by pip and PEP 517. Another issue is that setup.py test has no mechanism to specify the version of setuptools under which to run... dependencies need to be resolved before setup.py is invoked. Any tools that attempts to use setuptools plugins to run tests will be forever be responsible to ensure the requisite version of setuptools is installed in advance (it's too late by the time setuptools is imported).

This is a great way to kill sdists. (why have sdists if they have the same contents as wheels) How quickly the lesson of sourceforge.net has been forgotten.

I'm not sure I follow. Can you elaborate on what the risks are?

IMO setuptools should be moving in the opposite direction, automatically including in sdists the test modules found using test_require or identified using pytest dry-run.

I don't think you want to bundle any dependencies, even those for tests, for the same reasons as you don't want to bundle dependencies (you don't want to have to re-release your sdists every time a dependency updates). You want a lightweight, declarative means to declare dependencies. If bundled dependencies are valuable, a higher-level tool should build on the fundamental building blocks (independent packages) to produce and publish those bundles (similar to how Dockerhub works to host materialized environments).

Loading

@pganssle
Copy link
Member Author

@pganssle pganssle commented Jul 3, 2019

This is a great way to kill sdists. (why have sdists if they have the same contents as wheels)

To be clear, this would change nothing about the distinction between sdists and wheels. I personally still recommend that you include your test code in your sdist, but not in the wheel. I also recommend that your tox.ini and any other test configuration should also go into the sdist.

python setup.py test has raised an error in dateutil for a few years now, but it has not affected the content or utility of the source distributions.

Loading

@jayvdb
Copy link
Contributor

@jayvdb jayvdb commented Jul 3, 2019

To be clear, this would change nothing about the distinction between sdists and wheels.

The reason to include tests in sdists, or even create sdists, becomes less obvious if the existing declarative syntax to describe them causes a deprecation warning. This will obviously mean packages with existing working metadata which describes their tests to sdist recipients will need to remove that metadata to be compatible with the latest setuptools. And the latest setuptools wont run the tests any more ... ? So again, why would those package maintainers bother with including tests in the sdist.

Add to that there are members of PyPA who were already pushing projects to not include tests in sdists (because GitHub!), who will use this deprecation as additional justification for being more vocal and aggressive in their stance.

These are not direct outcomes of this issue. However, they are fairly predictable side-effects.

I also recommend that your tox.ini and any other test configuration should also go into the sdist.

Great, but you are not the typical package maintainer. Unfortunately most sdists these days are not testable, usually missing test configuration or test data, until someone from a distro tries to get the sdist working, and then has to try to convince the package maintainer because the Python Packaging Guide says almost nothing about tests. Rather than trying to use @pypa resources and products to address that growing problem, it seems the developers are washing their hands of the problem, and write so many conflicting PEPs that replace each other before a significant segment of the user base has adopted the previous one usually because the implementations were all only partial completed before the authors started writing a different PEP. Compare with the Perl ecosystem, where running the tests is an automatic default part of the package installation experience, and buildbots can (and do!) run the tests of almost every package because they all use consistent metadata for finding and running the tests - packages unmodified for 10 years still build. Tox manages creating virtualenvs and running python(OS-agnostic) commands in them. It does not declaratively describe the test modules nor the test runner (I dont consider the testenv commands = to be declarative.) If setuptools isnt going to allow its metadata to describe unittest test modules, maybe pytest needs to become the recommended test runner, as its metadata in setup.cfg/pytest.ini/etc can at least hold the globs needed to know which modules are test modules and which are not.

Loading

@graingert

This comment was marked as off-topic.

@native-api
Copy link

@native-api native-api commented Jul 4, 2019

The biggest issue with setup.py test now is that it relies on easy_install and setup_requires, both of which are superseded by pip and PEP 517.

If you are going to drastically reduce the setuptools' scope from a complete package management solution to just a build tool, please say that explicitly rather than trying to disguise it as a minor change.
(That said, then you probably want to deprecate setup.py install as well since it relies on the same "superseded" easy_install and bdist_egg. And other *requires arguments as well.)

The biggest issue with setup.py test now is that it relies on easy_install and setup_requires, both of which are superseded by pip and PEP 517

Likewise, if the real reason to deprecate it is that for all practical purposes, it has long been broken (which is true), and you have no desire to fix it, stating that openly is going to remove a whole lot of misunderstanding.

A fix would be to delegate "superseded" stuff to whatever it was superseded by -- to pip for installation and virtualenv (or whatever would work, this is just from the top of my head) to add a custom site-packages location.


@graingert

It would be good to drop sdists entirely for projects that ship pure python wheels

How would I get setup.py from a built wheel?

@jayvdb

Compare with the Perl ecosystem, where running the tests is an automatic default part of the package installation experience

Perl is much less scalable than Python and a a result, CPAN modules tend to be very small in comparison. I can't name a single Perl module whose tests consume a few gigabytes and take a few hours to run, making it impractical to run them as a part of every installation.

Loading

@pganssle
Copy link
Member Author

@pganssle pganssle commented Jul 4, 2019

If you are going to drastically reduce the setuptools' scope from a complete package management solution to just a build tool, please say that explicitly rather than trying to disguise it as a minor change.

I don't think we've ever tried to disguise this. For several years now we have been actively warning people not to invoke setup.py directly at every opportunity, and we've been quite explicit that the end goal is for setuptools to lose as many of its extraneous features as possible and become a standard build tool.

(That said, then you probably want to deprecate setup.py install as well since it relies on the same "superseded" easy_install and bdist_egg. And other *requires arguments as well.)

Yes, we fully intend to remove setup.py install. At the moment bdist_wheel relies on the install command, which is the only reason we have not started raising warnings.

Loading

@native-api
Copy link

@native-api native-api commented Jul 4, 2019

I don't think we've ever tried to disguise this. For several years now we have been actively warning people

Could you reference some overarching issue on this initiative in the top post then? That would work much better as justification than "some agreement" in some close-circle discussion.


At the moment bdist_wheel relies on the install command, which is the only reason we have not started raising warnings.

Raising a warning only if it's invoked via public interface should address that.

Loading

@jayvdb
Copy link
Contributor

@jayvdb jayvdb commented Jul 4, 2019

I can't name a single Perl module whose tests [compare with opencv] consume a few gigabytes and take a few hours to run, making it impractical to run them as a part of every installation.

I am not suggesting that Python packages should run tests by default when installing packages (, or that anyone should use Perl). Sure, PyPI hold entries for much larger codebases than are typically in CPAN modules. Large Perl codebases are often not in CPAN at all, and while there are noble attempts like PDL and BioPerl, most people working on large datasets have moved from Perl to R or Python, so there is more Python development occurring now when data sizes considered almost impossible 10 years ago are common place now. fwiw, R has tests also occurring during installation, and CRAN has similar buildbots, and there are some large R testsuites - not sure if there are any as big as opencv . So it is likely more useful to contrast with R instead of Perl.

It would be nice if there was a new "standard" way to discover tests for any Python packages before setuptools deprecates the existing mechanism that packages are exposing that metadata. Deprecation in software engineering usually means there is a documented replacement which is (roughly) fit for purpose.

Loading

@pganssle
Copy link
Member Author

@pganssle pganssle commented Jul 4, 2019

It would be nice if there was a new "standard" way to discover tests for any Python packages before setuptools deprecates the existing mechanism that packages are exposing that metadata. Deprecation in software engineering usually means there is a documented replacement which is (roughly) fit for purpose.

The problem that I think most people in this thread are having is that there currently is no standard way to execute tests in Python. To the extent that setup.py test was ever that way, it has dramatically declined in popularity, partially due to the rise in popularity of pytest and tox and partially due to the fact that setup.py test never really worked all that well to start with. Some projects use make test and some use some custom shell script.

To the extent that we could fix the problems with setup.py test, we would essentially just be re-inventing tox, because the requirements of a non-broken test target would be that it has a declarative configuration format (tox.ini), a clear way to declare dependencies (deps=) and can build and install those dependencies in an isolated environment (this is what tox does).

In any case, pretending that setup.py test is supported or non-broken is not helping anyone. There are multiple good replacements for setup.py test, I think it's time that we start warning people that they should migrate to one of those.

Loading

@jayvdb
Copy link
Contributor

@jayvdb jayvdb commented Jul 5, 2019

The problem that I think most people in this thread are having is that there currently is no standard way to execute tests in Python.

setup.py test is that standard way to execute tests. It has been like that for one and a half decades. It has been baked into uni courses and peoples scripts. It is the basis of Django's manage.py test.

Failing that, unittest test discovery is also standard https://docs.python.org/3/library/unittest.html#unittest-test-discovery

pytest is the very large elephant in the room. A lot of Python direction problems would be solved by declaring that pytest is the new standard (in addition to unittest). Wrap it in a PEP. ;-)

setup.py test never really worked all that well to start with.

It works very well for lots of packages.

It doesnt work well for all packages, of course, but that isnt reason to break it for all packages before there is a good replacement.
PEP 517 didnt cover test running, but it has laid the groundwork for it, and it wont be long before PEP 517/8 build systems incorporate solutions for that problem and/or a PEP is accepted which does standardised the hooks for test running.

setuptools should keep supporting its existing user-base, keep them working correctly, with all of the features they rely on.

With PEP 517/8, new build systems can flourish. The goal should be for setuptools to be deprecating itself, and the deprecation notice provide a list of all high quality replacement PEP 517/8 build systems which have implemented drop-in solutions for the many varied types of projects which setuptools has supported for the last decade or more. That would be in the spirit of PEP 517, and would avoid setuptools existing near-monopoly implicitly preventing other build systems from being used and supported.

Loading

@RonnyPfannschmidt
Copy link
Contributor

@RonnyPfannschmidt RonnyPfannschmidt commented Jul 5, 2019

status quo is setuptools maintaining legacy things with a really thinly stretched set of maintainers, all while nobody actually steps up to make things better

from my pov a "working" solution is the main impediment for having someone step up and do a better/good solution

Loading

@native-api
Copy link

@native-api native-api commented Jul 5, 2019

from my pov a "working" solution is the main impediment for having someone step up and do a better/good solution

A fix would be to delegate "superseded" stuff to whatever it was superseded by -- to pip for installation and virtualenv (or whatever would work, this is just from the top of my head) to add a custom site-packages location.

Any volunteers?

Loading

@pganssle
Copy link
Member Author

@pganssle pganssle commented Jul 6, 2019

A fix would be to delegate "superseded" stuff to whatever it was superseded by -- to pip for installation and virtualenv (or whatever would work, this is just from the top of my head) to add a custom site-packages location.

This is also something that we considered, but at the end of the day I think it would be much better to just deprecate these functions because we want to train people to use the right tool for the job, and invoking setup.py is essentially never the right tool for the job - at best it will end up being a thin wrapper around our preferred tool. What's worse is that if you have an old version of setuptools it will end up not being a pass-through to a supported tool, which can screw up your environment.

If we continue supporting these things instead of just warning and eventually erroring out with a message that tells you about the migration path, people will continue with the erroneous belief that executing setup.py is the standard and/or recommended way to do things. They will put it in their documentation and teach it in classes. Instead it's better for us to start communicating clearly that these things are and have been unsupported for a long time and start migrating to the new way of doing things.

Loading

@native-api
Copy link

@native-api native-api commented May 1, 2021

@pganssle @jaraco tests_require is still not deprecated and is causing problems due to being broken.

Loading

@jaraco
Copy link
Member

@jaraco jaraco commented May 19, 2021

To be more precise, tests_require isn't broken, but rather it doesn't implement newer features (PEP 503) that honor downloading requirements based on Python version. It still works as well as it always has worked, where developers were required to restrict their test dependencies to versions that were compatible with the relevant python versions. And actually, in late versions of Setuptools, if pip is present, it's preferred for installing packages, so should honor PEP 503 Requires-Python directives.

And to be sure, the test command is deprecated and the pytest-runner project is deprecated and as far as I know, those two projects are the only two that invoke tests_require, so I'm unsure of the value of deprecating the install_dists method.

If there's something more than needs to be done here, please open a new issue describing the missed expectation and possibly proposing an approach to address it.

Loading

soxofaan added a commit to Open-EO/openeo-python-client that referenced this issue Jul 7, 2021
pytest-runner breaks python 3.5 build in GitHub Actions
setup_requires/pytest-runner are deprecated anyway
(pypa/setuptools#1684)
ShaddyDC added a commit to ShaddyDC/osu_reader that referenced this issue Sep 16, 2021
sloretz added a commit to colcon/colcon-core that referenced this issue Sep 17, 2021
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.

Example installing test dependencies and running pytest:

```
python3 -m venv env3
. env3/bin/activate
cd colcon-core/
pip install -e .[test]
pytest
```

I chose `test`, but the naming is inconsistent. I've found other
examples using `tests` or `testing` as well.

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
sloretz added a commit to colcon/colcon-core that referenced this issue Oct 7, 2021
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.

Example installing test dependencies and running pytest:

```
python3 -m venv env3
. env3/bin/activate
cd colcon-core/
pip install -e .[test]
pytest
```

I chose `test`, but the naming is inconsistent. I've found other
examples using `tests` or `testing` as well.

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
sloretz added a commit to colcon/colcon-core that referenced this issue Oct 8, 2021
This replaces `tests_require` with a pattern that seems to have popped
up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test
dependencies to be installed by adding `[test]` to the end of the
package name.

Example installing test dependencies and running pytest:

```
python3 -m venv env3
. env3/bin/activate
cd colcon-core/
pip install -e .[test]
pytest
```

I chose `test`, but the naming is inconsistent. I've found other
examples using `tests` or `testing` as well.

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
Julien00859 referenced this issue in readthedocs-fr/bin-server Oct 19, 2021
In python there are source distribution (sdist) with developers and
contributers as target audience and there are build distribution (bdist)
with users as target audience.

A sdist will contain everything that is needed for developers to build,
test and document the source code while the bdist usualy just contains
an executable.

In our case, not only python source code should be included in both
distributions but also web assets and html templates. By default they
are not included nor in the sdist, nor in the bdist. In case of a source
distribution, the files are to be included in the `MANIFEST.in` file. In
case of a build distribution, the files are to be included in the
`package_data` option of setup.cfg. Because it is possible to install a
source distribution (`pip install bin.x.x.x.tar.gz`), every file needed
to build the sources must be included in the source distribution too.

Long story cut short : `MANIFEST.in` controls what's included in the
sdist. By default it includes a few metadata files (setup.(py|cfg),
readmes, ...) and python packages. The `package` and `package_data`
options of `setup.cfg` control what's included in the bdist.

1/ `package` now only contain `bin` and excluses other python packages,
   the unittests are no more included in the bdist, it is not necessary
   for users.
2/ `package_data` and `MANIFEST.in` now include the various web assets
   they are required to run the application.
3/ `MANIFEST.in` includes the unittests as they are valuable to devs
4/ `MANIFEST.in` includes `VERSION` as it is required by setup.py and
   because it is possible to pip install the sdist.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet