-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove test command and tests_require #931
Comments
Perhaps its relevant here. After discussing with @RonnyPfannschmidt we added the following to the Tox docs.
As a downstream we prefer to have one command that tests the package for the interpreter that is available and not any other versions. |
I've discovered that |
Yes, I agree, as long as no other envs are defined, then there's no problem. |
|
@techtonik tox works totally fine for dev, you can use |
@graingert how much time does it take on you system to execute a single test with tox. On my it takes more than 5 seconds, which is not acceptable. Also, |
@jaraco instead of test_requires, i'd like to see something that allows |
I don't like the idea of replacing simple test calling with tox with my openSUSE Python maintenance hat on. We strive hard to achieve reproducible builds in openSUSE and thus we strongly prefer packages with small dependency trees. |
To be clear,
Eliminate where or how? For your own projects or for projects you are packaging? |
Well, the problem with whole virtual environments tox promotes is that it is completely useless in the Linux packaging world: we have all builds working in their own chrooted completely isolated environments so whatever issues virtenv is trying to solve are completely irrelevant for us. Also, we have different mechanisms (full blown isolated builds in separate virtual machines or something of that calibre) for testing with different versions of Python or something. So, we usually end up with digging into tox.ini and ripping out only |
@mcepl Well, in that case I think then One nice thing about the very declarative nature of In any case, fewer and fewer projects are actually using |
That's exactly what I’ve meant, so general feeling that more and more projects rely on |
So, they are doing massively different things. To get comparable experience, a user needs to somehow figure out which tox environment specified ini |
@native-api None of these things are true. While It certainly supports Yes there are differences between |
+1 for the explanations. It's not in the reference documentation so I had no way to know this. (The whole purpose of a reference is to be a complete list of the project's official guarantees.) |
In Nixpkgs the default Python builder (that expects I can understand the motivation for using Tox, but I suggest that at this point it makes more sense to standardize by going through a PEP. PEP 517/518 is a big improvement, and getting testing included there as well is where we should be going. |
Looks like Quick <1s test runs for TTD in IDEs that support test-on-edit and using |
its not .. |
You are welcome to write a draft PEP for this. I suspect that such a thing would be welcomed. I think the reason there has been no clear spec for this is that it's not very common for end users to run a project's tests - usually the people who run the tests are the developers of the package themselves and certain downstream redistributors. The package developers usually have to manually configure CI to run their tests and they don't have to do this at scale, so a standard is less necessary. The redistributors tend to be testing their whole package ecosystem, which means that something defined in terms of the PyPI package ecosystem will still need manual intervention, reducing the need for any sort of standard. Still, if you and other downstream redistributors of Python packages would like to propose such a standard, I recommend doing so. I think the packaging category or the users category on the Python discourse would be a good place to start. I think you will get a better response if you come in with a concrete proposal rather than creating a general "brainstorming" topic, but that is just my experience with posting such things. |
Discussion on Python Packaging discourse about a section in |
Python packaging is [moving away from embedding commands in setup.py][1]. It is difficult to maintain external commands in Python this way. Managing the dependencies needed to run commands from within setup.py is gnarly, because dependencies cannot be resolved so early. Furthermore, it's difficult to test code that happens at the point at which tests themselves are triggered, so that code needs to be very simple. A shell script is simple and more appropriate for this use case. At another time we should look to extract the lint command as well. [1][pypa/setuptools#931]
Workaround for nose2 using the ability to define your own commands. It can probably be adapted for other testsuites. The OnSuccess nose2 plugin can be edited out, but it provides me with the ability to remove log-files generated during testing. Only imports specific to this solution are shown. from setuptools import Command
import nose2
from nose2.events import Plugin
class OnSuccess(Plugin):
def wasSuccessful(self, event):
if event.success:
print('Success!')
else:
print('Failure!')
class nose2Testing(Command):
""" Run my command.
"""
description = 'nose2testing'
user_options = [] # obligatory
def initialize_options(self):
self.onsuccesshook = OnSuccess()
print('initialize_options')
def finalize_options(self):
print('finalize_options')
def run(self):
nose2.discover(argv=['.'], exit=False, extraHooks=[('wasSuccessful',self.onsuccesshook)])
if __name__ == "__main__":
setup(
...
cmdclass={
'nose2': nose2Testing,
},
) Now I can call setup.py:
|
@RonaldAJ Glad you found a workaround. Note that Setuptools also supports defining that command as a plugin, in a third-party package the way pytest runner does. You could use that to implement the nose2 command. Still, if it were up to me, I'd try to decouple test running from building, as you're only adding constraints that may later become deprecated as well.
Setuptools is aiming to get out of the business of being a swiss-army knife of project management (as distutils was envisioned) and instead focus on providing a best-in-class build implementation (mainly the operations defined by PEP 517). The goal is to separate concerns (SCM tooling, testing, environments, installation, package indexes, distribution, building) into different, largely independent tools, coordinated by standards. |
@jaraco I guess that good working examples of migration would help. Setuptools is hard to use for people like me who need to make changes very occasionally. |
That would be nice. Unfortunately, because there are such a diverse array of workflows, there's no deterministic migration. However, for the use-case you indicate above, why not simply |
This prevents an unnecessary weight installed by users who probably don't want to run our unit tests; though it allows them to, if they read the (as yet unwritten) dev docs or get a tip from a developer helping them debug their install: just `pip install shimmingtoolbox[testing]`. This is the blessed method from the python packaging developers: pypa/setuptools#1684 (comment) The other(?) blessed method is to use tox's `deps` option: pypa/setuptools#931
https://setuptools.readthedocs.io/en/latest/userguide/commands.html#test-build-package-and-run-a-unittest-suite https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies https://tox.readthedocs.io/en/latest/example/basic.html#integration-with-setup-py-test-command https://pytest-runner.readthedocs.io/en/latest/#deprecation-notice pypa/setuptools#1160 pypa/setuptools#1684 pypa/setuptools#931 pytest-dev/pytest-runner@78a492c PyCQA/flake8#1098 PyCQA/flake8@4b72089
https://setuptools.readthedocs.io/en/latest/userguide/commands.html#test-build-package-and-run-a-unittest-suite https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies https://tox.readthedocs.io/en/latest/example/basic.html#integration-with-setup-py-test-command https://pytest-runner.readthedocs.io/en/latest/#deprecation-notice pypa/setuptools#1160 pypa/setuptools#1684 pypa/setuptools#931 pytest-dev/pytest-runner@78a492c PyCQA/flake8#1098 PyCQA/flake8@4b72089
This replaces `tests_require` with a pattern that seems to have popped up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test dependencies to be installed by adding `[test]` to the end of the package name. Example installing test dependencies and running pytest: ``` python3 -m venv env3 . env3/bin/activate cd colcon-core/ pip install -e .[test] pytest ``` I chose `test`, but the naming is inconsistent. I've found other examples using `tests` or `testing` as well. Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
This replaces `tests_require` with a pattern that seems to have popped up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test dependencies to be installed by adding `[test]` to the end of the package name. Example installing test dependencies and running pytest: ``` python3 -m venv env3 . env3/bin/activate cd colcon-core/ pip install -e .[test] pytest ``` I chose `test`, but the naming is inconsistent. I've found other examples using `tests` or `testing` as well. Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
This replaces `tests_require` with a pattern that seems to have popped up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test dependencies to be installed by adding `[test]` to the end of the package name. Example installing test dependencies and running pytest: ``` python3 -m venv env3 . env3/bin/activate cd colcon-core/ pip install -e .[test] pytest ``` I chose `test`, but the naming is inconsistent. I've found other examples using `tests` or `testing` as well. Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>
This replaces `tests_require` with a pattern that seems to have popped up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test dependencies to be installed by adding `[test]` to the end of the package name.
This replaces `tests_require` with a pattern that seems to have popped up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test dependencies to be installed by adding `[test]` to the end of the package name.
This replaces `tests_require` with a pattern that seems to have popped up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test dependencies to be installed by adding `[test]` to the end of the package name.
This replaces `tests_require` with a pattern that seems to have popped up since pypa/setuptools#931 and pypa/setuptools#1684. It allows test dependencies to be installed by adding `[test]` to the end of the package name.
In this comment, @graingert proposes that we may be able to completely remove support for tests_require instead of transitioning that tooling from
easy_install
topip install
.While he didn't directly propose removal, the effort would only benefit that ticket if the install functionality were completely removed, so let's explore what that will entail.
While I agree that
tox
is an excellent, powerful, robust solution, it's more heavy thantests_require
, requiring that the user have tox installed in advance. As a more thorough solution, it also is subject to bugs and constraints that a simpler test runner is not. There are several advantages tosetup.py test
andtests_require
over tox:setup.py test
) instead of multiple steps (e.g.pip install --user tox; python -m tox
).setup.py test
allows for invocation under a number of different Python versions naturally (i.e.python3.3 setup.py test
orpy -3.3 setup.py test
) whereas tox offers "run under the python in which tox is installed" or "run for explicitly-declared python versions".setup.py test
doesn't use virtualenv, it's not subject to virtualenv bugs or other constraints imposed by virtualenv (such as version 14 dropping support for Python 3.2.).setup.py test
doesn't rely on pip, it's not subject to the bugs of pip (such as issues with--editable
installs or namespace packages) or other constraints imposed by pip (such as dropping support for Python 3.2).I consider these advantages small and easy enough to overcome, especially now that many of these issues have been resolved in setuptools, pip, and virtualenv. If we can get to a place that tox can broadly supplant the uses cases of setup.py test and pytest-runner (and thus tests_require) in practice, then yes, deprecating and removing it would be in order. Given the amount of activity and bugs I see around these tools, I'd asses they're still in active use.
Before flagging this functionality as deprecated, I'd like to survey the community about the possibility to see if there are use cases that would prove difficult to support with tox.
@graingert, would you be interested in being the champion for this effort (removing tests_require), starting with the outreach on distutils-sig and then implementing the deprecation/removal changes?
The text was updated successfully, but these errors were encountered: