Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate the test command #1878

Merged
merged 1 commit into from Oct 22, 2019
Merged

Deprecate the test command #1878

merged 1 commit into from Oct 22, 2019

Conversation

jdufresne
Copy link
Contributor

@jdufresne jdufresne commented Oct 19, 2019

Refs #1684

Pull Request Checklist

  • Changes have tests
  • News fragment added in changelog.d. See documentation for details

Copy link
Member

@pganssle pganssle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to add some tests for this?

Here is the relevant section of the PR deprecating the upload and register commands.

@@ -2142,6 +2148,12 @@ distutils configuration file the option will be added to (or removed from).
``test`` - Build package and run a unittest suite
=================================================

.. warning::
**test** is deprecated in favor of using `tox
<https://tox.readthedocs.org/>`_, `unittest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be https://tox.readthedocs.io

Also, I think we can reword this a bit - I'd prefer it to be clear that tox is the actual solution here, and the other two are test runners (plus pytest should probably be prioritized over unittest), I'd word it as:

.. warning::
    ``test`` is deprecated in favor of using `tox
    <https://tox.readthedocs.io>`_, or directly invoking your test runner (e.g.
    `pytest <https://docs.pytest.org>`_ or `unittest
    <https://docs.python.org/library/unittest.html>`_).

Another wording that might put our thumbs on the scale in favor of tox a bit more lightly would be:

.. warning::
  ``test`` is deprecated and will be removed in a future version. Users looking
  for a generic test entry point independent of test runner are encouraged to
  use `tox <https://tox.readthedocs.io>`_.

@@ -0,0 +1 @@
Deprecate ``test`` commands.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Deprecate ``test`` commands.
Formally deprecated the ``test`` command, with the recommendation that users migrate to ``tox``.

@pganssle
Copy link
Member

Thank you @jdufresne for your work! I really appreciate the way you've been quietly (at least in my corner of the universe) and methodically improving the overall health of the Python ecosystem.

Provide a warning to users. Suggest using tox as an alternative generic
entry point.

Refs #1684
@jdufresne
Copy link
Contributor Author

Thanks for the review. I have implemented your suggestions and and added tests. I went with the stronger warning as I really like the "and will be removed in a future version" phrasing, but I'm happy to switch back to the softer wording if preferred. If you're looking for anything else out of the tests, let me know and I'll add them.

And thanks very much for the kind words! 🙂

@pganssle pganssle merged commit a768f8d into pypa:master Oct 22, 2019
@pganssle pganssle mentioned this pull request Oct 22, 2019
@graingert
Copy link
Contributor

It may be worth locking this PR and the associated issue and directing people to https://discuss.python.org/

@pganssle
Copy link
Member

@graingert Hm.. You're right that this will likely be a magnet for people upset by this news, but I think maybe we should leave it open. That will at least give people who don't want to follow possible objections a single thread (or maybe two threads) to ignore, and some people may indeed have constructive comments to make on the specifics of the implementation.

Let's revisit the issue of locking the thread later if the thread becomes toxic (anyone who feels that the thread is going that way, please feel free to send me an e-mail).

For anyone coming to this PR to complain or because someone linked it: please be respectful and try to understand the full history of the issue. I know it is voluminous, but please read all of #1684 before commenting.

@jdufresne jdufresne deleted the deprecate-test-suite branch October 22, 2019 17:43
@jdufresne jdufresne restored the deprecate-test-suite branch October 22, 2019 17:43
@jdufresne jdufresne deleted the deprecate-test-suite branch October 22, 2019 17:43
@FRidh
Copy link

FRidh commented Oct 24, 2019

It's a pity this PR wasn't mentioned in the original discussion about removing the test command.
#931 and neither at https://discuss.python.org/t/proposal-for-tests-entry-point-in-pyproject-toml/2077.

I don't think we should encourage users to use tox for this, as it's a too heavy solution for those simply wanting to test their installation, and would appreciate if the encouraged was replaced with may want. OTOh I do think encouraging tox is good because it keeps it uniform which was also the motivation of my discuss.python.org proposal.

@ChadBailey
Copy link

FYI i found this because while running tests via tox I received this warning

$ tox
GLOB sdist-make: C:\Users\Kamel\Documents\code\package_demo\setup.py
py27 create: C:\Users\Kamel\Documents\code\package_demo\.tox\py27
ERROR: InterpreterNotFound: python2.7
py35 create: C:\Users\Kamel\Documents\code\package_demo\.tox\py35
ERROR: InterpreterNotFound: python3.5
py36 create: C:\Users\Kamel\Documents\code\package_demo\.tox\py36
ERROR: InterpreterNotFound: python3.6
py37 inst-nodeps: C:\Users\Kamel\Documents\code\package_demo\.tox\.tmp\package\1\package_demo-0.1.0.zip
py37 installed: package-demo==0.1.0
py37 run-test-pre: PYTHONHASHSEED='31'
py37 run-test: commands[0] | python setup.py test
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
running egg_info
writing package_demo.egg-info\PKG-INFO
writing dependency_links to package_demo.egg-info\dependency_links.txt
writing top-level names to package_demo.egg-info\top_level.txt
reading manifest file 'package_demo.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found under directory '*'
warning: no files found matching '*.jpg' under directory 'docs'
warning: no files found matching '*.png' under directory 'docs'
warning: no files found matching '*.gif' under directory 'docs'
writing manifest file 'package_demo.egg-info\SOURCES.txt'
running build_ext
test_hello (tests.test_package_demo.TestPackage_demo)
Test something. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
py38 create: C:\Users\Kamel\Documents\code\package_demo\.tox\py38
ERROR: InterpreterNotFound: python3.8
flake8 inst-nodeps: C:\Users\Kamel\Documents\code\package_demo\.tox\.tmp\package\1\package_demo-0.1.0.zip
flake8 installed: entrypoints==0.3,flake8==3.7.9,mccabe==0.6.1,package-demo==0.1.0,pycodestyle==2.5.0,pyflakes==2.1.1
flake8 run-test-pre: PYTHONHASHSEED='31'
flake8 run-test: commands[0] | flake8 package_demo
______________________________________________________________________________________________________________________________ summary _______________________________________________________________________________________________________________________________
ERROR:  py27: InterpreterNotFound: python2.7
ERROR:  py35: InterpreterNotFound: python3.5
ERROR:  py36: InterpreterNotFound: python3.6
  py37: commands succeeded
ERROR:  py38: InterpreterNotFound: python3.8
  flake8: commands succeeded

I'm new to tox, so I'm not asserting that the implementation is flawed... however, the user experience of running tox then receiving a warning to use tox instead is not great.

The package under test in this demo is this cookiecutter template: https://github.com/audreyr/cookiecutter-pypackage with a "hello world" function added/tested.

@graingert
Copy link
Contributor

This is because you ran setup.py test within tox

FYI i found this because while running tests via tox I received this warning

$ tox
GLOB sdist-make: C:\Users\Kamel\Documents\code\package_demo\setup.py
py27 create: C:\Users\Kamel\Documents\code\package_demo\.tox\py27
ERROR: InterpreterNotFound: python2.7
py35 create: C:\Users\Kamel\Documents\code\package_demo\.tox\py35
ERROR: InterpreterNotFound: python3.5
py36 create: C:\Users\Kamel\Documents\code\package_demo\.tox\py36
ERROR: InterpreterNotFound: python3.6
py37 inst-nodeps: C:\Users\Kamel\Documents\code\package_demo\.tox\.tmp\package\1\package_demo-0.1.0.zip
py37 installed: package-demo==0.1.0
py37 run-test-pre: PYTHONHASHSEED='31'
py37 run-test: commands[0] | python setup.py test
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
running egg_info
writing package_demo.egg-info\PKG-INFO
writing dependency_links to package_demo.egg-info\dependency_links.txt
writing top-level names to package_demo.egg-info\top_level.txt
reading manifest file 'package_demo.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found under directory '*'
warning: no files found matching '*.jpg' under directory 'docs'
warning: no files found matching '*.png' under directory 'docs'
warning: no files found matching '*.gif' under directory 'docs'
writing manifest file 'package_demo.egg-info\SOURCES.txt'
running build_ext
test_hello (tests.test_package_demo.TestPackage_demo)
Test something. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
py38 create: C:\Users\Kamel\Documents\code\package_demo\.tox\py38
ERROR: InterpreterNotFound: python3.8
flake8 inst-nodeps: C:\Users\Kamel\Documents\code\package_demo\.tox\.tmp\package\1\package_demo-0.1.0.zip
flake8 installed: entrypoints==0.3,flake8==3.7.9,mccabe==0.6.1,package-demo==0.1.0,pycodestyle==2.5.0,pyflakes==2.1.1
flake8 run-test-pre: PYTHONHASHSEED='31'
flake8 run-test: commands[0] | flake8 package_demo
______________________________________________________________________________________________________________________________ summary _______________________________________________________________________________________________________________________________
ERROR:  py27: InterpreterNotFound: python2.7
ERROR:  py35: InterpreterNotFound: python3.5
ERROR:  py36: InterpreterNotFound: python3.6
  py37: commands succeeded
ERROR:  py38: InterpreterNotFound: python3.8
  flake8: commands succeeded

I'm new to tox, so I'm not asserting that the implementation is flawed... however, the user experience of running tox then receiving a warning to use tox instead is not great.

The package under test in this demo is this cookiecutter template: audreyr/cookiecutter-pypackage with a "hello world" function added/tested.

@graingert
Copy link
Contributor

@ChadBailey there's already an issue for this audreyfeldroy/cookiecutter-pypackage#500

@ChadBailey
Copy link

ChadBailey commented Nov 23, 2019

@graingert thanks for pointing that out... distribution in python is new territory for me, so plenty to learn. I did not discover (nor consider looking for) an existing bug in the cookiecutter package.

Perhaps a quick-fix to the concern I raised is to update the verbiage to the following:

WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox. If you are seeing this message while running tox, please update your tox configuration to not invoke setup.py

I did in fact see the warning in the tox documentation about how integrating tox into setup.py is discouraged due to breaking downstream packages, but still trying to sort all of this out in my mind.

If you wouldn't mind, could you confirm if my new understanding the best practice for testing in python is correct?

  • Use tox in lieu of setup.py test
  • Testing dependencies/requirements should not be specified in setup.py in the tests_require section
  • Testing dependencies/requirements should be specified in tox.ini in the [testenv] section
  • tox.ini should not contain any references to setup.py

@ChadBailey
Copy link

@graingert Thanks for your help, I've submitted a PR audreyfeldroy/cookiecutter-pypackage#544 to fix audreyfeldroy/cookiecutter-pypackage#500

eldercrow pushed a commit to eldercrow/open_model_zoo that referenced this pull request Feb 18, 2020
The "setup.py test" command is deprecated (see
pypa/setuptools#1878). I'd like to transition
CI to running pytest directly, but for that, I need an easy way to install
test dependencies. This provides such a way.
birdsarah added a commit to openwpm/domain_utils that referenced this pull request Apr 2, 2020
birdsarah added a commit to openwpm/domain_utils that referenced this pull request Apr 2, 2020
rohitpaulk added a commit to rohitpaulk/analytics-python that referenced this pull request Apr 16, 2020
Also replaced tests_require with extras_require. The test command (which
tests_required was used for) was deprecated in setuptools 41.5.0.

Ref: pypa/setuptools#1878
rohitpaulk added a commit to rohitpaulk/analytics-python that referenced this pull request Apr 16, 2020
Also replaced tests_require with extras_require. The test command (which
tests_required was used for) was deprecated in setuptools 41.5.0.

Ref: pypa/setuptools#1878
rohitpaulk added a commit to rohitpaulk/analytics-python that referenced this pull request Apr 16, 2020
Also replaced tests_require with extras_require. The test command (which
tests_required was used for) was deprecated in setuptools 41.5.0.

Ref: pypa/setuptools#1878
rohitpaulk added a commit to rohitpaulk/analytics-python that referenced this pull request Apr 17, 2020
Also replaced tests_require with extras_require. The test command (which
tests_required was used for) was deprecated in setuptools 41.5.0.

Ref: pypa/setuptools#1878
@ssbarnea
Copy link

One thing is for sure, putting test dependencies inside tox.ini is also a bad practice, a convenient one but still bad. This creates a dependency on a tool making very hard to use another test runner than tox.

While this was not documented, I have reasons to believe that adding a 'test' extra would be a much better and future proof way to declare dependencies. I also seen the same made with docs dependencies.

adrienverge added a commit to adrienverge/yamllint that referenced this pull request Sep 29, 2020
Using `python setup.py test` is now deprecated [1], users are encouraged
to be explicit about the test command.

Running yamllint tests using the Python standard library (`unittest`)
can be done using:

    python -m unittest discover

Why not nose, tox or pytest? Because they would add a dependency, make
tests running more complicated and verbose for new users, and their
benefit is not worth for this simple projects (only 2 runtime
dependencies: PyYAML and pathspec).

[1]: pypa/setuptools#1878
adrienverge added a commit to adrienverge/yamllint that referenced this pull request Sep 29, 2020
Using `python setup.py test` is now deprecated [1], users are encouraged
to be explicit about the test command.

Running yamllint tests using the Python standard library (`unittest`)
can be done using:

    python -m unittest discover

Why not nose, tox or pytest? Because they would add a dependency, make
tests running more complicated and verbose for new users, and their
benefit is not worth for this simple project (only 2 runtime
dependencies: PyYAML and pathspec).

Resolves #328.

[1]: pypa/setuptools#1878
adrienverge added a commit to adrienverge/yamllint that referenced this pull request Sep 29, 2020
Using `python setup.py test` is now deprecated [1], users are encouraged
to be explicit about the test command.

Running yamllint tests using the Python standard library (`unittest`)
can be done using:

    python -m unittest discover

Why not nose, tox or pytest? Because they would add a dependency, make
tests running more complicated and verbose for new users, and their
benefit is not worth for this simple project (only 2 runtime
dependencies: PyYAML and pathspec).

Resolves #328.

[1]: pypa/setuptools#1878
adrienverge added a commit to adrienverge/yamllint that referenced this pull request Sep 30, 2020
Using `python setup.py test` is now deprecated [1], users are encouraged
to be explicit about the test command.

Running yamllint tests using the Python standard library (`unittest`)
can be done using:

    python -m unittest discover

Why not nose, tox or pytest? Because they would add a dependency, make
tests running more complicated and verbose for new users, and their
benefit is not worth for this simple project (only 2 runtime
dependencies: PyYAML and pathspec).

Resolves #328.

[1]: pypa/setuptools#1878
ssato pushed a commit to ssato/yamllint that referenced this pull request Oct 1, 2020
Using `python setup.py test` is now deprecated [1], users are encouraged
to be explicit about the test command.

Running yamllint tests using the Python standard library (`unittest`)
can be done using:

    python -m unittest discover

Why not nose, tox or pytest? Because they would add a dependency, make
tests running more complicated and verbose for new users, and their
benefit is not worth for this simple project (only 2 runtime
dependencies: PyYAML and pathspec).

Resolves adrienverge#328.

[1]: pypa/setuptools#1878
@jab jab mentioned this pull request Feb 15, 2021
ods added a commit to ods/aiochsa that referenced this pull request Feb 18, 2021
`setup.py test` is deprecated for a quite long time:
pypa/setuptools#1878
@Jasha10 Jasha10 mentioned this pull request Oct 18, 2021
Harmon758 added a commit to Harmon758/tweepy that referenced this pull request Apr 6, 2022
setup.py test has been deprecated with pypa/setuptools#1684 and pypa/setuptools#1878 since setuptools v41.5.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants