Packaging cleanup, migration to full setuptools without distutils

[Pierre-Sassoulas]

Description

Move from distutil to setuptools. (Second step toward pyproject.toml after #4295 ).

We might also want to remove doc/ and tests/ directory from the packaging ? Are you using the tests when packaging for debian @sandrotosi ?

Type of Changes

	Type
✓	🔨 Refactoring

Related Issue

Relates to #4193

[coveralls]

Coverage decreased (-0.02%) to 91.593% when pulling 95994e5 on Pierre-Sassoulas:packaging-cleanup into f46a1fc on PyCQA:master.

[cdce8p]

I would prefer if the metadata where to be defined in setup.cfg instead. See here, for reference: https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html
It might also make sense to separate the build system changes from the removal of bin, debian, and man and put that in another MR.

[Pierre-Sassoulas]

I opened another MR to separate removal of legacy build file and setuptools migration. Regarding the move to setup.cfg I think it's going to be difficult because we have the version defined inside the package and also the test_requires and extra_requires so we need to get the value with code.

[Pierre-Sassoulas]

Nevermind that, most of it is doable with setup.cfg. For version we have pbr and setuptools_scm that could work. I don't know if we could access the version inside pylint for pylint --version though. But the setup.py became very reasonable already.

[cdce8p]

Left some comments. Even if you add an extras_require, please don't remove the requirements_test.txt files. They are need to pin exact versions for CI testing.

[Pierre-Sassoulas]

Please don't remove the requirements_test.txt files. They are need to pin exact versions for CI testing.

I thought the the setup.cfg could also work for that goal ? Are we talking about the hash part ?

[cdce8p]

Please don't remove the requirements_test.txt files. They are need to pin exact versions for CI testing.

I thought the the setup.cfg could also work for that goal ? Are we talking about the hash part ?

Unfortunately extras_require doesn't work and shouldn't be used for that purpose. Tbh I have only seen install -e .[test] as part of legacy configs that haven't been updated yet.
The reason is, that users would expect to be able to install the extra seamlessly in their environment. That is only possible if the requirements aren't fixed as it otherwise might create dependency resolution conflicts. The idea behind the requirements file on the other hand is exactly to provide a fixed environment that can be used for testing and will always yield the same result.

[sandrotosi]

We might also want to remove doc/ and tests/ directory from the packaging ? Are you using the tests when packaging for debian @sandrotosi ?

we do run tests and build the documentation during debian package build, but dont worry about that: i've been gradually migrating all my packages from fetching new upstream releases from PyPI to GitHub releases: it's just easier downstream (we get all the files that we need) and you can keep the PyPI release small (without extra files end users dont really need when pip installing pylint) -- thanks for asking tho!

[Pierre-Sassoulas]

i've been gradually migrating all my packages from fetching new upstream releases from PyPI to GitHub releases: it's just easier downstream (we get all the files that we need) and you can keep the PyPI release small (without extra files end users dont really need when pip installing pylint)

Amazing, we're going to be able to make the package a lot smaller !

[cdce8p]

@Pierre-Sassoulas Regarding the failing benchmark test. The reason is that the test tries to import all modules in the root dir to check for a register function. A patch fixes that:

from unittest.mock import patch

# ....

    def test_baseline_benchmark_j1_all_checks_lots_of_files(self, benchmark):
        # ...
        with patch("os.listdir", return_value=["pylint", "tests"]):
            register_plugins(
                linter, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
            )
        # ...

[cdce8p]

You should also be able to include .isort.cfg and pytest.ini in setup.cfg.