[BUG] Deprecation warning when building wheels

[domdfcoding]

setuptools version

setuptools==58.3.0 or setuptools==58.4.0

Python version

Python 3.8

OS

Ubuntu 20.04

Additional environment information

No response

Description

Attempting to install or build a wheel of a project using setuptools results in the following warning:

venv/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.

Full traceback:

  Traceback (most recent call last):
    File "venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
      self.run_setup()
    File "venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 20, in <module>
      setup(
    File "venv/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "venv/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 301, in run
      install = self.reinitialize_command('install',
    File "/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 214, in reinitialize_command
      cmd = _Command.reinitialize_command(self, command, reinit_subcommands)
    File "/usr/lib/python3.8/distutils/cmd.py", line 305, in reinitialize_command
      return self.distribution.reinitialize_command(command,
    File "/usr/lib/python3.8/distutils/dist.py", line 938, in reinitialize_command
      command = self.get_command_obj(command_name)
    File "/usr/lib/python3.8/distutils/dist.py", line 858, in get_command_obj
      cmd_obj = self.command_obj[command] = klass(self)
    File "venv/lib/python3.8/site-packages/setuptools/__init__.py", line 178, in __init__
      _Command.__init__(self, dist)
    File "/usr/lib/python3.8/distutils/cmd.py", line 62, in __init__
      self.initialize_options()
    File "/venv/lib/python3.8/site-packages/setuptools/command/install.py", line 34, in initialize_options
      warnings.warn(
  setuptools._deprecation_warning.SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.

This occurs with both build and pip (pip install and pip wheel).

Expected behavior

No warning is emitted when building a wheel or installing using pip or build.

How to Reproduce

1. Install setuptools 58.3.0 or 58.4.0
2. Run PYTHONWARNINGS=error pip wheel domdf_python_tools==3.1.0 --no-binary domdf_python_tools -v --no-deps

This also occurs with whey and natsort.

Output

Looking in indexes: http://localhost:3141/root/staging
Collecting domdf_python_tools==3.1.0
  Downloading http://localhost:3141/root/pypi/%2Bf/c55/b356a87d53841/domdf-python-tools-3.1.0.tar.gz (99 kB)
  Installing build dependencies: started
  Running command /tmp/venv/bin/python3 /tmp/pip-standalone-pip-7m38gblg/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-myy0rzw0/overlay --no-warn-script-location --no-binary domdf-python-tools --only-binary :none: -i http://localhost:3141/root/staging -- 'setuptools>=40.6.0' 'wheel>=0.34.2'
  Looking in indexes: http://localhost:3141/root/staging
  Collecting setuptools>=40.6.0
    Downloading http://localhost:3141/root/pypi/%2Bf/e8b/1d3127a0441fb/setuptools-58.4.0-py3-none-any.whl (946 kB)
  Collecting wheel>=0.34.2
    Downloading http://localhost:3141/root/pypi/%2Bf/210/14b2bd93c6d00/wheel-0.37.0-py2.py3-none-any.whl (35 kB)
  Installing collected packages: wheel, setuptools
  Successfully installed setuptools-58.4.0 wheel-0.37.0
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command /tmp/venv/bin/python3 /tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp32ew7q_j
  running egg_info
  writing domdf_python_tools.egg-info/PKG-INFO
  writing dependency_links to domdf_python_tools.egg-info/dependency_links.txt
  writing requirements to domdf_python_tools.egg-info/requires.txt
  writing top-level names to domdf_python_tools.egg-info/top_level.txt
  reading manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching '**/__pycache__'
  adding license file 'LICENSE'
  writing manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Running command /tmp/venv/bin/python3 /tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpenyjg9hk
  running dist_info
  creating /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info
  writing /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching '**/__pycache__'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-2tke0rjw/domdf_python_tools.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: domdf-python-tools
  Building wheel for domdf-python-tools (pyproject.toml): started
  Running command /tmp/venv/bin/python3 /tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp_1m0sf3w
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/domdf_python_tools
  copying domdf_python_tools/bases.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/paths.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/secrets.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/dates.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/delegators.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/import_tools.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/versions.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/iterative.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/utils.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/__init__.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/terminal.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/typing.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/words.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/doctools.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/stringlist.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/_is_match.py -> build/lib/domdf_python_tools
  copying domdf_python_tools/pretty_print.py -> build/lib/domdf_python_tools
  creating build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/sizes.py -> build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/utils.py -> build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/classes.py -> build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/__init__.py -> build/lib/domdf_python_tools/pagesizes
  copying domdf_python_tools/pagesizes/units.py -> build/lib/domdf_python_tools/pagesizes
  creating build/lib/domdf_python_tools/compat
  copying domdf_python_tools/compat/importlib_metadata.py -> build/lib/domdf_python_tools/compat
  copying domdf_python_tools/compat/__init__.py -> build/lib/domdf_python_tools/compat
  copying domdf_python_tools/compat/importlib_resources.py -> build/lib/domdf_python_tools/compat
  running egg_info
  creating domdf_python_tools.egg-info
  writing domdf_python_tools.egg-info/PKG-INFO
  writing dependency_links to domdf_python_tools.egg-info/dependency_links.txt
  writing requirements to domdf_python_tools.egg-info/requires.txt
  writing top-level names to domdf_python_tools.egg-info/top_level.txt
  writing manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching '**/__pycache__'
  adding license file 'LICENSE'
  writing manifest file 'domdf_python_tools.egg-info/SOURCES.txt'
  copying domdf_python_tools/google-10000-english-no-swears.txt -> build/lib/domdf_python_tools
  copying domdf_python_tools/py.typed -> build/lib/domdf_python_tools
  copying domdf_python_tools/compat/importlib_metadata.pyi -> build/lib/domdf_python_tools/compat
  copying domdf_python_tools/compat/importlib_resources.pyi -> build/lib/domdf_python_tools/compat
  Traceback (most recent call last):
    File "/tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 221, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 207, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 20, in <module>
      setup(
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 301, in run
      install = self.reinitialize_command('install',
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 214, in reinitialize_command
      cmd = _Command.reinitialize_command(self, command, reinit_subcommands)
    File "/usr/lib/python3.8/distutils/cmd.py", line 305, in reinitialize_command
      return self.distribution.reinitialize_command(command,
    File "/usr/lib/python3.8/distutils/dist.py", line 938, in reinitialize_command
      command = self.get_command_obj(command_name)
    File "/usr/lib/python3.8/distutils/dist.py", line 858, in get_command_obj
      cmd_obj = self.command_obj[command] = klass(self)
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 178, in __init__
      _Command.__init__(self, dist)
    File "/usr/lib/python3.8/distutils/cmd.py", line 62, in __init__
      self.initialize_options()
    File "/tmp/pip-build-env-myy0rzw0/overlay/lib/python3.8/site-packages/setuptools/command/install.py", line 34, in initialize_options
      warnings.warn(
  setuptools._deprecation_warning.SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  Building wheel for domdf-python-tools (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for domdf-python-tools
Failed to build domdf-python-tools
ERROR: Failed to build one or more wheels

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

Agreed this should be suppressed when building wheels.

[mara004]

I am still experiencing this warning with setuptools 60.8.2 when using bdist_wheel.
(The project in question is pypdfium2 - just clone the repository and call pip3 install . -v to reproduce)

[abravalheri]

@mara004, this seems to be related to the way pip is working internally.

I recon the project needs to adhere to PEP 517 and PEP 518, i.e. a pyproject.toml file needs to be present in the root of the project with contents similar to the following:

# pyproject.toml
[build-system]
requires = ["setuptools>=46.1.0", "wheel"]
build-backend = "setuptools.build_meta:__legacy__"

pypdfium2 in particular seems to depend on the __legacy__ behaviour, and also explicitly require the wheel dependency to run.

(I also recommend updating pip version).

[mara004]

@abravalheri First of all, thanks for the response.

(I also recommend updating pip version).

I am already running the latest pip version 22.0.3

i.e. a pyproject.toml file needs to be present in the root of the project

Sorry, adding the above pyproject.toml breaks the build with obscure errors. Anyway, I don't quite understand the background - why should setup.cfg not be sufficient?

pypdfium2 in particular seems to depend on the __legacy__ behaviour

What would I have to change so that the project does not rely on legacy behaviour?

[abravalheri]

Hi @mara004,

Sorry, adding the above pyproject.toml breaks the build with obscure errors. Anyway, I don't quite understand the background - why should setup.cfg not be sufficient?

The thing pyproject.toml would do is to change the way pip is building the project.
It seems to me that pip tries to run python setup.py when it does not find pyproject.toml.
When pyproject.toml exists, it will run the functions in setuptools.build_meta instead.

Recently, setuptools reserved to itself the capacity of executing setup.py files directly ("demoted" it to an internal API), for the reasons explained in this article. So the warning is a heads up to developers saying that things might change in the future.

What would I have to change so that the project does not rely on legacy behaviour?

The main difference between the legacy and the standard backend is that __legacy__ automatically adds the directory that contains the setup.py script to sys.path. The standard backend does not do that.

[mara004]

Thanks for the help, this is really appreciated.
I'll have to look into pyproject.toml and the related PEPs more thoroughly to see if I can change my setup infrastructure. This will be quite complex since pypdfium2 has multiple setup files for different platforms, and it's totally built around some peculiarities of setup.py. (The project uses external binaries, which is a thing that no other Python library I know of does.)

[abravalheri]

@mara004, there are 3 things to consider in the case of pypdfium.

• The setup.py script is importing from the platform_setup folder as if it was a package. However, it does not contain a __init__.py file.

• The sdist does not seem to contain the platform_setup folder.

  • The installers are very likely to consider the sdist as an intermediate step to obtain the wheel. So they will create the sdist first and then create the wheel from it, not from the original code.

  • Since the platform_setup is not part of the final package, setuptools will not include it by default in the sdist. You need to write your own MANIFEST.in or simply use a plugin such as setuptools-scm, for example:

    # pyproject.toml
    [build-system]
    requires = ["setuptools>=46.1.0", "setuptools-scm", "wheel"]
    build-backend = "setuptools.build_meta:__legacy__"

• If you don't want to use __legacy__, you either have to keep all your code inside setup.py or find a way to signal to Python's import system where to find the auxiliary packages you are importing in setup.py.

[mara004]

The sdist does not seem to contain the platform_setup folder.

I guess you mean only if switching to the pyproject.toml build system? When I run python3 platform_setup/setup_sdist.py sdist, the platform_setup folder (and everything else in the repository) is included as expected, without the need for additional configuration.

The setup.py script is importing from the platform_setup folder as if it was a package. However, it does not contain a __init__.py file.

It currently seems to be working without an __init__.py file, but adding one should be straightforward if needed.

[mara004]

I've now had some more time to look into this, and unfortunately it seems like it is simply not possible to do what I need with a PEP 517 compliant build system.

@abravalheri Re-reading your previous comment, I think you might not yet have understood what my setup code really does, so I'll try to explain the project's requirements and the current approach:

• The wheels I package for PyPI need to include pre-built binaries. It must be possible to build all wheels on a single platform to avoid being dependent on CIs or virtualisation.
• This is implemented with platform-specific setup files that each call a shared base setup function with the corresponding platform name as argument. Wheels for each platform can then be built using setup_{platform-name}.py bdist_wheel.
• There is another setup file to create the source distribution (sdist). This will simply package the content of the repository into a tarball, without moving in binaries/bindings.
• The main setup.py file checks whether pre-built binaries are available for the host platform. If so, it downloads them, creates the bindings, and calls the corresponding platform setup function. Otherwise, a source build will be attempted.

This has the following advantages:

• When installing from PyPI, users of platforms with pre-built binaries will get finished wheels, otherwise they get the sdist and PDFium will be built from source.
• setup.py also detects the fastest installation method, so it is possible to quickly install from git main and test changes.
• It is very simple to make a release.

The approach of PEP 517 and the deprecation of the setup.py commands completely break this strategy:

• There can only be a single setup file that is set as an entry point in pyproject.toml. One cannot call different setup files on an individual basis anymore.
• Consequently, cross-packaging is impossible to do, though it would technically be the easiest solution for this case.

A potential fix to allow cross-packaging again would be to add a --setup-file argument or similar to the python3 -m build command, so that one can point the wheel builder at custom setup code to use.

---

Since the platform_setup is not part of the final package, setuptools will not include it by default in the sdist. You need to write your own MANIFEST.in or simply use a plugin such as setuptools-scm

From what I have seen, sdist should automatically take all files in the repository. When I run python3 platform_setup/setup_sdist.py sdist, the platform_setup folder and everything else is included as expected, without the need for additional configuration. Furthermore, MANIFEST.in is only relevant for packaging wheels, not source distributions.
Unless PEP 517 should have changed this, your statement seems to be incorrect, or have I overlooked anything?

[abravalheri]

Hi @mara004, please find my comments bellow:

From what I have seen, sdist should automatically take all files in the repository.

When I run:

rm -rf dist build src/*.egg-info
python setup.py sdist
tar tf dist/pypdfium2-0.12.0.tar.gz

I cannot find the platform_setup folder inside the distribution.
When moving towards PEP 517, python setup.py sdist is what effectively runs, so I believe it will produce an sdist without the required files.

Furthermore, MANIFEST.in is only relevant for packaging wheels

This statement is incorrect, please check https://setuptools.pypa.io/en/latest/deprecated/distutils/sourcedist.html#specifying-the-files-to-distribute (which also states which files are included by default).

---

I understand that your current workflow has been optimised for years to work with the previous behaviour of setuptools. However the packaging ecosystem for Python has been changing for a while now. There are new standards in place and the setuptools wants to rely on these standards to allow us to remove complexity and be able to focus in other aspects of the implementation. This is one of the reasons why running setup.py has been deprecated.

If you want to keep the way things are current implemented, that is fine too! Both setuptools and pip have a lot of stuff implemented to support legacy packages. But you will have to be aware and workaround the limitations that are implied by that approach.

One of them is seeing the warning. You can still get around that by defining a filter via PYTHONWARNINGS.

On the other hand, if you want to give it a try to PEP 517, it might be worthy to start with the __legacy__ first to see how it goes. I also recommend looking how other projects (such as https://github.com/pybind/pybind11) deal with pre-built binaries and multiple setup_* files while still being able to rely on PEP 517.

You can always wrap an existing PEP 517 backends to tweak more deeply how they work for your package.

[mara004]

When I run:

rm -rf dist build src/*.egg-info
python setup.py sdist
tar tf dist/pypdfium2-0.12.0.tar.gz

I cannot find the platform_setup folder inside the distribution.

That is confusing. I cannot reproduce this behaviour, at least not with the latest state of the repository. When running your sequence of commands, I get a tarball including all directories and files that are not excluded in .gitignore (sample: pypdfium2-0.13.1.tar.gz).
However, as mentioned previously, setup.py is not intended for sdist, as it will copy platform-specific binary and bindings files into the src/pypdfium2/ directory. You need to use platform_setup/setup_source.py sdist instead.

I'll investigate the other mentioned points and reply in following comments.

[mara004]

If you want to keep the way things are current implemented, that is fine too!

It is not, because it means the code will eventually stop working, should the setuptools comman-line really be removed some day.

One of them is seeing the warning. You can still get around that by defining a filter via PYTHONWARNINGS.

The warning is there for a reason. Filtering it won't help anyone.