Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate and remove distutils #85454

Closed
jaraco opened this issue Jul 12, 2020 · 60 comments
Closed

Deprecate and remove distutils #85454

jaraco opened this issue Jul 12, 2020 · 60 comments
Labels
3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir

Comments

@jaraco
Copy link
Member

jaraco commented Jul 12, 2020

BPO 41282
Nosy @brettcannon, @doko42, @pfmoore, @jaraco, @ncoghlan, @tiran, @ned-deily, @merwok, @encukou, @methane, @ambv, @zooba, @dstufft, @pganssle, @hroncok, @frenzymadness, @pablogsal, @hugovk, @miss-islington, @tirkarthi, @FFY00
PRs
  • bpo-41282: (PEP 632) Deprecate distutils.sysconfig (partial implementation of the PEP) #23142
  • bpo-41282: Add deprecation warning and docs for distutils (PEP 632) #24355
  • bpo-41282: (PEP 632) Load install schemes from sysconfig #24549
  • bpo-41282: distutils: Fix stacklevel for DeprecationWarning #24657
  • bpo-41282: setup.py ignores distutils DeprecationWarning #25405
  • bpo-41282: Fix distutils.utils.byte_compile() DeprecationWarning #25406
  • bpo-41282: Consistent message and filter warning in setup.py (GH-25571) #25571
  • bpo-43976: add vendor config #25718
  • bpo-41282: Fix broken make install #26327
  • bpo-41282: Fix broken make install #26329
  • [3.10] bpo-41282: Fix broken make install (GH-26329) #26336
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2020-07-12.08:30:25.278>
    labels = ['library', '3.10', '3.11']
    title = 'Deprecate and remove distutils'
    updated_at = <Date 2021-10-13.14:03:04.437>
    user = 'https://github.com/jaraco'

    bugs.python.org fields:

    activity = <Date 2021-10-13.14:03:04.437>
    actor = 'hroncok'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Distutils']
    creation = <Date 2020-07-12.08:30:25.278>
    creator = 'jaraco'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 41282
    keywords = ['patch']
    message_count = 44.0
    messages = ['373548', '373549', '373586', '373612', '373613', '373614', '373615', '373622', '373629', '373630', '373631', '373633', '373649', '373651', '373652', '373653', '376390', '385812', '385946', '386453', '387897', '388224', '391082', '391087', '391175', '391683', '391768', '392322', '392327', '392329', '392331', '393017', '393018', '393020', '393021', '393023', '393026', '394241', '394242', '394248', '394275', '394277', '394302', '403835']
    nosy_count = 23.0
    nosy_names = ['brett.cannon', 'doko', 'paul.moore', 'jaraco', 'ncoghlan', 'christian.heimes', 'ned.deily', 'eric.araujo', 'Arfrever', 'ionelmc', 'petr.viktorin', 'methane', 'lukasz.langa', 'steve.dower', 'dstufft', 'p-ganssle', 'hroncok', 'frenzy', 'pablogsal', 'hugovk', 'miss-islington', 'xtreak', 'FFY00']
    pr_nums = ['23142', '24355', '24549', '24657', '25405', '25406', '25571', '25718', '26327', '26329', '26336']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue41282'
    versions = ['Python 3.10', 'Python 3.11']

    @jaraco
    Copy link
    Member Author

    jaraco commented Jul 12, 2020

    Setuptools has adopted distutils as outlined in pypa/packaging-problems#127. Although there are some straggling issues, the current release of Setuptools fully obviates distutils if a certain environment variable is set. Soon, that behavior will be default.

    Additionally, the distutils codebase remains maintained at pypa/distutils in a form suitable for releasing as a third-party package, should the need arise (i.e. pip install distutils).

    The plan now is to freeze, deprecate, and in Python N + 0.1, remove distutils.

    Already, Setuptools is identifying emergent bugs and other defects in distutils and providing fixes for them (bpo-41207, pypa/setuptools#2212). Keeping these changes in sync across three repos and different supported versions is tedious, so I'd like to move forward with the deprecation process as soon as possible.

    @jaraco jaraco added 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir labels Jul 12, 2020
    @jaraco
    Copy link
    Member Author

    jaraco commented Jul 12, 2020

    Łukasz, would it be possible to add the deprecation warning and documented deprecation to Python 3.9?

    @ned-deily
    Copy link
    Member

    So what is the plan to continue to support building cpython itself which depends on Distutils? Currently the build bootstraps itself without the aid of an existing Python interpreter instance. There would also be major impacts across the whole cpython development process. For example, there are many open Distutils issues in the bugs.python.org bug tracker. We need a plan on how those are to be handled (and that should take into account the expected transition from b.p.o to GitHub issues). People will continue to submit issues agains Distutils there so triage team members and core developers need to know how to handle such issues. What if an issue applies also or only to a previous release branch (i.e. where Distutils is still in the repo)? What about Distutils documentation in the Python docset? THose are just some off the top of my head.

    I don't think any of these issues are necessarily blockers but they need to be planned for and reviewed. I think a PEP is definitely in order for a change of this magnitude.

    @ambv
    Copy link
    Contributor

    ambv commented Jul 13, 2020

    It's too late to add a new deprecation in the Python 3.9 cycle. Next week is the *last* beta release. Most beta testing already took place.

    @pganssle
    Copy link
    Member

    So what is the plan to continue to support building cpython itself which depends on Distutils? Currently the build bootstraps itself without the aid of an existing Python interpreter instance. There would also be major impacts across the whole cpython development process.

    My understanding was that the plan was to move the standard library distutils into a private module somewhere in the standard library and presumably to slim it down to only the bare minimum required for what is necessary to build Python itself. We're really only concerned with the use of distutils to build packages.

    For example, there are many open Distutils issues in the bugs.python.org bug tracker. We need a plan on how those are to be handled (and that should take into account the expected transition from b.p.o to GitHub issues). People will continue to submit issues agains Distutils there so triage team members and core developers need to know how to handle such issues. What if an issue applies also or only to a previous release branch (i.e. where Distutils is still in the repo)?

    As far as I can tell we've already been telling people that issues in distutils should be fixed in setuptools instead for a few years. I don't think anything needs to be done about the currently open distutils tickets before we *deprecate* distutils, though during the deprecation period we'll probably want to decide whether we want to migrate them, do a mass closure or just leave them to be ad hoc closed as people stumble upon them later. Mass closure may be complicated because tickets affecting CPython itself will still need to be addressed.

    What about Distutils documentation in the Python docset? THose are just some off the top of my head.

    The distutils documentation is already basically just a warning page that says "stop using distutils": https://docs.python.org/3/library/distutils.html#module-distutils

    Before these reference materials are removed from the docs we'll need to make sure that all the stuff that's still supported is documented on the setuptools side.

    I don't think any of these issues are necessarily blockers but they need to be planned for and reviewed. I think a PEP is definitely in order for a change of this magnitude.

    A PEP may be a good idea, but I do think the change doesn't have a particularly large magnitude. Anyone using setuptools or pip has already been getting setuptools' monkey-patched version of distutils for ages now, and soon they will be getting setuptools' vendored version. The documentation already indicates that distutils is at least soft-deprecated in favor of setuptools and we've already been directing issues and PRs to setuptools instead of distutils. This last piece is really formalizing something we've been incrementally working towards for a long time now. Doesn't mean we shouldn't do it carefully and with a lot of notice, but it's also not a sudden and massive shift.

    @zooba
    Copy link
    Member

    zooba commented Jul 13, 2020

    Deprecating in 3.10 is fine - everyone who needs to know about it releases whenever they like anyway, so we just need to make _some_ announcement.

    I'd propose either moving it to Tools/distutils, or renaming it to _distutils. The point is that we're saying it's only fit for use for the core build now, and nobody else should ever import it (or complain about it ;) ).

    @dstufft
    Copy link
    Member

    dstufft commented Jul 13, 2020

    Maybe it would make sense to remove distutils from the name completely, _buildutils or something. Dunno, seems like it might be reasonable just to further separate it from the concept of "distutils" the public library.

    @brettcannon
    Copy link
    Member

    FYI PEP-387 (which I expect will be accepted once I catch up from vacation) specified deprecations are to be public for two releases before removal or approval from the SC for a shorter cycle.

    So if distutils is deprecated in 3.10 then it can be removed in 3.12 or you can ask the SC for an exemption to do it in 3.11.

    @doko42
    Copy link
    Member

    doko42 commented Jul 14, 2020

    It's too late to add a new deprecation in the Python 3.9 cycle

    Please can we add a note in 3.9, that it will be deprecated in 3.10?

    @doko42
    Copy link
    Member

    doko42 commented Jul 14, 2020

    Renaming distutils to _buildutils only delays the problem to remove it. But yes, it explicitly makes it explicit that code needs to be changed.

    I would like to see that neither distutils or _buildutils is installed by default, and only is available internally for building the extensions of CPython.

    The "old" build system to build builtins instead of extensions is still functional, so it should be ok to build the extensions also with the old build system.

    That would require moving all the config stuff in setup.py to autoconf tests, which is perfectly doable. The MacOS and Windows builds would need some attention too, but afaicr when asking Ned Deily and Steve Dower at the language summits, they didn't have a concern about this approach.

    @tiran
    Copy link
    Member

    tiran commented Jul 14, 2020

    +1

    I would like to propose three changes:

    1. rename distutils, either _distutils or _buildutils sounds good to me
    2. make distutils a build-only dependency and no longer install it with make install and other install targets
    3. start to build extensions from Makefile or Modules/Setup

    For (3) we have to move some checks into autoconf and maybe extend Modules/Setup to support conditional compilation.

    @doko42
    Copy link
    Member

    doko42 commented Jul 14, 2020

    A PEP may be a good idea, but I do think the change doesn't have a
    particularly large magnitude. Anyone using setuptools or pip has
    already been getting setuptools' monkey-patched version of distutils
    for ages now, and soon they will be getting setuptools' vendored
    version. The documentation already indicates that distutils is at
    least soft-deprecated in favor of setuptools and we've already been
    directing issues and PRs to setuptools instead of distutils.

    I don't think it's a good idea to replace bad habits from distutils with bad habits in setuptools._distutils. And this is exactly what you get with pointing directly to setuptools.

    While splitting out distutils to a separate package in a Linux distro, I found some creative usages at runtime of a package (see my lightning talk at the language summit 2018, and [1]). From my point of view, CPython should provide documentation how to forward-port these issues without using setuptools._distutils.

    Currently setuptools only has one component (pkg_resources, [2]) which is used at runtime. I dislike it if more than that is used at runtime of a package.

    [1] https://mail.python.org/archives/list/distutils-sig@python.org/thread/74WZ7D3ARF7B3N6MAV2JBV3DW6TRHFIV/
    [2] pypa/setuptools#863

    @zooba
    Copy link
    Member

    zooba commented Jul 14, 2020

    The Windows build doesn't depend on distutils at all. We've had dedicated build scripts for each module since before I started contributing.

    @tiran
    Copy link
    Member

    tiran commented Jul 14, 2020

    The Windows build system didn't use setu.py even before I upgrade the VS build system to VS 2010.

    @pganssle
    Copy link
    Member

    I don't think it's a good idea to replace bad habits from distutils with bad habits in setuptools._distutils. And this is exactly what you get with pointing directly to setuptools.

    These are two different questions. We're not asking people to migrate to setuptools._distutils (a private module which may not continue to exist in that location), setuptools is adopting distutils, such that distutils is a project provided by pip install distutils (mind you, this is happening independent of what the standard library does — the only question is whether import distutils continues to work if you don't have setuptools installed).

    While splitting out distutils to a separate package in a Linux distro, I found some creative usages at runtime of a package (see my lightning talk at the language summit 2018, and [1]). From my point of view, CPython should provide documentation how to forward-port these issues without using setuptools._distutils.

    At this point, the extent of CPython's documentation on this should probably be, "We are removing distutils and moving it into the setuptools namespace. In future versions, you will need to install setuptools to import the distutils package." setuptools should almost certainly deprecate distutils and probably remove large swathes of it in the process, but that's probably on a case-by-case basis, and it's a separate issue from what needs to happen in CPython.

    Currently setuptools only has one component (pkg_resources, [2]) which is used at runtime. I dislike it if more than that is used at runtime of a package.

    I don't think anyone is planning to recommend the use of any setuptools-provided packages at runtime, including pkg_resources. This move is actually a good one from that point of view, because it will require that projects using distutils declare a runtime dependency on setuptools, which will, hopefully, raise some eyebrows. Better than the current situation, where these dependencies are totally undeclared (though probably worse than if setuptools, pkg_resources and distutils were all separate PyPI packages).

    @pganssle
    Copy link
    Member

    Oops, just realized my previous post said pip install distutils. I meant to say that pip install setuptools will provide the distutils module (right now you do import setuptools; import distutils and you get the setuptools-provided version; we're working on a version where import distutils comes from setuptools regardless of the import order).

    @terryjreedy terryjreedy removed 3.9 only security fixes labels Jul 18, 2020
    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented Sep 4, 2020

    I noticed that a new PEP draft [1] about deprecating distutils is uploaded. The current version [2] proposes to deprecate distutils in 3.10 and 3.11 and remove distutils in 3.12.

    [1] https://www.python.org/dev/peps/pep-0632/
    [2] python/peps@5d5c685

    @zooba
    Copy link
    Member

    zooba commented Jan 27, 2021

    That PR is just to add the import warning and update docs. I want to make sure that's in asap so we don't miss the release.

    Is there anywhere else in the docs that needs a note? Distutils has been marked as deprecated for years already, so it's really just emphasising that and adding the 3.12 removal date. I think whatsnew, library and the old doc sections is enough, yeah?

    @zooba
    Copy link
    Member

    zooba commented Jan 29, 2021

    New changeset 62949f6 by Steve Dower in branch 'master':
    bpo-41282: Add deprecation warning and docs for distutils (PEP-632) (GH-24355)
    62949f6

    @zooba
    Copy link
    Member

    zooba commented Feb 3, 2021

    Everyone probably noticed, but I closed all the other distutils-tagged issues (as stated in the PEP), so now this is the only one left.

    Anything new that is opened that relates to distutils either needs to be a release blocker (very unlikely), or closed and directed to setuptools instead.

    @methane
    Copy link
    Member

    methane commented Mar 2, 2021

    New changeset 5bfa945 by Inada Naoki in branch 'master':
    bpo-41282: distutils: Fix stacklevel for DeprecationWarning (GH-24657)
    5bfa945

    @tirkarthi
    Copy link
    Member

    I have created below issues where deprecation warning is emitted due to distutils usage in tests. Probably there are other places that need an update to setuptools like setup.py used by make that emits deprecation warning during building cpython.

    https://bugs.python.org/issue43426
    https://bugs.python.org/issue43425

    rg '(from|import) distutils' | rg -v 'Lib/distutils|rst'
    Modules/_decimal/tests/formathelper.py:from distutils.spawn import find_executable
    Doc/includes/setup.py:from distutils.core import setup, Extension
    Doc/includes/test.py:from distutils.util import get_platform
    setup.py:from distutils import log
    setup.py:from distutils.command.build_ext import build_ext
    setup.py:from distutils.command.build_scripts import build_scripts
    setup.py:from distutils.command.install import install
    setup.py:from distutils.command.install_lib import install_lib
    setup.py:from distutils.core import Extension, setup
    setup.py:from distutils.errors import CCompilerError, DistutilsError
    setup.py:from distutils.spawn import find_executable
    Lib/_osx_support.py: from distutils import log
    Lib/_osx_support.py: Currently called from distutils.sysconfig
    Lib/test/support/init.py: from distutils import ccompiler, sysconfig, spawn, errors
    Lib/test/test_distutils.py:import distutils.tests
    Lib/test/test_importlib/test_windows.py:from distutils.util import get_platform
    Lib/test/test_peg_generator/test_c_parser.py:from distutils.tests.support import TempdirManager
    Lib/test/test_sundry.py: import distutils.bcppcompiler
    Lib/test/test_sundry.py: import distutils.ccompiler
    Lib/test/test_sundry.py: import distutils.cygwinccompiler
    Lib/test/test_sundry.py: import distutils.filelist
    Lib/test/test_sundry.py: import distutils.text_file
    Lib/test/test_sundry.py: import distutils.unixccompiler
    Lib/test/test_sundry.py: import distutils.command.bdist_dumb
    Lib/test/test_sundry.py: import distutils.command.bdist_msi
    Lib/test/test_sundry.py: import distutils.command.bdist
    Lib/test/test_sundry.py: import distutils.command.bdist_rpm
    Lib/test/test_sundry.py: import distutils.command.build_clib
    Lib/test/test_sundry.py: import distutils.command.build_ext
    Lib/test/test_sundry.py: import distutils.command.build
    Lib/test/test_sundry.py: import distutils.command.clean
    Lib/test/test_sundry.py: import distutils.command.config
    Lib/test/test_sundry.py: import distutils.command.install_data
    Lib/test/test_sundry.py: import distutils.command.install_egg_info
    Lib/test/test_sundry.py: import distutils.command.install_headers
    Lib/test/test_sundry.py: import distutils.command.install_lib
    Lib/test/test_sundry.py: import distutils.command.register
    Lib/test/test_sundry.py: import distutils.command.sdist
    Lib/test/test_sundry.py: import distutils.command.upload
    Tools/peg_generator/pegen/build.py: import distutils.log
    Tools/peg_generator/pegen/build.py: from distutils.core import Distribution, Extension
    Tools/peg_generator/pegen/build.py: from distutils.command.clean import clean # type: ignore
    Tools/peg_generator/pegen/build.py: from distutils.command.build_ext import build_ext # type: ignore
    Tools/peg_generator/pegen/build.py: from distutils.tests.support import fixup_build_ext # type: ignore
    Tools/c-analyzer/c_parser/preprocessor/common.py:import distutils.ccompiler
    Tools/c-analyzer/c_parser/preprocessor/init.py:import distutils.ccompiler
    Tools/test2to3/setup.py:from distutils.core import setup
    Tools/test2to3/setup.py: from distutils.command.build_py import build_py_2to3 as build_py
    Tools/test2to3/setup.py: from distutils.command.build_py import build_py
    Tools/test2to3/setup.py: from distutils.command.build_scripts import build_scripts_2to3 as build_scripts
    Tools/test2to3/setup.py: from distutils.command.build_scripts import build_scripts
    Tools/test2to3/test/runtests.py: from distutils.util import copydir_run_2to3
    Misc/HISTORY:- Issue bpo-5394: removed > 2.3 syntax from distutils.msvc9compiler.

    @arhadthedev
    Copy link
    Member

    @vstinner Since setup.py was totally removed in gh-94474, can gh-92585 be repuprosed to close this issue?

    @merwok
    Copy link
    Member

    merwok commented Jul 14, 2022

    Branches and PRs are cheap, just make a new PR. The closed one is a record of one proposed change that was not taken.

    @tiran
    Copy link
    Member

    tiran commented Jul 14, 2022

    We like to remove distutils completely. However we are not there yet. There are still some test cases and tools that have to be ported to virtual environments with setuptools or use different approaches. From the top of my head test_cppext, peg generator tests, and c-analyzer have to be changed first.

    @tiran
    Copy link
    Member

    tiran commented Jul 14, 2022

    @arhadthedev If you like to help, then you could start with updating our documentation. You could update the "distributing" section to refer to setuptools and PyPA packaging instead of distutils. There are also references to :mod:`distutils` and various functions in the distutils namespace in the whatsnew. They have to be replaced by ``distutils``, too.

    @encukou
    Copy link
    Member

    encukou commented Jul 15, 2022

    FWIW, test_cppext uses venv + setuptools already, but it's not exactly a shining example of how to do this well. (Helping to improve it would be appreciated in #92906 & #94751.)
    By chance, are there setuptools→venv migration success stories in the testsuite to use as examples of what to do?

    @arhadthedev
    Copy link
    Member

    There are also references to :mod:`distutils` and various functions in the distutils namespace in the whatsnew. They have to be replaced by ``distutils``, too.

    @tiran Thank you for the advice, I've created gh-95192 to address the second part.

    tiran added a commit to tiran/cpython that referenced this issue Jul 25, 2022
    Most places now refer to setuptools or link to setuptools documentation.
    Some examples like zipapp need to be updated later.
    ambv pushed a commit that referenced this issue Jul 25, 2022
    Most places now refer to setuptools or link to setuptools documentation.
    Some examples like zipapp need to be updated later.
    tiran added a commit to tiran/cpython that referenced this issue Jul 25, 2022
    Extract the wheel from ensurepip's bundle and inject setuptools and
    distutils into sys.path.
    @vstinner
    Copy link
    Member

    vstinner commented Aug 3, 2022

    FWIW, test_cppext uses venv + setuptools already, but it's not exactly a shining example of how to do this well. (Helping to improve it would be appreciated in #92906 & #94751.)

    Using setuptools on a newly built Python is not the most common case (run Python from its source directory, not on an "installed" Python). It causes multiple issues in sysconfig and setuptools. Hopefully, it seems like these issues are being fixed one by one, but it still seems to be bumpy work-in-progress (sysconfig+setuptools issues come back time to time).

    As you wrote, it doesn't work on Windows yet.

    @tiran
    Copy link
    Member

    tiran commented Aug 4, 2022

    #95254 is a different approach. It unpacks setuptools wheel and injects it into sys.path.

    @arhadthedev
    Copy link
    Member

    We like to remove distutils completely. [...] There are still some test cases and tools that have to be ported to virtual environments with setuptools or use different approaches. From the top of my head test_cppext, peg generator tests, and c-analyzer have to be changed first.

    After gh-103316 vendored setuptools into Lib/test, should we consider this plan outdated and close the issue as done?

    @arhadthedev
    Copy link
    Member

    By the way, gh-92584 and gh-95254 are still open so I'm not sure.

    @vstinner
    Copy link
    Member

    The distutils package was removed in Python 3.12 by issue #92584. Remaining issues related to this removal are tracked by issue #92584. I close this one.

    @abartlet
    Copy link

    abartlet commented Sep 5, 2024

    Debian advises at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1080851 that this breaks builds of Samba, as we use distutils.sysconfig.get_python_lib. Can distutils.sysconfig be made an alias for sysconfig so that software like ours that just needs this detail doesn't get unnecessarily broken?

    It is very helpful for Samba to be able to build historical versions on modern operating systems, and changes like this continue to make our development process more painful.

    Thanks for your understanding,

    Andrew Bartlett

    @zooba
    Copy link
    Member

    zooba commented Sep 5, 2024

    sysconfig has existed for 14 years already (including in Python 2) - is that not long enough to switch to it?

    Installing setuptools should bring in the shim that you need. If we were to keep shims in the standard library, it would be impossible for them to properly migrate the old distutils interface.

    @jaraco
    Copy link
    Member Author

    jaraco commented Sep 5, 2024

    Related - CPython has given this namespace over to Setuptools, so it no longer has the ability to present that name without breaking migration plans for distutils in Setuptools. Currently, Setuptools does present distutils, including distutils.sysconfig, but it's deprecated from there as well, and we're working to provide the necessary interfaces to migrate users away from that (including distutils.sysconfig -> sysconfig), so the best bet would be not to rely on Setuptools for this behavior if you can help it.

    @doko42
    Copy link
    Member

    doko42 commented Sep 5, 2024

    please can you check, if things like python3-config can solve your issues? this is also available for some time

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir
    Projects
    None yet
    Development

    No branches or pull requests