[BUG] SETUPTOOLS_USE_DISTUTILS=local makes any distutils import very slow
#3585
Labels
bug
Needs Implementation
Issues that are ready to be implemented.
Needs Investigation
Issues which are likely in scope but need investigation to figure out the cause
Needs Triage
Issues that need to be evaluated for severity and status.
Comments
asottile
added
bug
Needs Triage
|
FWIW, on Windows x64 with Python 3.10.2 (conda) and Setuptools 65.5.0, I'm seeing a 25x slowdown (40 ms to 1 s) with With timeit: $ SETUPTOOLS_USE_DISTUTILS=local python -m timeit -n 1 -r 1 'import distutils.version'
1 loop, best of 1: 806 msec per loop
$ SETUPTOOLS_USE_DISTUTILS=stdlib python -m timeit -n 1 -r 1 'import distutils.version'
1 loop, best of 1: 10 msec per loopWith importtime: $ SETUPTOOLS_USE_DISTUTILS=local python -X importtime -c 'import distutils.version'
import time: self [us] | cumulative | imported package
[...]
import time: 9827 | 1014210 | distutils
import time: 2366 | 1016576 | distutils.version
$ SETUPTOOLS_USE_DISTUTILS=stdlib python -X importtime -c 'import distutils.version'
import time: self [us] | cumulative | imported package
[...]
import time: 2598 | 39466 | distutils
import time: 2423 | 41889 | distutils.versionFor reference, I've also provided the full Full `-X importtime` output with `local`$ SETUPTOOLS_USE_DISTUTILS=local python -X importtime -c 'import distutils.version'
import time: self [us] | cumulative | imported package
import time: 350 | 350 | _io
import time: 121 | 121 | marshal
import time: 410 | 410 | nt
import time: 206 | 206 | winreg
import time: 2280 | 3365 | _frozen_importlib_external
import time: 739 | 739 | time
import time: 626 | 1364 | zipimport
import time: 286 | 286 | _codecs
import time: 1808 | 2093 | codecs
import time: 1617 | 1617 | encodings.aliases
import time: 3370 | 7079 | encodings
import time: 1067 | 1067 | encodings.utf_8
import time: 1082 | 1082 | encodings.cp1252
import time: 135 | 135 | _signal
import time: 126 | 126 | _abc
import time: 1432 | 1558 | abc
import time: 1596 | 3153 | io
import time: 266 | 266 | _stat
import time: 1546 | 1812 | stat
import time: 3741 | 3741 | _collections_abc
import time: 1236 | 1236 | genericpath
import time: 2137 | 3372 | ntpath
import time: 3008 | 11931 | os
import time: 1359 | 1359 | _sitebuiltins
import time: 4810 | 4810 | _distutils_hack
import time: 3728 | 3728 | sitecustomize
import time: 1546 | 1546 | usercustomize
import time: 7218 | 30590 | site
import time: 1604 | 1604 | warnings
import time: 2255 | 3859 | importlib
import time: 1952 | 1952 | importlib.machinery
import time: 1032 | 1032 | importlib._abc
import time: 1163 | 1163 | itertools
import time: 1189 | 1189 | keyword
import time: 170 | 170 | _operator
import time: 1629 | 1798 | operator
import time: 1153 | 1153 | reprlib
import time: 184 | 184 | _collections
import time: 3597 | 9082 | collections
import time: 1153 | 1153 | collections.abc
import time: 1460 | 1460 | types
import time: 221 | 221 | _functools
import time: 2910 | 4590 | functools
import time: 2686 | 7276 | contextlib
import time: 2484 | 2484 | enum
import time: 485 | 485 | _sre
import time: 1571 | 1571 | sre_constants
import time: 2293 | 3864 | sre_parse
import time: 2036 | 6385 | sre_compile
import time: 274 | 274 | _locale
import time: 1472 | 1472 | copyreg
import time: 3201 | 13813 | re
import time: 6226 | 37548 | typing
import time: 3279 | 43809 | importlib.abc
import time: 1973 | 1973 | importlib.util
import time: 1378 | 1378 | token
import time: 3549 | 4926 | tokenize
import time: 1139 | 1139 | distutils.debug
import time: 1395 | 1395 | distutils.errors
import time: 2096 | 2096 | posixpath
import time: 1748 | 3844 | fnmatch
import time: 243 | 243 | errno
import time: 1566 | 1566 | urllib
import time: 3291 | 4856 | urllib.parse
import time: 3635 | 12576 | pathlib
import time: 1572 | 1572 | email
import time: 129 | 129 | _string
import time: 3230 | 3359 | string
import time: 4598 | 4598 | gettext
import time: 2156 | 6754 | getopt
import time: 2555 | 12667 | distutils.fancy_getopt
import time: 2296 | 2296 | signal
import time: 1262 | 1262 | _weakrefset
import time: 2671 | 3933 | threading
import time: 1567 | 1567 | fcntl
import time: 170 | 170 | msvcrt
import time: 215 | 215 | _winapi
import time: 3162 | 11341 | subprocess
import time: 2048 | 2048 | sysconfig
import time: 840 | 840 | distutils.dep_util
import time: 1088 | 1088 | distutils.log
import time: 1133 | 2220 | distutils.spawn
import time: 2172 | 18620 | distutils.util
import time: 2925 | 48358 | distutils.dist
import time: 1098 | 1098 | distutils.dir_util
import time: 1051 | 1051 | distutils.file_util
import time: 2190 | 2190 | binascii
import time: 206 | 206 | zlib
import time: 1309 | 1309 | _compression
import time: 5625 | 5625 | _bz2
import time: 2070 | 9003 | bz2
import time: 4912 | 4912 | _lzma
import time: 2001 | 6913 | lzma
import time: 3119 | 19241 | shutil
import time: 206 | 206 | _struct
import time: 1635 | 1840 | struct
import time: 3120 | 26388 | zipfile
import time: 1638 | 1638 | pwd
import time: 1576 | 1576 | grp
import time: 1735 | 31335 | distutils.archive_util
import time: 2118 | 35600 | distutils.cmd
import time: 6275 | 6275 | configparser
import time: 1498 | 7773 | distutils.config
import time: 1291 | 1291 | distutils.extension
import time: 8353 | 108832 | _distutils_hack.override
import time: 1005 | 1005 | setuptools._deprecation_warning
import time: 2200 | 2200 | weakref
import time: 2047 | 4246 | pkgutil
import time: 6983 | 6983 | platform
import time: 171 | 171 | math
import time: 304 | 304 | _datetime
import time: 3036 | 3510 | datetime
import time: 1501 | 1501 | xml
import time: 1409 | 2910 | xml.parsers
import time: 3791 | 3791 | pyexpat
import time: 1799 | 8499 | xml.parsers.expat
import time: 3996 | 16004 | plistlib
import time: 1661 | 1661 | email.errors
import time: 1143 | 1143 | email.quoprimime
import time: 1790 | 1790 | base64
import time: 1327 | 3116 | email.base64mime
import time: 1182 | 1182 | quopri
import time: 1113 | 2295 | email.encoders
import time: 1660 | 3954 | email.charset
import time: 3053 | 11266 | email.header
import time: 132 | 132 | _bisect
import time: 1733 | 1865 | bisect
import time: 137 | 137 | _random
import time: 116 | 116 | _sha512
import time: 2670 | 4786 | random
import time: 6794 | 6794 | _socket
import time: 2282 | 2282 | select
import time: 2620 | 4902 | selectors
import time: 5870 | 17566 | socket
import time: 3063 | 3063 | locale
import time: 2489 | 5552 | calendar
import time: 1766 | 7317 | email._parseaddr
import time: 2591 | 32258 | email.utils
import time: 2226 | 45749 | email._policybase
import time: 2318 | 49727 | email.feedparser
import time: 1854 | 51580 | email.parser
import time: 1693 | 1693 | tempfile
import time: 3502 | 3502 | textwrap
import time: 2310 | 2310 | _ast
import time: 5474 | 7783 | ast
import time: 142 | 142 | _opcode
import time: 1847 | 1988 | opcode
import time: 2286 | 4273 | dis
import time: 1252 | 1252 | linecache
import time: 5497 | 18804 | inspect
import time: 2236 | 2236 | pkg_resources.extern
import time: 1112 | 1112 | pkg_resources._vendor
import time: 1488 | 2599 | pkg_resources._vendor.jaraco
import time: 1404 | 6238 | pkg_resources.extern.jaraco
import time: 978 | 978 | importlib._adapters
import time: 1251 | 2229 | importlib._common
import time: 1416 | 3645 | importlib.resources
import time: 175 | 175 | _heapq
import time: 2705 | 2880 | heapq
import time: 2561 | 2561 | _queue
import time: 1821 | 4382 | queue
import time: 1271 | 1271 | pkg_resources._vendor.more_itertools.recipes
import time: 3173 | 11704 | pkg_resources._vendor.more_itertools.more
import time: 1559 | 13262 | pkg_resources._vendor.more_itertools
import time: 868 | 14129 | pkg_resources.extern.more_itertools
import time: 1659 | 15788 | pkg_resources.extern.jaraco.functools
import time: 1753 | 1753 | pkg_resources.extern.jaraco.context
import time: 3221 | 30642 | pkg_resources.extern.jaraco.text
import time: 14852 | 14852 | win32api
import time: 3026 | 3026 | _win32sysloader
import time: 2721 | 5747 | pywintypes
import time: 23570 | 29316 | pythoncom
import time: 920 | 920 | win32com.gen_py
import time: 3920 | 49007 | win32com
import time: 1491 | 50498 | win32com.shell
import time: 1409 | 51906 | pkg_resources._vendor.appdirs
import time: 722 | 52627 | pkg_resources.extern.appdirs
import time: 1164 | 1164 | pkg_resources._vendor.packaging.__about__
import time: 1743 | 2906 | pkg_resources._vendor.packaging
import time: 495 | 3401 | pkg_resources.extern.packaging
import time: 872 | 872 | pkg_resources.extern.packaging._structures
import time: 7822 | 8694 | pkg_resources.extern.packaging.version
import time: 1787 | 1787 | traceback
import time: 172 | 172 | atexit
import time: 5963 | 7921 | logging
import time: 1897 | 1897 | pkg_resources._vendor.packaging._manylinux
import time: 1471 | 1471 | pkg_resources._vendor.packaging._musllinux
import time: 2627 | 13915 | pkg_resources.extern.packaging.tags
import time: 2013 | 15927 | pkg_resources.extern.packaging.utils
import time: 17582 | 33509 | pkg_resources.extern.packaging.specifiers
import time: 2130 | 2130 | pkg_resources._vendor.pyparsing.util
import time: 1900 | 1900 | pkg_resources._vendor.pyparsing.unicode
import time: 6088 | 7988 | pkg_resources._vendor.pyparsing.exceptions
import time: 1224 | 1224 | pkg_resources._vendor.pyparsing.actions
import time: 1551 | 1551 | org
import time: 408 | 1958 | org.python
import time: 352 | 2309 | org.python.core
import time: 1430 | 3738 | copy
import time: 2083 | 2083 | dataclasses
import time: 1857 | 3939 | pprint
import time: 1802 | 5741 | pkg_resources._vendor.pyparsing.results
import time: 20025 | 29503 | pkg_resources._vendor.pyparsing.core
import time: 3108 | 3108 | html.entities
import time: 2928 | 6036 | html
import time: 573 | 6608 | html.entities
import time: 7789 | 14396 | pkg_resources._vendor.pyparsing.helpers
import time: 1521 | 1521 | pkg_resources._vendor.pyparsing.testing
import time: 13619 | 13619 | pkg_resources._vendor.pyparsing.common
import time: 4689 | 75067 | pkg_resources._vendor.pyparsing
import time: 583 | 75649 | pkg_resources.extern.pyparsing
import time: 5159 | 5159 | pkg_resources.extern.packaging.markers
import time: 22587 | 103395 | pkg_resources.extern.packaging.requirements
import time: 184535 | 519610 | pkg_resources
import time: 1908 | 521518 | setuptools.version
import time: 1071 | 1071 | distutils.filelist
import time: 1282 | 2352 | setuptools.monkey
import time: 1407 | 3759 | setuptools.extension
import time: 1453 | 1453 | numbers
import time: 903 | 903 | distutils.command
import time: 1558 | 1558 | glob
import time: 1155 | 1155 | setuptools.extern
import time: 928 | 928 | setuptools._vendor
import time: 1161 | 1161 | setuptools._vendor.packaging.__about__
import time: 1835 | 3924 | setuptools._vendor.packaging
import time: 739 | 4663 | setuptools.extern.packaging
import time: 1558 | 1558 | setuptools._vendor.ordered_set
import time: 552 | 2110 | setuptools.extern.ordered_set
import time: 1122 | 1122 | setuptools._vendor.more_itertools.recipes
import time: 2943 | 4064 | setuptools._vendor.more_itertools.more
import time: 1437 | 5501 | setuptools._vendor.more_itertools
import time: 574 | 6074 | setuptools.extern.more_itertools
import time: 188 | 188 | _csv
import time: 1758 | 1945 | csv
import time: 1380 | 1380 | uu
import time: 1335 | 1335 | email._encoded_words
import time: 1681 | 1681 | email.iterators
import time: 3440 | 7833 | email.message
import time: 890 | 890 | importlib.metadata._functools
import time: 1271 | 2160 | importlib.metadata._text
import time: 2111 | 12103 | importlib.metadata._adapters
import time: 1294 | 1294 | importlib.metadata._meta
import time: 1215 | 1215 | importlib.metadata._collections
import time: 833 | 833 | importlib.metadata._itertools
import time: 4737 | 22125 | importlib.metadata
import time: 1367 | 23491 | setuptools._importlib
import time: 2059 | 2059 | distutils.command.bdist
import time: 1558 | 3617 | setuptools.command
import time: 27218 | 27218 | setuptools.windows_support
import time: 1843 | 1843 | setuptools._vendor.pyparsing.util
import time: 1624 | 1624 | setuptools._vendor.pyparsing.unicode
import time: 4246 | 5869 | setuptools._vendor.pyparsing.exceptions
import time: 1289 | 1289 | setuptools._vendor.pyparsing.actions
import time: 1531 | 1531 | setuptools._vendor.pyparsing.results
import time: 19838 | 21368 | setuptools._vendor.pyparsing.core
import time: 7374 | 7374 | setuptools._vendor.pyparsing.helpers
import time: 1568 | 1568 | setuptools._vendor.pyparsing.testing
import time: 11755 | 11755 | setuptools._vendor.pyparsing.common
import time: 4005 | 55068 | setuptools._vendor.pyparsing
import time: 491 | 55559 | setuptools.extern.pyparsing
import time: 1506 | 1506 | setuptools._vendor.packaging._manylinux
import time: 1416 | 1416 | setuptools._vendor.packaging._musllinux
import time: 2039 | 4960 | setuptools.extern.packaging.tags
import time: 973 | 973 | setuptools.extern.packaging._structures
import time: 2809 | 3781 | setuptools.extern.packaging.version
import time: 1780 | 10519 | setuptools.extern.packaging.utils
import time: 2973 | 13491 | setuptools.extern.packaging.specifiers
import time: 4931 | 18422 | setuptools.extern.packaging.markers
import time: 9815 | 83794 | setuptools.extern.packaging.requirements
import time: 1357 | 1357 | setuptools._path
import time: 3735 | 5092 | setuptools.config.expand
import time: 3330 | 92216 | setuptools.config.setupcfg
import time: 1815 | 94030 | setuptools.config
import time: 1016 | 1016 | setuptools.errors
import time: 5210 | 5210 | email._header_value_parser
import time: 2164 | 7374 | email.headerregistry
import time: 2746 | 10120 | setuptools.config._apply_pyprojecttoml
import time: 2576 | 13710 | setuptools.config.pyprojecttoml
import time: 2138 | 2138 | setuptools.discovery
import time: 972 | 972 | setuptools._vendor.jaraco
import time: 616 | 1587 | setuptools.extern.jaraco
import time: 1192 | 1192 | setuptools.extern.jaraco.functools
import time: 1112 | 1112 | setuptools.extern.jaraco.context
import time: 4001 | 7892 | setuptools.extern.jaraco.text
import time: 1401 | 9292 | setuptools._reqs
import time: 1414 | 1414 | setuptools._itertools
import time: 2423 | 3836 | setuptools._entry_points
import time: 5387 | 200628 | setuptools.dist
import time: 820 | 820 | setuptools.py34compat
import time: 1345 | 2164 | setuptools._imp
import time: 1573 | 3737 | setuptools.depends
import time: 982 | 982 | setuptools.logging
import time: 153 | 153 | _json
import time: 2034 | 2187 | json.scanner
import time: 1965 | 4151 | json.decoder
import time: 1785 | 1785 | json.encoder
import time: 1911 | 7846 | json
import time: 1128 | 1128 | unittest.util
import time: 1481 | 2608 | unittest.result
import time: 1909 | 1909 | difflib
import time: 2119 | 4028 | unittest.case
import time: 1228 | 1228 | unittest.suite
import time: 1865 | 1865 | unittest.loader
import time: 3592 | 3592 | argparse
import time: 958 | 958 | unittest.signals
import time: 1549 | 2507 | unittest.runner
import time: 1605 | 7703 | unittest.main
import time: 2818 | 20246 | unittest
import time: 1184 | 1184 | concurrent
import time: 3440 | 3440 | concurrent.futures._base
import time: 1790 | 6412 | concurrent.futures
import time: 7964 | 7964 | _ssl
import time: 9274 | 17238 | ssl
import time: 1391 | 1391 | asyncio.constants
import time: 868 | 868 | asyncio.format_helpers
import time: 1117 | 1984 | asyncio.base_futures
import time: 820 | 820 | asyncio.log
import time: 1571 | 4374 | asyncio.coroutines
import time: 145 | 145 | _contextvars
import time: 2057 | 2202 | contextvars
import time: 1602 | 1602 | asyncio.exceptions
import time: 1230 | 1230 | asyncio.base_tasks
import time: 3242 | 6072 | _asyncio
import time: 2104 | 10377 | asyncio.events
import time: 1124 | 1124 | asyncio.futures
import time: 1119 | 1119 | asyncio.protocols
import time: 1142 | 1142 | asyncio.transports
import time: 1674 | 2816 | asyncio.sslproto
import time: 860 | 860 | asyncio.mixins
import time: 1351 | 2210 | asyncio.locks
import time: 2600 | 2600 | asyncio.tasks
import time: 2229 | 7038 | asyncio.staggered
import time: 1900 | 1900 | asyncio.trsock
import time: 5153 | 58936 | asyncio.base_events
import time: 1002 | 1002 | asyncio.runners
import time: 1074 | 1074 | asyncio.queues
import time: 1249 | 1249 | asyncio.streams
import time: 1088 | 1088 | asyncio.subprocess
import time: 1053 | 1053 | asyncio.threads
import time: 3728 | 3728 | _overlapped
import time: 1143 | 1143 | asyncio.base_subprocess
import time: 2159 | 2159 | asyncio.proactor_events
import time: 1679 | 1679 | asyncio.selector_events
import time: 1086 | 1086 | asyncio.windows_utils
import time: 3436 | 13228 | asyncio.windows_events
import time: 3165 | 80792 | asyncio
import time: 3879 | 104916 | unittest.mock
import time: 1526 | 1526 | distutils.ccompiler
import time: 9827 | 1014210 | distutils
import time: 2366 | 1016576 | distutils.versionFull `-X importtime` output with `stdlib`$ SETUPTOOLS_USE_DISTUTILS=stdlib python -X importtime -c 'import distutils.version'
import time: self [us] | cumulative | imported package
import time: 354 | 354 | _io
import time: 99 | 99 | marshal
import time: 375 | 375 | nt
import time: 170 | 170 | winreg
import time: 1999 | 2995 | _frozen_importlib_external
import time: 740 | 740 | time
import time: 590 | 1330 | zipimport
import time: 271 | 271 | _codecs
import time: 1776 | 2047 | codecs
import time: 1661 | 1661 | encodings.aliases
import time: 3131 | 6838 | encodings
import time: 944 | 944 | encodings.utf_8
import time: 966 | 966 | encodings.cp1252
import time: 113 | 113 | _signal
import time: 122 | 122 | _abc
import time: 1401 | 1522 | abc
import time: 1357 | 2879 | io
import time: 181 | 181 | _stat
import time: 1794 | 1975 | stat
import time: 2946 | 2946 | _collections_abc
import time: 1108 | 1108 | genericpath
import time: 2004 | 3111 | ntpath
import time: 2994 | 11025 | os
import time: 1277 | 1277 | _sitebuiltins
import time: 5237 | 5237 | sitecustomize
import time: 1356 | 1356 | usercustomize
import time: 6361 | 25254 | site
import time: 1623 | 1623 | warnings
import time: 278 | 278 | itertools
import time: 1877 | 1877 | keyword
import time: 205 | 205 | _operator
import time: 1821 | 2025 | operator
import time: 1326 | 1326 | reprlib
import time: 1163 | 1163 | _collections
import time: 4015 | 10680 | collections
import time: 1702 | 1702 | types
import time: 217 | 217 | _functools
import time: 2845 | 15442 | functools
import time: 2614 | 2614 | enum
import time: 200 | 200 | _sre
import time: 1563 | 1563 | sre_constants
import time: 1938 | 3500 | sre_parse
import time: 2976 | 6675 | sre_compile
import time: 165 | 165 | _locale
import time: 1200 | 1200 | copyreg
import time: 2944 | 13596 | re
import time: 1141 | 1141 | token
import time: 3498 | 18235 | tokenize
import time: 1570 | 35246 | linecache
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
import time: 2598 | 39466 | distutils
import time: 2423 | 41889 | distutils.version |
|
In Gentoo: Note that the number of setuptools plugins installed probably matters (which is why using setuptools itself is slow). |
webknjaz
added
Needs Triage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
setuptools version
setuptools==65.3.0
Python version
python 3.8.10
OS
ubuntu 20.04
Additional environment information
No response
Description
when using the bundled
distutils, importing any part of distutils takes almost 10x longer than previouslyI originally started with trying to figure out why
virtualenvgot way slower recently and ended up tracing it to setuptools>=60Expected behavior
I expect the bundled distutils to not incur the import time problems of
setuptoolsHow to Reproduce
compare the following:
time env SETUPTOOLS_USE_DISTUTILS=local venv3/bin/python3 -c 'import distutils.version'time env SETUPTOOLS_USE_DISTUTILS=stdlib venv3/bin/python3 -c 'import distutils.version'Output
best-ofhere is just a script which runs the command in a loop and reports the fastest time --timealso works to reproduce thisThe text was updated successfully, but these errors were encountered: