-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
site.py imports relatively large sysconfig
module.
#73771
Comments
site.py uses sysconfig (and sysconfigdata, _osx_support) module for user-site package. But sysconfig module is not so lightweight, and very rarely used. And it takes about 7% of startup time, only for searching user-site path. I tried to port minimal subset of sysconfig into site.py (GH-136). How can I do to solve this? a) Drop "osx_framework_user" ( |
Instead of using slow sysconfig and loading the big _sysconfig_data dictionary in memory, would it be possible to extract the minimum set of sysconfig needed by the site module and put it in a builtin module? In site.py, I only found 4 variables: from sysconfig import get_config_var
USER_BASE = get_config_var('userbase')
from sysconfig import get_path
USER_SITE = get_path('purelib', 'osx_framework_user')
USER_SITE = get_path('purelib', '%s_user' % os.name)
from sysconfig import get_config_var
framework = get_config_var("PYTHONFRAMEWORK") Because of the site module, the _sysconfig_data module dictionary is always loaded in memory even for for a dummy print("Hello World!"). I suggest to start building a _site builtin module: subset of site.py which would avoid sysconfig and reimplement things in C for best performances. speed.python.org:
Importing site takes 6 ms: 42% of 14 ms... I'm interested to know if it would be possible to reduce these 6 ms by rewriting some parts of site.py in C. |
Serhiy collected interesting numbers, copy/paste of this message: On my computer: Importing empty module: 160 us |
What's your platform, Inada? Are you running macOS? I optimized site.py for Linux and BSD users a couple of years ago. |
Christian: I'm using macOS on office and Linux on home. sysconfig is imported even on Linux |
I don't think rewriting party of site.py in C is a good idea. It's a rather maintenance intense module. However, optimizing access is certainly something that's possible, e.g. by placing the few variables that are actually needed by site.py into a bootstrap module for sysconfig, which only contains the few variables needed by interpreter startup. Alternatively, sysconfig data could be made available via a C lookup function; with the complete dictionary only being created on demand. get_config_var() already is such a lookup API which could be used as front-end. |
Marc-Andre Lemburg added the comment:
Right, I don't propose to rewrite the 598 lines of site.py in C, but I'm proposing to add a new private module because I don't want to I looked at site.py history: I don't see *major* changes last 2 years.
I don't think that it's worth it to reimplement partially sysconfig in Well, I'm not sure about what is the best approach, but I'm sure that I never liked site.py. It seems like a huge workaround. I also dislike That's why I asked Steve Dower to removing the code to create the |
Instead of _site, would it make sense to include the four vars in sys, perhaps as named structure like sys.flags? |
FYI, here is profile of site: |
no-site default: GH-136 + skip abs_paths(). profile of GH-136 + skip abs_paths(): Most of time is consumed by importlib. |
On 17.02.2017 13:06, STINNER Victor wrote:
Sorry, I was just referring to the data part of sysconfig, Having a lookup function much like we have for unicodedata |
PR 136 now adds |
Test fails on macOS: http://buildbot.python.org/all/builders/x86-64%20Sierra%203.x/builds/402/steps/test/logs/stdio ====================================================================== Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.x.billenstein-sierra/build/Lib/test/test_site.py", line 266, in test_getsitepackages
self.assertEqual(len(dirs), 2)
AssertionError: 1 != 2 |
New changeset b01c574 by Victor Stinner in branch 'master': |
test_get_path fails on macOS installed framework builds: ====================================================================== Traceback (most recent call last):
File "/Users/nad/Projects/PyDev/active/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.7/lib/python3.7/test/test_site.py", line 188, in test_get_path
sysconfig.get_path('purelib', os.name + '_user'))
AssertionError: '/Users/nad/Library/pytest_10.12/3.7/lib/python/site-packages' != '/Users/nad/Library/pytest_10.12/3.7/lib/python3.7/site-packages'
- /Users/nad/Library/pytest_10.12/3.7/lib/python/site-packages
+ /Users/nad/Library/pytest_10.12/3.7/lib/python3.7/site-packages
? +++ Ran 27 tests in 0.471s FAILED (failures=1, skipped=4) |
https://docs.python.org/3.6/library/site.html#site.USER_SITE
So it seems I broke sysconfig.get_path('purelib', 'posix_user'). |
https://github.com/python/cpython/pull/136/files
+ if USER_SITE is None: OK, I need to use |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: