Implement sysconfig-based location inference

[uranusjr]

This detects location differences between distutils and sysconfig, and tell users to comment in #9617.

[pypa-bot]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

[uranusjr]

Okay, I think I’ve caught all the pip-specific edge cases. The only remaining failures are all PyPy. Its purelib and platlib values differ:

distutils: {sys.prefix}/site-packages
sysconfig: {sys.prefix}/lib/pypy3.6/site-packages

and this is arguably exactly the kind of mismatches we want to catch and help implementations fix. Who should we reach out to discuss this with PyPy folks?

[pradyunsg]

This is really great and looks good to me. The only thought I have is we should probably add more "proper unit tests" for the newly introduced sysconfig-related code, instead of only patching the existing ones.

But, eh.

[pradyunsg]

/cc @mattip for help with PyPy's sysconfig vs distutils.sysconfig stuff. :)

[mattip]

Thanks for pinging me. Please test the latest PyPy3.6 and 3.7, which is available on github actions via the actions/setup-python@v2 action, this will also help you move yet another few builds off travis.

As for distutils.sysconfig vs. sysconfig: where is there an _INSTALL_SCHEME in distutils.sysconfig? PyPy intentionally sets up site-packages to be in a different location than CPython, to prevent confusion, so the _INSTALL_SCHEME in sysconfig is changed on purpose. Could you help me understand what is actually failing in the tests?

[uranusjr]

The distutils-based implementation uses distutils.command.install.INSTALL_SCHEME to get the site-packages directory.

I checked PyPy’s implementation, it seems like the mismatch happens because our sysconfig implementation does not use the pypy key in sysconfig._INSTALL_SCHEMES, but posix_home when the “home” scheme is requested. Should we use pypy here instead?

---

This also seems to surface another issue in sysconfig: There is no way or a tools to ask the interpreter what are their preferred scheme names. We have _get_default_scheme() for the “prefix” schemes, but no equivalents for “user” and “home”, so pip can only guess the scheme names when it needs “the current platform/implementation’s user scheme name” for pip install --user (and likewise for “home” on --target), and implementations like PyPy has no way to communicate to tools messages like “please use pypy instead of posix_home”. This should probably be raised in Discourse for CPython core devs to consider.

[mattip]

Hmm. It seems there are a few problems. One is that PyPy never bothered to sync up distutils.command.install with the changes it made to sysconfig. Another is the difference in schemes, that it never backported to CPython. But +- bugs, there should be no need for pip to adjust posix_name or posix_user, this should be provided by the sysconfig implementation, that each implementation can modify to fit its needs.

If we want a single source for sysconfig across implementations, I think the best course would be to modify sysconfig's scheme layout and add a implementation key, that could get its value from platform.python_implementation(). This would look something like what PyPy did. Note that this also impacts site.py since they copy out parts of sysconfig.

[mattip]

We have _get_default_scheme() for the “prefix” schemes ...

It seems that is a private interface, the public one is sysconfig.get_path()

[mattip]

Yeah, this is all very emperical work, and on top of what the implementations do, the OS distributors do their thing too. Anyhow, thanks for pointing out the distutils install command, I have already made a stab at fixing that in PyPy. Could you point me to the tests that discovered the discrepancy?

[pradyunsg]

I mean, part of why I want distutils out of the standard library is this stuff, but I’m glad that @uranusjr filed this PR, because this has always felt like a field of thorns - it touches legacy defacto stuff + new stuff and alternate Python implementations. :)

[uranusjr]

I’ve raised this in bpo-43312.

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

[pradyunsg]

@uranusjr Let's get this up to date, mark the tests as xfail(strict=True) on PyPy, and merge? The longer we let this linger, the slower our progress would be here. :)

[uranusjr]

Let me try fixing PyPy instead. I’ll do the xfails if this does not work…

[uranusjr]

🔔 THIS WORKS 🔔

[pradyunsg]

AHAHAHAHA. Let's do this! :)

[pradyunsg]

FWIW, I'm seeing that this fails with GitHub's pypy3:

https://github.com/pradyunsg/pip/pull/9/checks?check_run_id=2250267986#step:6:4862

[mattip]

I don't see any failures there. Could you copy-paste the failing test or otherwise point out what is failing on pypy3 ?

[pradyunsg]

https://github.com/pradyunsg/pip/runs/2250246469?check_suite_focus=true

Whoops, I think what was the wrong tab's URL. Sorry!

[mattip]

If I understand the log correctly, it seems there is a mismatch between distutils and sysconfig?

WARNING: Value for scheme.platlib does not match.  \
    Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: <workspace>/venv/user/lib/python3.6/site-packages
sysconfig: <workspace>/venv/user/lib/pypy3.6/site-packages
 WARNING: Value for scheme.purelib does not match. \
    Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: <workspace>/venv/user/lib/python3.6/site-packages
sysconfig: <workspace>/venv/user/lib/pypy3.6/site-packages
WARNING: Additional context:
        user = True
        home = None
        root = None
        prefix = None

... and going over to issue #9617 I see you already reached the same conclusion :). I will continue there.