Importing pip crashes when site.USER_SITE is None

[brechtm]

• Pip version: 9.0.1
• Python version: embeddable Python 3.6.1
• Operating system: Windows

In the embeddable Python distribution for Windows, site.USER_SITE is None by design since 3.6.0. Pip does not handle this case and crashes on import:

Traceback (most recent call last):
  ...
  File "C:\Program Files\rinoh\pkgs\rinoh\resource.py", line 13, in <module>
    import pip
  File "C:\Program Files\rinoh\pkgs\pip\__init__.py", line 26, in <module>
    from pip.utils import get_installed_distributions, get_prog
  File "C:\Program Files\rinoh\pkgs\pip\utils\__init__.py", line 23, in <module>
    from pip.locations import (
  File "C:\Program Files\rinoh\pkgs\pip\locations.py", line 88, in <module>
    bin_user = os.path.join(user_site, 'Scripts')
  File "ntpath.py", line 75, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

See also this CPython ticket.

[pfmoore]

While I agree that pip should handle a value of None for USER_SITE gracefully, I would point out that it isn't really intended for users to be running pip with the embeddable distribution directly. The intended workflow is to develop your application using a standard Python environment, and once complete, bundle it with the embeddable distribution as part of the release process for your application.

If you have a use case for running pip from the embeddable distribution, then you should probably describe it in more detail.

[brechtm]

My application, rinohtype manages optional dependencies such as fonts using Python packages on PyPI. Here's the list of fonts that are currently available, for example. When a specific font is referenced in a style sheet, rinohtype attempts to install it automatically from PyPI using pip.

In the future, I would also like to use pip to allow the stand-alone rinohtype distribution to update itself. In that case the embedded Python is not updated, but the rinohtype package is.

[pfmoore]

Hmm, so in effect you're using PyPI as your application's plugin repository. I don't know whether that's something we'd consider to be a reasonable use case, but thanks for the clarification. Let's see what the other @pypa/pip-committers think.

[pfmoore]

I should note that as I stated, I agree pip should handle the lack of USER_SITE gracefully. However, I assume your code doesn't actually need USER_SITE to be set, and you use something like --target to install the packages you request somewhere suitable for your application.

Note the following comment from the documentation (https://docs.python.org/3.6/using/windows.html#embedded-distribution):

Using pip to manage dependencies as for a regular Python installation is not supported with this distribution, though with some care it may be possible to include and use pip for automatic updates. In general, third-party packages should be treated as part of the application (“vendoring”) so that the developer can ensure compatibility with newer versions before providing updates to users.

[brechtm]

Indeed, USER_SITE is of no importance to rinohtype. And, yes, I will eventually want to install the fonts to a directory somewhere under %USERPROFILE%.

And thanks for the pointer to the documentation. That's good to remember.

[zooba]

FWIW, one of the teams I work with is also considering using PyPI as essentially a (public) plugin repository like this, and will eventually distribute using the embeddable distro.

It actually looks like this would be an issue whenever you run python -S -m pip ..., since as of some version of Python 3 we stopped running site.main() on import:

# Prevent extending of sys.path when python was started with -S and
# site is imported later.
if not sys.flags.no_site:
    main()

So if pip is going to import site and rely on it being fully initialized, it needs to call site.main() explicitly or use site.getusersitepackages() rather than directly accessing site.USER_SITE. I don't see anything else that needs to change in pip.locations other than the one line accessing USER_SITE.

[pfmoore]

@zooba Thanks for the analysis - the call to getusersitepackages() needs to be for Python 3.2+ only, but otherwise I agree. #4442 raised for this. @brechtm Could you test the PR and confirm it fixes your issue?

[zooba]

@pfmoore It was also added to 2.7, so it may be available in all supported Python versions.

[pfmoore]

Weird. I'm getting incomprehensible failures in CI ("module 'site' has no attribute 'getusersitepackages'") I can't reproduce this locally outside of pip. Can anyone suggest what might be going wrong? Do we mock the site module in our tests or something like that?

[dstufft]

Virtual environments use a site.py copied from like Python 2.4 and randomly updated with new features. I bet it needs updated too.

…

Sent from my iPhone

On Apr 12, 2017, at 2:23 PM, Paul Moore ***@***.***> wrote: Weird. I'm getting incomprehensible failures in CI ("module 'site' has no attribute 'getusersitepackages'") I can't reproduce this locally outside of pip. Can anyone suggest what might be going wrong? Do we mock the site module in our tests or something like that? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pfmoore]

Oh lovely. Well I'm not going down that rabbit hole now. I can do a check for the attribute and use it if present. That's probably better anyway. Thanks for that!

[ljgww]

embedded python - love it
pip on anything larger than py3.5 - :(
cx_Freeze and the bunch is useful but cumbersome to work with for quick solutions that need frequent update (e.g. change one script often). Embedded python + pip is great option to distribute quick internal apps (without installing python on the target - which is sometimes in enterprise environment a big trouble)
in essence pip here serves only as help to prepare runnable image, not to distribute pip with the app.
You shall make an alternative way for code authors how to add libs to embedded python that is not very 'hacky'

[brechtm]

@pfmoore I can confirm the PR fixes my problem. Thanks!

[pfmoore]

Thanks. I've merged the PR so it should be in the next release of pip.

[xavfernandez]

Closed by #4442