New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an "isolated mode" that ignores user provided default settings #1397

Closed
ncoghlan opened this Issue Dec 23, 2013 · 13 comments

Comments

Projects
None yet
5 participants
@ncoghlan
Member

ncoghlan commented Dec 23, 2013

In #1359 I suggested that pip should pay attention to Python's "isolated mode" flag and ignore environment variables in that case.

Paul Moore suggested it was inappropriate for pip to do that, so instead I changed CPython to remove all PIP_* environment variables before invoking pip (http://bugs.python.org/issue19734).

However, in implementing that, I realised we had the same problem with the default config file: the settings for direct invocation of pip are unlikely to be correct for ensurepip (http://bugs.python.org/issue20053)

In the near-term, I think I can work around this in ensurepip (by setting PIP_CONFIG_FILE appropriately), but longer term it might be cleaner to have an "isolated mode" equivalent in pip that ignores all user provided settings (alternatively, ignoring all other settings when ENSUREPIP_OPTIONS is set could also work).

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Dec 28, 2013

Contributor

the isolation mode would amount to clearing PIP_* from the environment, and blocking the lookup of pip.ini files.
what about pydistutils files? we have a scheme function (for wheel installs) that uses distutils classes that honors pydistutils files.

btw, we really need the config file path as a cli option (not just the PIP_CONFIG_FILE environment var)

Contributor

qwcode commented Dec 28, 2013

the isolation mode would amount to clearing PIP_* from the environment, and blocking the lookup of pip.ini files.
what about pydistutils files? we have a scheme function (for wheel installs) that uses distutils classes that honors pydistutils files.

btw, we really need the config file path as a cli option (not just the PIP_CONFIG_FILE environment var)

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 23, 2014

Member

Coming back around to this, we have a number of places configuration can be introduced.

  1. Through the command line.
  2. Through environment variables.
  3. Through a virtual environment pip.conf.
  4. Through a user specific pip.conf.
  5. Through a site specific pip.conf.
  6. Through a user specific pydistutils.cfg.
  7. Through a site specific distutils.cfg.

My gut is that when running in the isolated mode we should allow configuration through 1 and 7, we should disallow it through 2, 4, and 6. The question then to me is should we allow /etc/pip/pip.conf and should we allow a virtualenv's pip.conf file?

Member

dstufft commented Nov 23, 2014

Coming back around to this, we have a number of places configuration can be introduced.

  1. Through the command line.
  2. Through environment variables.
  3. Through a virtual environment pip.conf.
  4. Through a user specific pip.conf.
  5. Through a site specific pip.conf.
  6. Through a user specific pydistutils.cfg.
  7. Through a site specific distutils.cfg.

My gut is that when running in the isolated mode we should allow configuration through 1 and 7, we should disallow it through 2, 4, and 6. The question then to me is should we allow /etc/pip/pip.conf and should we allow a virtualenv's pip.conf file?

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Nov 23, 2014

Member

I think the answer is "yes" for both 3 & 5. After our "when is overriding appropriate?" discussion, I'm thinking the following model makes sense:

  1. Virtual environments can override system level settings (that's what they're for)
  2. User level settings either can't override system settings at all, or you can opt out of them by using isolated mode

CPython itself isn't entirely consistent on that front, but that's a situation that's evolved over several years - it should be possible for pip to come up with something a bit more coherent.

Member

ncoghlan commented Nov 23, 2014

I think the answer is "yes" for both 3 & 5. After our "when is overriding appropriate?" discussion, I'm thinking the following model makes sense:

  1. Virtual environments can override system level settings (that's what they're for)
  2. User level settings either can't override system settings at all, or you can opt out of them by using isolated mode

CPython itself isn't entirely consistent on that front, but that's a situation that's evolved over several years - it should be possible for pip to come up with something a bit more coherent.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 23, 2014

Member

For pip it works a virtual environment overrides the user overrides the site specific, and I think that makes the most sense. It's going from most to least specific.

Member

dstufft commented Nov 23, 2014

For pip it works a virtual environment overrides the user overrides the site specific, and I think that makes the most sense. It's going from most to least specific.

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Nov 23, 2014

Member

Yep, agreed. I was agreeing with the notion that an "isolated" mode should drop the user level settings, but keep the virtual environment and system/site settings.

That's the way "-I" works for CPython:

$ python3 -Im site
sys.path = [
    '/home/ncoghlan/.virtualenvs/py3misc/lib64/python34.zip',
    '/home/ncoghlan/.virtualenvs/py3misc/lib64/python3.4',
    '/home/ncoghlan/.virtualenvs/py3misc/lib64/python3.4/plat-linux',
    '/home/ncoghlan/.virtualenvs/py3misc/lib64/python3.4/lib-dynload',
    '/usr/lib64/python3.4',
    '/usr/lib/python3.4',
    '/home/ncoghlan/.virtualenvs/py3misc/lib/python3.4/site-packages',
]
USER_BASE: '/home/ncoghlan/.local' (exists)
USER_SITE: '/home/ncoghlan/.local/lib/python3.4/site-packages' (exists)
ENABLE_USER_SITE: False
Member

ncoghlan commented Nov 23, 2014

Yep, agreed. I was agreeing with the notion that an "isolated" mode should drop the user level settings, but keep the virtual environment and system/site settings.

That's the way "-I" works for CPython:

$ python3 -Im site
sys.path = [
    '/home/ncoghlan/.virtualenvs/py3misc/lib64/python34.zip',
    '/home/ncoghlan/.virtualenvs/py3misc/lib64/python3.4',
    '/home/ncoghlan/.virtualenvs/py3misc/lib64/python3.4/plat-linux',
    '/home/ncoghlan/.virtualenvs/py3misc/lib64/python3.4/lib-dynload',
    '/usr/lib64/python3.4',
    '/usr/lib/python3.4',
    '/home/ncoghlan/.virtualenvs/py3misc/lib/python3.4/site-packages',
]
USER_BASE: '/home/ncoghlan/.local' (exists)
USER_SITE: '/home/ncoghlan/.local/lib/python3.4/site-packages' (exists)
ENABLE_USER_SITE: False
@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 23, 2014

Member

Ah, I misread what you were saying. I thought you were saying to drop virtualenv/site settings and then going into a tangent :)

Member

dstufft commented Nov 23, 2014

Ah, I misread what you were saying. I thought you were saying to drop virtualenv/site settings and then going into a tangent :)

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Nov 23, 2014

Member

Go off on a tangent? Me? That never happens! :)

Member

ncoghlan commented Nov 23, 2014

Go off on a tangent? Me? That never happens! :)

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 23, 2014

Member

Oh and to be clear, that solution sounds good to me. I'll implement something like that soon.

Member

dstufft commented Nov 23, 2014

Oh and to be clear, that solution sounds good to me. I'll implement something like that soon.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Nov 23, 2014

Member

Can I just check one point? I didn't realise pip configuration could be read from pydistutils.cfg/distutils.cfg. Are we talking about pip config here, or about setuptools config that would be read when pip executes (say) setup.py install? Because one use of pydistutils.cfg is to set up a central location where C libraries can be put on Windows (which doesn't have anything like /usr/lib). Having that stop working because pip was in isolated mode seems odd.

Member

pfmoore commented Nov 23, 2014

Can I just check one point? I didn't realise pip configuration could be read from pydistutils.cfg/distutils.cfg. Are we talking about pip config here, or about setuptools config that would be read when pip executes (say) setup.py install? Because one use of pydistutils.cfg is to set up a central location where C libraries can be put on Windows (which doesn't have anything like /usr/lib). Having that stop working because pip was in isolated mode seems odd.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 23, 2014

Member

Sorry there's no pip specific configuration in the distutils config files, it does control where Wheels and setuptools will drop things though. I don't use those files at all though I know homebrew uses the site specific one. So my suggestion of ignoring the user specific one was more of just a guess to be consistent. If it makes more sense not to ignore them that's fine with me.

Member

dstufft commented Nov 23, 2014

Sorry there's no pip specific configuration in the distutils config files, it does control where Wheels and setuptools will drop things though. I don't use those files at all though I know homebrew uses the site specific one. So my suggestion of ignoring the user specific one was more of just a guess to be consistent. If it makes more sense not to ignore them that's fine with me.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Nov 23, 2014

Member

Well, I'm not even sure distutils/setuptools has an isolated mode, so it may not be possible to invoke setup.py in a way that ignores the config files. This would affect pip wheel and pip install from sdist, which are not relevant to the original ensurepip use case, AIUI.

So it may not even be something worth worrying about.

Member

pfmoore commented Nov 23, 2014

Well, I'm not even sure distutils/setuptools has an isolated mode, so it may not be possible to invoke setup.py in a way that ignores the config files. This would affect pip wheel and pip install from sdist, which are not relevant to the original ensurepip use case, AIUI.

So it may not even be something worth worrying about.

@tdsmith

This comment has been minimized.

Show comment
Hide comment
@tdsmith

tdsmith Nov 30, 2014

Contributor

it may not be possible to invoke setup.py in a way that ignores the config files

distutils and setuptools setup.py's take a --no-user-cfg option, mentioned here. Homebrew passes that switch to the respective setup.py's when installing pip and setuptools after building Python to make sure they land in the Homebrew site-packages and script directories.

A similar flag that disables user pydistutils.cfg's for ensurepip but respects whatever site configuration there is would be helpful for Homebrew if we migrated to using ensurepip.

(Right now we install pip and setuptools by staging their sdist tarballs on the user's machine and running setup.py after Python is installed (either after a build or after unpacking our "bottled" prebuilt archive). When we install pip and setuptools, we're running in our user's environment. I don't know for sure that anyone has ever written a pydistutils.cfg but in the interest of robustness...)

Contributor

tdsmith commented Nov 30, 2014

it may not be possible to invoke setup.py in a way that ignores the config files

distutils and setuptools setup.py's take a --no-user-cfg option, mentioned here. Homebrew passes that switch to the respective setup.py's when installing pip and setuptools after building Python to make sure they land in the Homebrew site-packages and script directories.

A similar flag that disables user pydistutils.cfg's for ensurepip but respects whatever site configuration there is would be helpful for Homebrew if we migrated to using ensurepip.

(Right now we install pip and setuptools by staging their sdist tarballs on the user's machine and running setup.py after Python is installed (either after a build or after unpacking our "bottled" prebuilt archive). When we install pip and setuptools, we're running in our user's environment. I don't know for sure that anyone has ever written a pydistutils.cfg but in the interest of robustness...)

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 30, 2014

Member

I'm going to reopen this. I asked @tdsmith to post the above. I think it might make sense for --isolated to ignore ~/.pydistutils.cfg and for that to imply passing the --no-user-cfg option to setup.py as well.

Member

dstufft commented Nov 30, 2014

I'm going to reopen this. I asked @tdsmith to post the above. I think it might make sense for --isolated to ignore ~/.pydistutils.cfg and for that to imply passing the --no-user-cfg option to setup.py as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment