Add python.ini file for embedded/applocal installs

[zooba]

BPO	23955
Nosy	@pfmoore, @ronaldoussoren, @ncoghlan, @tjguk, @ned-deily, @merwok, @takluyver, @zware, @zooba
Files	23955_1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2015-05-22.22:21:14.326>
created_at = <Date 2015-04-14.19:16:30.917>
labels = ['interpreter-core', 'type-feature', 'OS-windows']
title = 'Add python.ini file for embedded/applocal installs'
updated_at = <Date 2015-05-22.22:21:14.325>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2015-05-22.22:21:14.325>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2015-05-22.22:21:14.326>
closer = 'steve.dower'
components = ['Interpreter Core', 'Windows']
creation = <Date 2015-04-14.19:16:30.917>
creator = 'steve.dower'
dependencies = []
files = ['39276']
hgrepos = []
issue_num = 23955
keywords = ['patch']
message_count = 18.0
messages = ['240989', '241343', '241422', '241424', '241670', '242002', '242020', '242038', '242042', '242044', '242100', '242125', '242127', '242450', '242589', '242865', '243857', '243859']
nosy_count = 11.0
nosy_names = ['richard', 'paul.moore', 'ronaldoussoren', 'ncoghlan', 'tim.golden', 'ned.deily', 'eric.araujo', 'python-dev', 'takluyver', 'zach.ware', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23955'
versions = ['Python 3.5']

[zooba]

When Python starts running we (getpathp.c) looks in the registry for its library directory. To avoid this, you can set PYTHONPATH and PYTHONHOME environment variables, but this isn't always easy if you've copied Python into your app's directory.

We should look for a python35.ini alongside python35.dll when initializing and extract values from it. Possibly this could just be an [Environment] section with key-value pairs to set before loading.

This would ensure that app-local Python installations never look for a global install in the registry. (As an aside, for the 3.5 beta we'll be shipping an app-local "installer", which will make this situation easier to end up in, but it also has value for xcopy installs.)

[merwok]

Hm, could you reuse the venv config file with use-site-packages=true?

[zooba]

I could reuse the venv config file but it would need a new property to forcibly use the 'home' value regardless of whether '$home\Lib\os.py' exists (in case the stdlib is zipped) and to make it relative to the file's path rather than the user's current directory.

How about adding an 'applocal=True' property to pyvenv.cfg that basically implies -I, PYTHONHOME=os.dirname(pyvenv_cfg), and skips the registry lookup on Windows completely (which there's currently no way to achieve)?

[zooba]

Arguably we should be making 'home' in pyvenv.cfg relative to the pyvenv.cfg file's directory anyway... AFAIK it's always generated as an absolute path, but I see no good reason to prevent people from manually configuring a relative path here.

[takluyver]

Relative paths would be nice for Pynsist - I would prefer to create the config file at build time and then install it along with the other files. If it needed absolute paths, then the installer would have to write the config file after the user selects the installation directory. That's doable, but it's easier to write and test the build steps than the install steps.

[ncoghlan]

I'd actually like a python.ini file defined as a Python 3.5 feature in order to provide a better backporting story for the PEP-476 certificate changes in downstream long term support releases of Python 2.7.

So +1 in principle, but I'd like to see this written up as a PEP (the SSL configuration setting doesn't need to be in the PEP, but the optional existence of the config file and reading it at startup does).

[pfmoore]

As described. this seems to be a Windows-only feature (it's replacing a registry lookup with a config file alongside the Python DLL).

So I'm not sure I understand Nick's comment - Nick, are you suggesting this should be a cross-platform feature? If so, what's the equivalent of "alongside the Python DLL", and what would the ini file be used for on Unix, where there's no registry lookup to override?

[ncoghlan]

If we're adding a static startup configuration file, then yes, I believe that should be a cross-platform feature, not something Windows specific, as it's a major change to our configuration philosophy. (I think it's a *necessary* change, but it's also a significant one)

Even if it started out as Windows only, we should have a forward looking plan for how it relates to the startup sequence changes proposed in PEP-432.

[zooba]

I'm leaning towards clarifying/fixing the behavior of the venv config, since it's most of the way there for what I care about. Adding another option for applocal may be needed, but possibly not, in which case defining how relative paths are handled would be sufficient.

Any other startup file I'd want to define as setting environment variables only, to avoid having multiple ways to set properties. So for most cases a shell script would suffice (doesn't help people linking to libpython directly, but they can call SetPythonHome themselves).

[ncoghlan]

Treating "local Python" as a special-case of venv sounds like a good approach, then (especially as it currently seems plausible we'll end up going for environment variable based approach to the downstream PEP-476 backport)

[zooba]

Having looked at the implementation, the easiest way to do this will be to add an "applocal = true" option to pyvenv.cfg, which would behave identically to setting %PYTHONHOME% to the directory containing the config before launch.

I wanted to support relative paths for "home = " but the path handling is so far from sufficient that it wouldn't be secure. As it is, I've fixed a lot of potential buffer overruns in getpathp.c already, but I don't want to implement true relative paths, so I'd prefer to require "home = " to be absolute and add a second property that effectively means "home = .".

Any thoughts/comments/preferences?

[takluyver]

Would that option be the only thing that needs to be set to make Python app-local? I'm not familiar with what lives in pyvenv.cfg.

[zooba]

None of the other options really have much use in this case, since they're mostly about how to combine multiple environments rather than excluding implicit ones.

Setting PYTHONHOME to the current directory certainly seems to be enough AFAICT, at least for Windows, as that bypasses the most problematic registry lookup. Having an "applocal" option means we can go further and ignore all registry keys and environment variables.

I haven't looked into how to duplicate the behaviour on other OSs, but Modules/getpath.c looks similar enough to PC/getpathp.c that I guess it'll be similar. Adding the MacOS folks in case they have suggestions/concerns.

[zooba]

Added a patch that:

• adds the applocal option to pyvenv.cfg
• updates the docs to describe how pyvenv.cfg (home and applocal) affect sys.path
• puts a default pyvenv.cfg in the ZIP distribution
• adds significantly more buffer overflow protection to getpathp.c
• prevents a case where sys.path may include multiple blank entries

Currently none of this applies to Linux or Mac builds, though I'm led to believe there would be some value if it did. I'm not the one who can figure all the intricacies out for those platforms though.

[ncoghlan]

bpo-23857 was the one where I thought there might be an overlap with a design consideration on Linux (related to coming up with a conventional for backporting PEP-476 as a CPython redistributor).

However, I've now suggested a different path forward there, closer to what happened with PEP-394 (which provides guidance to redistributors on symlink definitions, without making any changes to upstream CPython itself).

[zooba]

Any further comments/bikeshedding on the applocal parameter? I still need to do some docs, but I want the basic design agreed upon first.

[python-dev]

New changeset 620a247b4960 by Steve Dower in branch 'default':
Issue bpo-23955: Add pyvenv.cfg option to suppress registry/environment lookup for generating sys.path.
https://hg.python.org/cpython/rev/620a247b4960

[zooba]

The support is in and the option remains named "applocal".

Hopefully this (and the ZIP file version, which will include the preconfigured pyvenv.cfg) will help with app distributions, though I'm still open to make changes if it doesn't.