PYTHONPATH is not picked up in isolated mode even if use_environment is 1 #101471
Comments
|
This might be the culprit: initconfig.c, _PyConfig_Read() function, but I am not sure: My breakpoint triggers on this like if I start it with config.use_environment==1 Breakpoint 1, _PyConfig_Read (config=0x7fffffffbd20, compute_path_config=0) at ../Python/initconfig.c:2942 |
This is an attempt to work around python/cpython#101471 until a more permanent solution is found. Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
|
This behaviour is intentional. The documentation is likely unclear, but the If you want to pick up paths from the environment, don't use an isolated config, or (much much better), read them yourself and add them to the search path directly. You might even consider using a different variable for your app, to avoid being broken by users who set |
|
Hi,
I has not been my intention to allow the use of PYTHONPATH as an
environment variable myself. It was just a means to an end, namely: how do
I extend the standard search path in an isolated interpreter?
If try to do that with
changing module_search_paths and module_search_paths_set then that
overrides sys.path completely, dropping the standard directories too.
I want the standard directories, plus an extra directory in front. How can
I do that from my embedding application, intending to use isolated mode,
without writing code that changes sys.path after initialization? Or that's
the recommended way?
Thanks
…On Fri, Feb 3, 2023, 17:22 Steve Dower ***@***.***> wrote:
Closed #101471 <#101471> as
completed.
—
Reply to this email directly, view it on GitHub
<#101471 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFOK5R4JEMIO2AUABZ5MPTWVUWEBANCNFSM6AAAAAAUMXTMPY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
There aren't really any "standard" directories once you're embedding the interpreter - it's all up to you at that point. So make sure you include the directory that has the standard library (putting the stdlib into a zip file is often a good idea, too), and the directory that has the standard library extension modules, and also any other directories you want to reference. (Personally if I had my way, I'd pull all the "standard" path configuration out of the DLL and put it in |
|
In my embedded use-case I want to allow the use of the Python stdlib. I don't want to replicate what is in Modules/getpath.py to achieve that. |
|
If you installed Python as part of your app (which is what we call "embedding"), then you should know the directory already. It's going to be relative to your app, at a location you get to decide. Just hard code it in relative to your install dir. If you're using a separate install of Python, then you may as well just assume the locations relative to the install location, since you already have to assume a lot of other stuff about it to load and initialise the DLL directly. Including the entire runtime as part of your application is much safer. |
I am using the system Python instance on a Linux distribution and I don't know where the libpython.so resides, I just link against it using -lpython3.10 as per the python-3.10-embed.pc pkg-config file (which is pulled in by the configure script). Although prefix and exec_prefix are specified in the pkg-config file, but that starts along the path that is a reimplementation of getpath.py, which takes those values and calculates sys.path If I start python3, I get this sys.path without any environment variables: |
|
I know a lot less about embedding on Linux, but I recognise that embedding the whole runtime is more difficult than on Windows. You might be best off to just add the few calls to read |
This is an attempt to work around python/cpython#101471 until a more permanent solution is found. Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Bug report
I am embedding Python into a larger application (https://github.com/syslog-ng/syslog-ng)
As we were updating to Python 3.x we started using the new PyConfig based initialization, which worked nicely with Python 3.10 and is now broken with 3.11.1, it might be our fault, but I've now spent the better part of a day figuring out how things are intended to work. I am reading the documentation as well as reading the related source code.
When implementing our PyConfig based initialization, I choose to use the "isolated" mode, as I don't like Python interfering with our signal handlers and would like in general be as detached from the system as possible.
We are shipping our own "glue" modules written in Python and we want its location to be added to sys.path. We are also using a virtualenv at runtime, which we activate ourselves. Our glue modules and the virtualenv are in separate locations.
Glue modules: /usr/lib/syslog-ng/python/
Virtualenv: /var/lib/syslog-ng/python-venv/
This is how this is initialized:
_py_configure_virtualenv_python() will in turn initialize config.pythonpath_env and config.argv
The problem:
Starting with 3.11 the value of pythonpath_env is not picked up even if I set it explicitly. The reason is probably the reimplementation of Python PATH calculation from C to Python in Modules/getpath.py
It seems that the new implementation does not pick this value up if use_environment is FALSE. This probably worked earlier, as use_environment only controlled the getenv() call but since I am setting this explicitly, the earlier path calculation picked it up anyway.
But I can understand that now even the path calculation logic is guarded by the value of use_environment, so I tried set use_environment to TRUE in my _py_configure_interpreter() function. This did not help, probably because at one point "isolated" being TRUE causes use_environment to be reset to FALSE, even if I set it to TRUE in my original code.
As you can see, use_environment is FALSE, even though I set it to TRUE, originally.
As an alternative, I've switched to using "normal" config and reset most config values to match those of the isolated config.
This seems to work now, but I have a feeling this behaviour is not entirely intentional.
Your environment
I am doing all the above in a Debian testing based container:
The text was updated successfully, but these errors were encountered: