Can not link to virtual environment on OS X #1741
Labels
Comments
|
Hmmm, let's try to figure this out. First, can you please run and share here the output from the |
|
Thanks for the response and thank you for letting me know about this tool! I ran the print-config command twice. The first time outside of my virtual environment and the second time from within my virtual environment. Outside virtual environment Inside virtual environment It appears that the virtual environment is using the correct Python interpreter, but it still won't load any of my pip dependencies. Might this be because the libdir is not being updated? P.S. This is an amazing project and I'm really grateful for all of the hard work from yourself and other contributors. I would be interested in becoming involved with this project in the future. |
Hmm. As far as I know cc @messense - If I remember you have a Mac, perhaps you have some experience in this area?
Awesome, really glad to hear it! I think we're a friendly bunch and everyone is welcome to help out. All of the design ideas are floating around in Github issues, so feel free to take on anything that takes your fancy. Or propose new ideas, etc. 👍 |
|
Never tried running pyo3 code directly in virtualenv before. Here is a sysconfig dump of a venv in macOS
|
|
I think I am still unsure why the embedded venv does not get detected on macOS but it does on Linux. (This could potentially be worthy of an upstream CPython bug report? I doubt they would change it for backwards-compatibility.) You might be able to do what you need using the PEP 587 APIs manually before calling any PyO3 APIs: https://docs.python.org/3/c-api/init_config.html |
|
I have observed the same problem as highlighted above. if I run But if I run the similar code inside my pyo3: The output is I was previously running this same codebase on an intel Mac and did not see this issue (I can't easily re-run that at the moment (it was python3.8) |
|
I installed python3.8 as an experiment and retried this same rust pyo3 above and have same reply. So still not picking up the venv path for sys.path to allow it to pick up the available modules. |
|
Next step was to use the env var PYTHONPATH to set the paths to search (as an alternative to finding them via the venv). This seems to completely replace the sys.path rather than prepending it. So as a follow on step I set the all the relevant paths using the PYTHONPATH env var. This seems to work and I have bene able to load my python modules. I base this on how python is working: So it points me to the sys.path is not being initialised correctly. I was not able to easily locate where in the code this was set. But if you can point me in the right direction I am happy to take a look. |
|
I went through some news groups, etc with regards to python embedded and identified the following related articles: Both seem to be aligned in: How this is done is not trivial and seems to be in flux on python (and may take some time to resolve). So for perhaps the mean time using PYTHONPATH to set the path may be the best option. |
|
Thank you for the research and for sharing. I was aware it was complicated but was unaware that it's also in flux. Hopefully things will settle soon; I'd be glad to implement something in PyO3 to support this eventually if possible. |
|
Something like this could work? diff --git a/src/gil.rs b/src/gil.rs
index 37e0ed2495..397f9560b0 100644
--- a/src/gil.rs
+++ b/src/gil.rs
@@ -71,6 +71,28 @@ pub fn prepare_freethreaded_python() {
if ffi::Py_IsInitialized() == 0 {
ffi::Py_InitializeEx(0);
+ #[cfg(target_os = "macos")]
+ if let Ok(venv) = std::env::var("VIRTUAL_ENV") {
+ // Safety: the GIL is already held because of the Py_IntializeEx call.
+ let pool = GILPool::new();
+ let py = pool.python();
+
+ let version_info = py.version_info();
+ let sys = py.import("sys").unwrap();
+ let sys_path = sys.getattr("path").unwrap();
+ sys_path
+ .call_method1(
+ "append",
+ (format!(
+ "{}/lib/python{}.{}/site-packages",
+ venv, version_info.major, version_info.minor
+ ),),
+ )
+ .unwrap();
+
+ drop(pool);
+ }
+
// Release the GIL.
ffi::PyEval_SaveThread();
}
|
|
Perhaps, yes. Probably want to detect Conda environments in a similar way. (like we do in There's also ffi calls we could run before I fear that whatever logic we introduce may not work for everyone. I wonder if we can have |
Hiya, I'm experiencing the same issue using poetry. The "solution" is to append to the export PYTHONPATH=${PYTHONPATH}:/Users/naamanhirschfeld/Library/Caches/pypoetry/virtualenvs/starlite-s4i-xLL2-py3.10/lib/python3.10/site-packages/This is of course very ugly and quite brittle. Can I suggest that as a safeguard, the user will be able to specify the env or virtual env (depending if running in system or virtual env of course) as a parameter passed to |
This was referenced Jul 20, 2024
|
The solution proposed by @messense works great, but does not work if you have editable installed packages (aka packages installed with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🐛 Bug Reports
I can not link to the correct virtual environment in OS X. Instead, the system Python is always used.
🌍 Environment
brew install python@3.8A virtual environment was created withpython3 -m venv redvoxand is located at /Users/anthony/venvs/redvoxrustc --version): rustc 1.56.0-nightly (9c25eb7aa 2021-07-25)version = "0.x.y"withgit = "https://github.com/PyO3/pyo3")?:No
💥 Reproducing
source /Users/anthony/venvs/redvox/bin/activatepip install matplotlibcargo runThe following shows how I activate the virtual environment and attempt to call the rust code. The error message I receive is also included.
I've tried setting PYTHONHOME and PYTHONPATH. Neither of these have worked for me. The system Python is always used no matter what I try. How can I get pyo3 to work with my virtual environment on OS X?
The text was updated successfully, but these errors were encountered: