Support virtual environment #578

tkf · 2018-09-21T07:04:03Z

This PR grew out of a small feature addition to a patch that includes a few more. Here is a new summary:

Main feature addition: Control Python environment at run-time by the environment variables PYCALL_JL_RUNTIME_PYTHON (and PYCALL_JL_RUNTIME_PYTHONHOME).
fix ImportError: No module named 'encodings' when using with virtualenv + python3 #410: Support building Pycall against a Python executable generated by python -m venv.
Refactoring: I merged how Py_SetPythonHome and Py_SetProgramName are ccalled with and without PYCALL_JL_RUNTIME_PYTHON. wpyprogramname and wPYTHONHOME are removed.

Original summary:

I suggest to let users control Python environment at runtime by (say) ENV["PYCALL_JL_RUNTIME_PYTHON"] (and ENV["PYCALL_JL_RUNTIME_PYTHONHOME"]). This can be used to make sure that Julia and Python environments for a project are fully reproducible. Combining Pkg3.jl and pipenv, a typical usage would be very simple:

$ ls
activate.jl  Manifest.toml  Pipfile  Pipfile.lock  Project.toml
$ cat activate.jl
import Pkg
Pkg.activate(@__DIR__)
ENV["PYCALL_JL_RUNTIME_PYTHON"] = strip(read(setenv(`pipenv --py`, dir=@__DIR__), String))
$ julia -i activate.jl
julia> using PyCall
julia> pyimport("numpy")  # loads numpy specified by Pipfile

Since this is useful outside PyCall testing, I included this functionality in python_cmd function instead of inlining it to the test.

tkf · 2018-10-21T22:09:44Z

I added some tests and they all pass on CIs now. I think it's good to go. (But please review.)

codecov-io · 2018-10-21T22:57:17Z

Codecov Report

Merging #578 into master will decrease coverage by 0.65%.
The diff coverage is 30.23%.

@@            Coverage Diff             @@
##           master     #578      +/-   ##
==========================================
- Coverage   53.11%   52.45%   -0.66%     
==========================================
  Files          19       19              
  Lines        1525     1567      +42     
==========================================
+ Hits          810      822      +12     
- Misses        715      745      +30

Impacted Files	Coverage Δ
src/pyinit.jl	`54.63% <30.23%> (-19.91%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e3f2ac...31ef039. Read the comment docs.

tkf · 2018-10-22T21:56:21Z

I'm trying to run the change 05877b9 on AppVeyor but it's not get triggered... (Tried re-open and force-push)

* `venv` support is now properly tested. This finds a bug in `pythonhome_of`: `sys.base_prefix` should be used instead of `sys.prefix` (same for `sys.base_exec_prefix`). * The path to Python executable passed to `pythonhome_of` was ignored when `python_cmd` was introduced. This is fixed now. * Use `python_cmd` in `find_libpython`. This properly sets `PYTHONIOENCODING`.

tkf · 2018-10-23T05:58:43Z

deps/depsutils.jl

+        if hasattr(sys, "base_exec_prefix"):
+            sys.stdout.write(sys.base_prefix)
+            sys.stdout.write(":")
+            sys.stdout.write(sys.base_exec_prefix)


It looks like we should be using sys.base_prefix and sys.base_exec_prefix instead of sys.prefix and sys.exec_prefix to support venv. With this change, this PR (accidentally) solves #410 as well. The reason is commented just below: 9dde9a8#diff-d16c3d4423fa5349bf01729c8beea4cdR90

tkf · 2018-11-04T18:56:12Z

@stevengj I think it's good to go, unless you want to bikeshed the environment variable names PYCALL_JL_RUNTIME_PYTHON and PYCALL_JL_RUNTIME_PYTHONHOME and/or there are some problems in the code.

Resolved conflicts due to: 45cbcee (drop 0.6 support)

as in 45cbcee

deps/depsutils.jl

src/PyCall.jl

stevengj · 2019-01-24T04:00:39Z

Should it be documented in the README?

tkf · 2019-01-24T04:35:00Z

Sure, I added a section in README.

preview: https://github.com/tkf/PyCall.jl/tree/venv#python-virtual-environment

stevengj · 2019-01-24T15:25:58Z

deps/depsutils.jl

+# Fix the environment for running `python`, and setts IO encoding to UTF-8.
+# If cmd is the Conda python, then additionally removes all PYTHON* and
+# CONDA* environment variables.
+function pythonenv(cmd::Cmd)


Isn't it sufficient to just run python -E to ignore PYTHON*? Why do we need to remove CONDA* since we are not running conda?

Oh, I guess we don't want to ignore these variables for non-Conda Python.

pythonenv is just moved from build.jl to depsutils.jl to make it usable at runtime.

stevengj · 2019-01-24T15:53:02Z

Why doesn't it work for Conda environments?

tkf · 2019-01-24T20:50:04Z

Each conda environment has its own copy of libpython (or rather a hard link). We can't change libpython path at run-time so it's not possible to support it. But maybe it's possible to do it with always_softlink option in conda. I haven't checked it.

…ctions

tkf · 2019-02-07T00:27:17Z

Thaaaaanks!!! 🎉

lassepe · 2021-08-30T08:38:58Z

@tkf Do you know whether this would work with poetry? I'm running into the same issues about incompatible libpython as you pointed out for conda. I guess that is also due to the fact that poetry copies the lib?

Edit: I was able to fix it. In case someone is hitting a similar issue: The poetry environment that I setup was using a python binary build via python-build and by default those binaries don't come with a shared object. I had to explicitly build with --enable-shared (see this link)

tkf force-pushed the venv branch from 14359fd to 9b00c8f Compare September 24, 2018 01:21

tkf mentioned this pull request Oct 12, 2018

Sequester mutable state outside of package directories JuliaLang/Pkg.jl#796

Closed

Support virtual environment

69f08db

tkf force-pushed the venv branch from 9b00c8f to 69f08db Compare October 13, 2018 07:16

tkf force-pushed the venv branch 4 times, most recently from f60bae3 to df1ef06 Compare October 21, 2018 17:51

tkf added 2 commits October 21, 2018 10:51

Add test_venv.jl

66eb069

Add python3.4-venv in Travis

0f80a51

tkf force-pushed the venv branch from df1ef06 to 0f80a51 Compare October 21, 2018 17:54

tkf added 2 commits October 21, 2018 11:04

Fix pythonhome_of for Julia 0.6

f9f53e6

Properly treat venv path in Windows tests

9be4dd7

Since this is useful outside PyCall testing, I included this functionality in python_cmd function instead of inlining it to the test.

tkf force-pushed the venv branch from 45ac46d to 9be4dd7 Compare October 21, 2018 19:06

Mark venv test broken in Windows

31ef039

tkf closed this Oct 22, 2018

tkf reopened this Oct 22, 2018

tkf force-pushed the venv branch from 26e351f to 05877b9 Compare October 22, 2018 21:50

tkf force-pushed the venv branch from 05877b9 to 31ef039 Compare October 23, 2018 00:36

tkf force-pushed the venv branch from 4320881 to 131369c Compare October 23, 2018 03:39

tkf added 3 commits October 22, 2018 21:22

Test with virtualenv command as well

7866c93

Specify Python version in virtualenv activation test

d16a4d4

Move pythonhome_of to depsutils.jl and use it during build

9dde9a8

tkf commented Oct 23, 2018

View reviewed changes

Support "venv activation" test inside virtualenv

7bb0d13

tkf closed this Oct 23, 2018

tkf added 3 commits November 1, 2018 23:32

reinterpret source array instead of dest

6d47896

Improve docs and comments

6699140

Simplify __init__() by merging venv code path

5e3d621

tkf mentioned this pull request Nov 9, 2018

WIP: Use python binary instead of libpython when it's a PIE #614

Open

tkf added 2 commits November 13, 2018 12:42

Merge remote-tracking branch 'origin/master' into venv

b517aa3

Merge remote-tracking branch 'origin/master' into venv

cc96a4b

tkf mentioned this pull request Dec 19, 2018

Segfault with Julia MKL build (library conflict) #443

Open

tkf added 3 commits January 23, 2019 19:18

Merge remote-tracking branch 'origin/master' into venv

e9a6c52

Resolved conflicts due to: 45cbcee (drop 0.6 support)

Stop using Compat in test_venv.jl

c56ec24

as in 45cbcee

Use property-based syntax in test_venv.jl

99ec542

stevengj reviewed Jan 24, 2019

View reviewed changes

deps/depsutils.jl Show resolved Hide resolved

stevengj reviewed Jan 24, 2019

View reviewed changes

src/PyCall.jl Show resolved Hide resolved

tkf added 2 commits January 23, 2019 20:30

Add docs on virtual environment usage

923feb9

Merge remote-tracking branch 'origin/master' into venv

21f08b3

stevengj reviewed Jan 24, 2019

View reviewed changes

tkf mentioned this pull request Feb 1, 2019

Test find_libpython #595

Merged

1 task

tkf and others added 3 commits February 4, 2019 18:07

Remove unnecessary VERSION < v"0.7.0-" branch

44beb7c

grammar fixes; rm references to internal details about Python API fun…

6a20a5f

…ctions

inline links

5d50878

stevengj merged commit 7e1c835 into JuliaPy:master Feb 6, 2019

tkf deleted the venv branch February 7, 2019 00:27

tkf mentioned this pull request Oct 17, 2019

Vendoring Python Wheels as Artifacts JuliaLang/Pkg.jl#1439

Open

yuvipanda mentioned this pull request Jun 21, 2021

Only set PYTHONHOME when using conda #411

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support virtual environment #578

Support virtual environment #578

tkf commented Sep 21, 2018 •

edited

tkf commented Oct 21, 2018 •

edited

codecov-io commented Oct 21, 2018 •

edited

tkf commented Oct 22, 2018

tkf Oct 23, 2018 •

edited

tkf commented Nov 4, 2018

stevengj commented Jan 24, 2019

tkf commented Jan 24, 2019

stevengj Jan 24, 2019

stevengj Jan 24, 2019

tkf Jan 24, 2019

stevengj commented Jan 24, 2019

tkf commented Jan 24, 2019

tkf commented Feb 7, 2019

lassepe commented Aug 30, 2021 •

edited

Support virtual environment #578

Support virtual environment #578

Conversation

tkf commented Sep 21, 2018 • edited

tkf commented Oct 21, 2018 • edited

codecov-io commented Oct 21, 2018 • edited

Codecov Report

tkf commented Oct 22, 2018

tkf Oct 23, 2018 • edited

Choose a reason for hiding this comment

tkf commented Nov 4, 2018

stevengj commented Jan 24, 2019

tkf commented Jan 24, 2019

stevengj Jan 24, 2019

Choose a reason for hiding this comment

stevengj Jan 24, 2019

Choose a reason for hiding this comment

tkf Jan 24, 2019

Choose a reason for hiding this comment

stevengj commented Jan 24, 2019

tkf commented Jan 24, 2019

tkf commented Feb 7, 2019

lassepe commented Aug 30, 2021 • edited

tkf commented Sep 21, 2018 •

edited

tkf commented Oct 21, 2018 •

edited

codecov-io commented Oct 21, 2018 •

edited

tkf Oct 23, 2018 •

edited

lassepe commented Aug 30, 2021 •

edited