Broken virtualenv creation

[mikicz]

I've got a problem with creating a virtualenv in travis. Or, more specifically, how the created virtualenv behaves after it's created.

python -m venv /home/travis/build/mikicz/travis-venv-test 

runs without error and doesn't print anything. A virtualenv is actually created, however it's almost empty. When you run venv creation from normal python installation or correctly working virtualenvs, pip and setuptools are usually installed.

However the actual problem is with installing packages to the created virtualenv:

/home/travis/build/mikicz/travis-venv-test/bin/python -m pip install arca

outputs the expected result - it downloads the package and installs it, however, it's not actually installed in the virtualenv, it's installed in the python the virtualenv was created from. You can see that when you run python -m pip freeze.

That is not standard behaviour.

I made an example travis script for this at https://github.com/mikicz/travis-venv-test/blob/af47e2f5143fab04054a9f75fbacaec9dbc4c404/.travis.yml (build https://travis-ci.org/mikicz/travis-venv-test/builds/287923885)

As a small sidenote, I checked the travis python installation and found that there is a python installation in /opt/python which requires sudo: required to be set for it to be accessed. When I tried creating a virtualenv from that python, it worked as expected (https://github.com/mikicz/travis-venv-test/blob/d5c589893ec7047b8eef5dbedd4f85da39b342c9/.travis.yml, build https://travis-ci.org/mikicz/travis-venv-test/builds/287923321). However I don't like this solution because it says in the documentation to not rely on this python, I don't actually need the sudo permissions and I don't want the extra boot time.

Am I doing something wrong? Is there a way to create a virtualenv without using the /opt/ python and the sudo permissions? I am targeting Python 3.6 and higher.

[achimnol]

I've faced the same problem when I tried to perform an integration test of my project with another my project which is supposed to be installed in a separate virtualenv at ~/virtualenv/anotherproject in Travis CI.
Even after source ~/virtualenv/anotherproject/bin/activate, the results of pip --version and python -c 'import sys; print(sys.prefix)' show that it still stays in the original virtualenv created by Travis.
FYI, things get worse if I don't wrap my commands into a separate shell script — pip comes from the system's Python 2.7 installation regardless of the python setting in travis.yml.

[achimnol]

I have worked around this by using a prebuilt binary for Debian from the Docker library, and changing the config to language: generic.
Downloading "python:3.6" image does not take that long — the below step only takes about 25-30 seconds.

- stage: test
  before_install:
    - |
      docker create --name pybin python:3.6
      sudo docker cp "pybin:/usr/local/bin" /usr/local
      sudo docker cp "pybin:/usr/local/lib/python3.6" /usr/local/lib/python3.6
      sudo docker cp "pybin:/usr/local/lib/libpython3.6m.so" /usr/local/lib
      sudo docker cp "pybin:/usr/local/lib/libpython3.6m.so.1.0" /usr/local/lib
      sudo docker cp "pybin:/usr/local/lib/libpython3.so" /usr/local/lib
      sudo docker cp "pybin:/usr/local/include/python3.6m" /usr/local/include
      docker rm pybin
      sudo ldconfig
      rm -rf ~/virtualenv/python*
      /usr/local/bin/python -c 'import sys; print(sys.prefix)'
      /usr/local/bin/python -m venv ~/virtualenv/python3.6
      source ~/virtualenv/python3.6/bin/activate
      python -c 'import sys; print(sys.prefix)'

With the above extra steps, all Python virtualenv functionalities work as expected.

[achimnol]

Maybe, Travis CI could also utilize the same method by mirroring the docker images in their infrastructure.
However, this approach may break subtle things if the user program rely on apt-installed C/C++-based binary Python packages, but I think it should be okay for most "modern" Python applications since nowadays many popular binary packages are distributed as "manylinux" binary wheels which work great for both Debian and Ubuntu.

[jedie]

My work-a-round is to use /opt/python/3.6/bin/python and create a new virtualenv with this interpreter, see:

https://github.com/jedie/bootstrap_env/blob/595b904a475e2576bfe10ad7ba825d66fab1acaf/.travis.yml#L45-L48

Example build: https://travis-ci.org/jedie/bootstrap_env/builds/352272422

But you can only test against Python v3.6 :(

[btotharye]

was this ever fixed or any other supported way of handling projects that create/require a separate virtualenv? I think I'm having the same issue on my travis setup: https://github.com/btotharye/mycroft-homeassistant/blob/master/.travis.yml

[achimnol]

Be aware — my workaround using docker binaries began to break due to OpenSSL incompatibilities...
I'm going to change the way...

[cs01]

We're also running into this with pipx.

I made the logs verbose in my travis build, so I can see that pip is definitely not in there. pip's documentation says

pip is already installed if you are using Python 2 >=2.7.9 or Python 3 >=3.4 downloaded from python.org or if you are working in a Virtual Environment created by virtualenv or pyvenv. Just make sure to upgrade pip.

1.31s$ python -m tests.run_tests
Installing pipx
virtualenv location is /tmp/tmp8igw3w2y/pipxhome/pipx
running /home/travis/virtualenv/python3.6.3/bin/python -m venv /tmp/tmp8igw3w2y/pipxhome/pipx
/tmp/tmp8igw3w2y/pipxhome/pipx/bin/pip not found. Contents of /tmp/tmp8igw3w2y/pipxhome/pipx/bin:
/tmp/tmp8igw3w2y/pipxhome/pipx/bin/activate.csh
/tmp/tmp8igw3w2y/pipxhome/pipx/bin/python3
/tmp/tmp8igw3w2y/pipxhome/pipx/bin/python
/tmp/tmp8igw3w2y/pipxhome/pipx/bin/activate
/tmp/tmp8igw3w2y/pipxhome/pipx/bin/activate.fish
pip binary not found in virtual environment

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue in 7 days. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please respond before the issue is closed, or open a new one after. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[A-312]

Auto close...

Do you find a solution ?

[orsinium]

@A-312, the solution described by @jedie works to me. Just use /opt/python/3.6/bin/python. For example:

before_install:
  - curl https://raw.githubusercontent.com/dephell/dephell/master/install.py | /opt/python/3.6/bin/python

[cclauss]

That workaround works on dist: trusty but by default, Travis CI dist: xenial looks like this:
$ ls /opt/python/ # --> 2.7.15 3.6.7 3.7 3.7.1

[asottile]

(copied from twitter)

to make venv-in-virtualenv work reliably you need to find the original python. here's code from pre-commit which does this: https://github.com/pre-commit/pre-commit/blob/183c8cbb3a74ca241d524f82f6cc482f8b361b8f/pre_commit/languages/python_venv.py#L22-L48

also note on macos you need to do some wacky things to avoid pollution from parent pythons (due to being a "framework" build) -- here's the code for that too: https://github.com/pre-commit/pre-commit/blob/183c8cbb3a74ca241d524f82f6cc482f8b361b8f/pre_commit/main.py#L32-L36

(note the macos bit is only necessary if you're using python to invoke virtualenv / pip / etc.)