creating virutal environment using virtualenv is breaking with anaconda setup

[kp89]

My setup -

1. OS - Ubuntu 14.04
2. python - using miniconda
3. packages installed using conda package manager -

• pip
• virtualenv
• ipython

And I use virtualenv for development and everything is working perfectly fine with system's python version + pip + virtualenv.

Recently I switched to miniconda to avoid pain of installing numpy and scikit-learn packages manually which fails in compilation most of the time with pip.

But now virtualenv or something which is used by virtualenv is broken with this setup. I am sure that conda supports virtualenv and it should support if not supporting - my personal opinion. We adopted virtualenv since very long time and it makes development setup + deployment very very easy. And I am sure it is the case for many out there.

I am facing following error while creating virtualenv.

$ virtualenv foo
Using base prefix '/opt/mconda-py3'
New python executable in foo/bin/python3
Also creating executable in foo/bin/python
Installing setuptools, pip...
  Complete output from command /home/kppatel/envs/foo/bin/python3 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv_support/pip-6.1.1-py2.py3-none-any.whl/pip/__init__.py", line 15, in <module>
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv_support/pip-6.1.1-py2.py3-none-any.whl/pip/vcs/subversion.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv_support/pip-6.1.1-py2.py3-none-any.whl/pip/index.py", line 29, in <module>
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv_support/pip-6.1.1-py2.py3-none-any.whl/pip/req/__init__.py", line 3, in <module>
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv_support/pip-6.1.1-py2.py3-none-any.whl/pip/req/req_install.py", line 17, in <module>
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv_support/pip-6.1.1-py2.py3-none-any.whl/pip/_vendor/distlib/markers.py", line 20, in <module>
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv_support/pip-6.1.1-py2.py3-none-any.whl/pip/_vendor/distlib/markers.py", line 48, in Evaluator
  File "/opt/mconda-py3/lib/python3.4/platform.py", line 1319, in python_implementation
    return _sys_version()[0]
  File "/opt/mconda-py3/lib/python3.4/platform.py", line 1282, in _sys_version
    repr(sys_version))
ValueError: failed to parse CPython sys.version: '3.4.0 (default, Apr 11 2014, 13:08:40) \n[GCC 4.8.2]'
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "/opt/mconda-py3/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv.py", line 993, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv.py", line 961, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/opt/mconda-py3/lib/python3.4/site-packages/virtualenv.py", line 903, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/kppatel/envs/foo/bin/python3 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

After doing some research, I found some patch work which solves above problem. It is explained here

I believe there must be something better you guys can do to fix this problem - may be in next update.

And just FYI - I didn't face the same problem in Windows setup so this may be platform specific.

[patelkunal]

I am also facing same issue with conda.
However I installed pip using conda install pip and then use pip for virtualenv installation.

Not sure what is the problem.

[tswicegood]

Hey @patelkunal. This is a common point of confusion and something we're trying to clarify better. You don't need to use virtualenv with conda. Conda has environments baked right in. In place of creating the virtual environment, you create a conda environment. For example (using virtualenv-wrapper):

mkvirtualenv my-dev
workon my-dev

Becomes:

conda create -n my-dev python=3.4
source activate my-dev

Now you're in something that works just like a virtual environment. Pip is automatically installed along side Python, so you even have pip for managing packages that way if you want to.

[patelkunal]

I agree @tswicegood. It really helpful that we can do similar stuff with conda itself.

My next question would be, how can we export project dependencies to something called requirements.txt or other simple format.

I am sure that there must be something similar in conda as well.

My personal opinion - I like this approach because it becomes extremely easy to ship all dependencies in one text file. And it makes deployment easy. Another thing is virtualenv is managing everything in separate directory itself which makes you very confident about not messing up with system installation.

P.S. I have activated root environment (I guess that is default) and then installed all common dependencies like IPython, pip and virtualenv. Then just use virtualenv for all project development work.

[groutr]

Conda can export the environment info. If you want to export the environment foo then you can do that with:
conda env export -n foo -f requirements.txt

Conda environments are stored in the envs subdirectory. When you activate an environment, it simply prepends the appropriate directories to the system PATH. When you deactivate, it removes the environment from the path. Conda environments are designed to be independent of every other environment.

[Locky1138]

Dealing with this issue as well. The problem I am still dealing with, after exporting the conda config, is that we cannot create a conda virtual-environment in a separate folder.

we want to create a folder structure like this:
My Site/
--Source/ # site source in here
--database/
--static/
--virtualenv/ # python virtual environment in here

I'm fairly sure this is common practice, but i'm just following the Test-Driven Development with Python book (https://github.com/hjwp/book-example/tree/chapter_08), and conda is making it difficult to follow.

[mivade]

@tswicegood Many tools for testing (e.g., tox) rely on virtualenv for running their tests. This bug therefore completely breaks my unit testing workflow since it always gives an InvocationError.

[patelkunal]

Any updates on this?

It is still failing with current latest version. :(

[holdenk]

Were looking at using virtual env to enable consistent linting but in the current state it seems like we need a seperate path in out shell scripts just for conda users, it would be nice if virtualenv and conda played nice together - but alternatively some clear error messaging around this would be pretty helpful too.

[mingwandroid]

We released a new virtualenv package with patches to allow it to find the correct libpython. Please test and close this if it works for you.

[tijiang13]

My suggestion is that you can install virtualenv with Mac's built-in version of python.

Like that

     /usr/local/bin/pip3.5 install virtualenv

Then every time you call virtualenv, it's in fact calling the one that under the control of a different pip from conda's, and the advantages is that there is totally no overlap, since there won't have a virtualenv under the conda environment.

[mingwandroid]

The issue is that conda's virtualenv was broken, so clearly people reporting this issue want to use conda with virtualenv, so your advice of "don't use it" is somewhat redundant.

[artforlife]

I believe I am having the same issues. I installed Anaconda with Python3 and used IPython for awhile. Today I went to the regular Python and wanted to create a virtualenv for a Flask project.
Whenever I run python3 I get the following error:

dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /Users/name/anaconda3/bin/python
  Reason: image not found
Abort trap: 6

The same error show when I run conda. I am able to enter some of my virtual environments, but only the ones that contained Python 2.7. I value IPython and Anaconda, but it seems unacceptable that it breaks things in such a way. I am now at a point of wanting to wipe them out from my machine because I need to use the virtualenv.

If anyone has an idea of how one could fix this, please let me know.

[mingwandroid]

/Users/name/anaconda3/bin/python

Correct me if I'm wrong, but from what you've said and the above, it would appear that somehow a virtualenv created by your 'regular Python' broke Anaconda's Python.

Regardless, if you want to use virtualenv with conda and ignore your 'regular Python' altogether, you can use the latest virtualenv package in conda defaults.

[artforlife]

@mingwandroid I cannot run conda because I get the same error. Basically, I need to get Python3 working somehow for my regular projects. Any suggestions on how I could do that?

[mingwandroid]

Install miniconda3 over the top of your current installation.

…

On Mar 10, 2017 3:20 AM, "artforlife" ***@***.***> wrote: @mingwandroid <https://github.com/mingwandroid> I cannot run conda because I get the same error. Basically, I need to get Python3 working somehow for my regular projects. Any suggestions on how I could do that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#305 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_pdHmnUaxzgRzG2fCZ_0XBX0uL7bCOks5rkMGHgaJpZM4EAIam> .

[artforlife]

@mingwandroid I did this now and it seems to have fixed the problem in the sense that I now have two more Pythons (2.7 and 3.6) which came with Miniconda. Could you tell me why you thought this was going to work? What is Miniconda? Can I now wipe the Anaconda itself? Should I uninstall the original Pythons which I installed through brew install python? Lastly, why does the problem occur in the first place?

[mingwandroid]

Miniconda3 does not come with two Pythons, it comes with Python 3.6 only. Miniconda2 comes with Python 2.7.

Could you tell me why you thought this was going to work

This works (if done correctly) because you have a broken Python executable and this procedure will overwrite it with one that works without changing much else in your Anaconda installation.

What is Miniconda?

Please read https://conda.io/miniconda.html and/or https://conda.io/docs/install/quick.html#miniconda-quick-install-requirements

Can I now wipe the Anaconda itself?

No, because if you installed "miniconda3 over the top of your current installation" then wiping Anaconda would also entail wiping Miniconda3 as you will have installed them to the same location.

Lastly, why does the problem occur in the first place?

I have no idea what commands you issued and what state your environment is in so I cannot answer that. Perhaps you can look at your bash history with the history command and figure out what you did? I've never seen this before; my best guess on the information you've presented is, as I said earlier:

it would appear that somehow a virtualenv created by your 'regular Python' broke Anaconda's Python

[artforlife]

@mingwandroid Understood, thank you.

By the way, when I run which python I get /Users/name/miniconda3/bin/python
and when I run which python3 I get /Users/name/miniconda3/bin/python3
so it does appear that somehow miniconda came with both Pythons. Am I missing something?

Also, I believe that I installed Anaconda and Miniconda into different locations. I was not sure what you meant by "over the top of the existing one". That is, both of the directories exist now:

~/anaconda3
~/miniconda3

Does this mean that I could wipe the anaconda3? Am I correct to think that Miniconda is just a stripped, bare-bones version of Anaconda? Would IPython still work with the former?

[mingwandroid]

By the way, when I run which python I get /Users/name/miniconda3/bin/python
and when I run which python3 I get /Users/name/miniconda3/bin/python3
so it does appear that somehow miniconda came with both Pythons. Am I missing something?

Both of those executables are probably python3, try:

$ python --version
$ python3 --version

Also, I believe that I installed Anaconda and Miniconda into different locations. I was not sure what you meant by "over the top of the existing one". That is, both of the directories exist now:

You'd need to install more packages for IPython to work in your Miniconda installation. You can turn a Miniconda installation into an full Anaconda installation by running conda install anaconda.

Be careful about ~/bash_profile. Miniconda and/or Anaconda may have added PATH entries to that file, for whichever you decide to delete, you should remove its entries.

[artforlife]

@mingwandroid You were right. Both of those executables return 3.6 as their version.

In the spirit of cleaning up, if I wanted to turn Miniconda into a full-blown version of Anaconda, should I simply delete the anaconda3 directory from the user's home directory?

What about the ~/.bash_profile?

export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/mysql/bin

[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as$
export PATH="/usr/local/sbin:$PATH"
export PATH=~/.local/bin:$PATH

# added by Anaconda3 4.2.0 installer
export PATH="/Users/name/anaconda3/bin:$PATH"

# added by Miniconda3 4.3.11 installer
export PATH="/Users/name/miniconda3/bin:$PATH"

Is it only this line that needs to be stricken out?

# added by Anaconda3 4.2.0 installer
export PATH="/Users/name/anaconda3/bin:$PATH"

[mingwandroid]

Personally, I would remove all of that stuff. I never let programs add themselves to my PATH when not necessary.

For conda you should prefer to use activate and deactivate, as in:

source /Users/name/miniconda3/bin/activate

You can put this in ~/.bash_profile instead and that will properly activate your environment (which involves more than setting PATH).

Personally I wouldn't even do that and instead type that command in each terminal where I want to use conda, but I expect some would prefer no to do that.

The reason I don't want it in there is because there are times (e.g. when using homebrew) when I do not want conda programs to be run because homebrew will not work right if that happens.

[artforlife]

The reason I don't want it in there is because there are times (e.g. when using homebrew) when I do not want conda programs to be run because homebrew will not work right if that happens.

Could you give an example of this?

[mingwandroid]

I cannot give a specific example but in general, most programs (and esp. software distros like the Anaconda Distribution and Homebrew) are written with built-in assumptions (even if the authors didn't do that deliberately) about what other exact programs will be run when they run other programs (for example a shell script that calls 'python'). If these assumptions turn out to be incorrect (for example, ArchLinux expects 'python' to always be Python 3 and never Python 2) then things will fail, sometimes catastrophically.

[mingwandroid]

I call this 'environmental pollution'. On Windows a variant of this is called 'DLL hell': https://en.wikipedia.org/wiki/DLL_Hell

Avoiding problems boils down to not adding things globally to your system environment or to the environment of your command line interface to your system (which is what changing ~/.bash_profile does).

[artforlife]

Avoiding problems boils down to not adding things globally to your system environment or to your command line interface to your system.

In this particular case, are you sourcing the conda instead of having it in the PATH because you want to make sure that when python is run without conda having been sourced, it will run the default system python; however, when the conda has been sourced, the python used will be coming from within the miniconda directory?
That is, we try to make sure that Conda's Python is kept pristine. The same can be said for other packages. What's more, if I remove Miniconda from my bash_profile, I will be able to use brew install python(3) to have a separate version for my non-IPython and non-scientific projects. Am I understanding this correctly?

[mingwandroid]

Yes, but having conda in PATH was never good enough anyway. Activating conda (or activating a conda environment) does more than adding something to PATH:

1. It deactivates any currently activated env.
2. It runs each package's deactivate script in the env. it is deactivating.
3. It sets some env. vars for housekeeping purposes around activation and deactivation.
4. It runs each package's activate script in the env it is activating.

[qytz]

@mingwandroid This works for me, will these patches contribute to the virtualenv upstream?

We released a new virtualenv package with patches to allow it to find the correct libpython. Please test and close this if it works for you.

[mingwandroid]

@qytz: not until they are in a good enough state to survive an upstream review and until I have enough time to go through that process. That will not be for at least a month.

[qytz]

OK, thank you. Does that mean, when using conda, we should install package with conda first, use pip when the package does not exists in conda

[BFMarks]

+1

[honnibal]

@tswicegood I'd like to use the "normal" virtualenv instead of conda environments. Is there a reason why conda prevents me from doing that?

I tend to be using conda when I'm doing chores I don't enjoy, so I'm often fairly grumpy. At these times I often feel like conda has broken my workflow on purpose, just to force me into abandoning my other tools. I know that's probably not true, but still. Grump!

Why shouldn't python -m venv my-env work if python resolves to my miniconda distribution? Even though I'm constantly being told conda environments are better than virtualenv, I actually find them much much worse. Virtualenv doesn't rely on source: if you execute the python binary that's within the virtualenv, it'll do the right thing. Virtualenv also doesn't require me to centrally manage my environments. I can have a local directory isn't symlinked anywhere. To uninstall the environment, I delete the directory.

The other reason I don't enjoy using conda environments is that I think it's bad for the one command to have responsibility over the project environment, the Python versions, and package installation. To me it's much better to have pyenv, virtualenv and pip are separate tools. If I write conda remove, am I removing a package or an environment? If I add the --clobber flag, what do I clobber --- the package, the environment, the Python installation...?

I do think there are times where software authors should answer, "Have you tried not wanting to do that?" to their users. However, I think there's really good reason to want to mix anaconda's tooling with some of the tools it's trying to replace. I hope this can be supported better.

[mingwandroid]

python -m venv my-env works now. You need to run conda update python

FWIW, pip and conda do not work very well together. We will probably need to patch pip to make them work better together for our users. The pip installed by ensurepip (which is used by venv) is not currently the Anaconda Distribution's pip so if and when we do patch pip to work better with conda I will want to take steps to make ensurepip installs our patched pip instead.

Your objections to conda are very python-centric, ignoring that conda is used to manage other software. I recommend you read https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/

Why shouldn't python -m venv my-env work

It does work now (for some definitions of 'work').

Virtualenv doesn't rely on source: if you execute the python binary that's within the virtualenv, it'll do the right thing

This is a Python-centric world-view. If you were to create an environment containing the OpenJDK then activation of that environment needs to modify the calling shell environment so that JAVA_HOME is set correctly. Many other packages require some such activation, even some Python ones, and having the ability to modify the calling shell environment adds a lot of power to conda packages.

The fact is, virtualenv and pip can be used with the Anaconda Distribution but those projects do not take care not to break conda and we haven't yet done enough to research and fix those issues but the fact remains, if you are using Miniconda or Anaconda you are best off using conda for all of your fake virtualization, environment and package management needs. conda does not pretend to be virtualenv nor pip largely because it has different requirements and a broader scope than those tools.

[honnibal]

@mingwandroid Thanks!! I'll be sure to update and try that out.

I appreciate that the problem is on both sides, and compatibility across tools is non-trivial.

I'd actually read that blog post before. I think it distills my problem here fairly well. The answer to Myth 5 ("conda breaks my workflow"), boils down to "you should change your workflow". But the workflow conda's offering isn't better for me. It doesn't really help me to know about how well conda supports the many workflows I don't have.

Thanks again for your reply, and your efforts on the software. Out of respect for your time, I'll read your next reply if you make one, but won't make another myself.

[kalefranz]

Just make sure you conda install virtualenv now, and all issues should be resolved. The virtualenv you get from pip install is not compatible with conda environments.

[daimon99]

I encountered this problem too. My virtualenv creation even had took 4 minutes to complete. At last, I found that the pip.conf in ~/.pip/pip.conf or /etc/pip.conf, had a wrong extra-index-url. When I fixed it, All is ok now.

[pbuckley]

I got this error with a "regular" python virtualenv, created for python3 (specifically 3.6.1). Using homebrew "incidentally" upgraded me to python 3.7.3 (not sure what changes I made) and broke my virtualenv. Just a single symlink change fixed python for me, specifically ~/venv/p3virt/.Python -> /usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/Python switched to ~/.venv/p3virt/.Python -> /usr/local/Cellar/python3/3.7.3/Frameworks/Python.framework/Versions/3.7/Python. Still needed to pip install -r requirements.txt to get my libs reinstalled in the virtualenv.