This repository has been archived by the owner. It is now read-only.

VirtualEnv error with Windows + Vagrant #2327

Closed
clone1018 opened this Issue Apr 26, 2014 · 27 comments

Comments

Projects
None yet
6 participants
@clone1018
Contributor

clone1018 commented Apr 26, 2014

No idea where to even start with this one. Fresh clone with fresh vagrant box.

Here's the weird part, at home my Windows 8.1 x64 computer experiences this problem but my Windows 7 x64 computer at work doesn't. This also happens to @abnor

vagrant@precise64:~/www.gittip.com$ make clean env
rm -rf env *.egg *.egg-info
find . -name \*.pyc -delete
"/usr/bin/python2.7"  "./vendor/virtualenv-1.9.1.py"\
                                --unzip-setuptools \
                                --prompt="[gittip] " \
                                --never-download \
                                --extra-search-dir=./vendor/ \
                                --distribute \
                                ./env/
Traceback (most recent call last):
  File "./vendor/virtualenv-1.9.1.py", line 2577, in <module>
    main()
  File "./vendor/virtualenv-1.9.1.py", line 979, in main
    no_pip=options.no_pip)
  File "./vendor/virtualenv-1.9.1.py", line 1081, in create_environment
    site_packages=site_packages, clear=clear))
  File "./vendor/virtualenv-1.9.1.py", line 1275, in install_python
    copyfile(join(stdlib_dir, fn), join(lib_dir, fn))
  File "./vendor/virtualenv-1.9.1.py", line 437, in copyfile
    copyfileordir(src, dest)
  File "./vendor/virtualenv-1.9.1.py", line 412, in copyfileordir
    shutil.copytree(src, dest, True)
  File "/usr/lib/python2.7/shutil.py", line 206, in copytree
    raise Error, errors
shutil.Error: [('/usr/lib/python2.7/config/libpython2.7.so', './env/lib/python2.
7/config/libpython2.7.so', '[Errno 71] Protocol error')]
make: *** [env] Error 1

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

@patcon

This comment has been minimized.

Show comment
Hide comment
@patcon
Contributor

patcon commented Apr 26, 2014

@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre Apr 29, 2014

Contributor

Blech. I looked at this thinking it might be a simple Python thing but [Errno 71] Protocol error looks more virtualization-related.

Contributor

chadwhitacre commented Apr 29, 2014

Blech. I looked at this thinking it might be a simple Python thing but [Errno 71] Protocol error looks more virtualization-related.

@chadwhitacre chadwhitacre added the TeamX label May 8, 2014

@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre May 8, 2014

Contributor

TeamX

Contributor

chadwhitacre commented May 8, 2014

TeamX

@patcon

This comment has been minimized.

Show comment
Hide comment
@patcon

patcon May 9, 2014

Contributor

Let's try this?
pypa/virtualenv#409

Can someone on a failing windows version give the --always-copy flag a shot?

Contributor

patcon commented May 9, 2014

Let's try this?
pypa/virtualenv#409

Can someone on a failing windows version give the --always-copy flag a shot?

@lukaszbanasiak

This comment has been minimized.

Show comment
Hide comment
@lukaszbanasiak

lukaszbanasiak Sep 10, 2014

@patcon

For me it works. Without --always-copy flag:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ virtualenv --no-site-packages env
New python executable in /vagrant/env/bin/python
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 985, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1405, in install_python
    os.symlink(py_executable_base, full_pth)
OSError: [Errno 71] Protocol error

With --always-copy flag:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ virtualenv --no-site-packages env --always-copy
New python executable in env/bin/python
Installing setuptools, pip...done.

Windows 7 64-bit, vagrant (Ubuntu 14.04 64-bit)

lukaszbanasiak commented Sep 10, 2014

@patcon

For me it works. Without --always-copy flag:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ virtualenv --no-site-packages env
New python executable in /vagrant/env/bin/python
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 985, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1405, in install_python
    os.symlink(py_executable_base, full_pth)
OSError: [Errno 71] Protocol error

With --always-copy flag:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ virtualenv --no-site-packages env --always-copy
New python executable in env/bin/python
Installing setuptools, pip...done.

Windows 7 64-bit, vagrant (Ubuntu 14.04 64-bit)

@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre Sep 10, 2014

Contributor

Thanks @lukaszbanasiak! So where do we need to add --always-copy? To the Makefile? Want to give us a PR for that?

Contributor

chadwhitacre commented Sep 10, 2014

Thanks @lukaszbanasiak! So where do we need to add --always-copy? To the Makefile? Want to give us a PR for that?

@techtonik techtonik self-assigned this Oct 29, 2014

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

Just run into this. I think I'll be able to tackle this. The problem is likely related to permissions for files mounted as VirtualBox share. Git also chokes on them, because all files are executable, permissions are immutable etc.

Contributor

techtonik commented Oct 29, 2014

Just run into this. I think I'll be able to tackle this. The problem is likely related to permissions for files mounted as VirtualBox share. Git also chokes on them, because all files are executable, permissions are immutable etc.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

Tracing the chain so far. This is invoked from /vagrant directory.

[vagrant provision] -> [scripts/vagrant-setup.sh] -> [make env] ->
  -> [virtualenv-1.9.1.py ...]
Contributor

techtonik commented Oct 29, 2014

Tracing the chain so far. This is invoked from /vagrant directory.

[vagrant provision] -> [scripts/vagrant-setup.sh] -> [make env] ->
  -> [virtualenv-1.9.1.py ...]
@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

virtualenv-1.9.1 calls failing copy tree two times until fault:

shutil.copytree(src, dst, symlinks=False):

==> default: Copying /usr/lib/python2.7/lib-dynload ./env/lib/python2.7/lib-dynload
==> default: Copying /usr/lib/python2.7/config ./env/lib/python2.7/config
Contributor

techtonik commented Oct 29, 2014

virtualenv-1.9.1 calls failing copy tree two times until fault:

shutil.copytree(src, dst, symlinks=False):

==> default: Copying /usr/lib/python2.7/lib-dynload ./env/lib/python2.7/lib-dynload
==> default: Copying /usr/lib/python2.7/config ./env/lib/python2.7/config
@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

/usr/lib/python2.7/config appears to be a directory:

drwxr-xr-x  2 root root     4096 Apr 14  2014 .
drwxr-xr-x 26 root root    24576 Apr 14  2014 ..
...
-rw-r--r--  1 root root 18965158 Feb 27  2014 libpython2.7-pic.a
lrwxrwxrwx  1 root root       23 Feb 27  2014 libpython2.7.so -> ../../libpython2.7.so.1
...

This is a symlink that can not be symlinked to /vagrant share, because it is mounted from Windows which doesn't support this.

Contributor

techtonik commented Oct 29, 2014

/usr/lib/python2.7/config appears to be a directory:

drwxr-xr-x  2 root root     4096 Apr 14  2014 .
drwxr-xr-x 26 root root    24576 Apr 14  2014 ..
...
-rw-r--r--  1 root root 18965158 Feb 27  2014 libpython2.7-pic.a
lrwxrwxrwx  1 root root       23 Feb 27  2014 libpython2.7.so -> ../../libpython2.7.so.1
...

This is a symlink that can not be symlinked to /vagrant share, because it is mounted from Windows which doesn't support this.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

virtualenv 1.11.6 doesn't fix this. --always-copy was added in 1.10, so we need to upgrade virtualenv anyway.

Contributor

techtonik commented Oct 29, 2014

virtualenv 1.11.6 doesn't fix this. --always-copy was added in 1.10, so we need to upgrade virtualenv anyway.

@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre Oct 29, 2014

Contributor

@techtonik If you want help ping @joeyespo. He's into Vagrant + Windows.

Contributor

chadwhitacre commented Oct 29, 2014

@techtonik If you want help ping @joeyespo. He's into Vagrant + Windows.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

Don't need it yet, but I'll keep that in mind. Thanks. The only help needed right now is to merge some virtualenv pull requests.

Contributor

techtonik commented Oct 29, 2014

Don't need it yet, but I'll keep that in mind. Thanks. The only help needed right now is to merge some virtualenv pull requests.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

Filled (pypa/virtualenv#663). Going to add --always-copy for now and upgrade virtualenv to 1.11.6 to see how it goes.

Contributor

techtonik commented Oct 29, 2014

Filled (pypa/virtualenv#663). Going to add --always-copy for now and upgrade virtualenv to 1.11.6 to see how it goes.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

Didn't get far. Hit hashicorp/vagrant#4745 on the way. Shot myself in the foot with this one, actually.

Contributor

techtonik commented Oct 29, 2014

Didn't get far. Hit hashicorp/vagrant#4745 on the way. Shot myself in the foot with this one, actually.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

Next fail with virtualenv 1.11.6:

...
==> default: Cannot find a wheel for setuptools
==> default: Cannot find a wheel for pip
==> default: Installing setuptools, pip...
==> default:   Complete output from command /vagrant/env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
==> default:   Traceback (most recent call last):
==> default:   File "<string>", line 1, in <module>
==> default: ImportError: No module named pip
==> default: ----------------------------------------
==> default: ...Installing setuptools, pip...done.
==> default: Traceback (most recent call last):
==> default:   File "./vendor/virtualenv.py", line 2338, in <module>
==> default:     main()
==> default:   File "./vendor/virtualenv.py", line 824, in main
==> default:     symlink=options.symlink)
==> default:   File "./vendor/virtualenv.py", line 992, in create_environment
==> default:     install_wheel(to_install, py_executable, search_dirs)
==> default:   File "./vendor/virtualenv.py", line 960, in install_wheel
==> default:     'PIP_NO_INDEX': '1'
==> default:   File "./vendor/virtualenv.py", line 902, in call_subprocess
==> default:     % (cmd_desc, proc.returncode))
==> default: OSError: Command /vagrant/env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1
==> default: make: *** [env] Error 1

Looks like pip upgrade is needed.

Contributor

techtonik commented Oct 29, 2014

Next fail with virtualenv 1.11.6:

...
==> default: Cannot find a wheel for setuptools
==> default: Cannot find a wheel for pip
==> default: Installing setuptools, pip...
==> default:   Complete output from command /vagrant/env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
==> default:   Traceback (most recent call last):
==> default:   File "<string>", line 1, in <module>
==> default: ImportError: No module named pip
==> default: ----------------------------------------
==> default: ...Installing setuptools, pip...done.
==> default: Traceback (most recent call last):
==> default:   File "./vendor/virtualenv.py", line 2338, in <module>
==> default:     main()
==> default:   File "./vendor/virtualenv.py", line 824, in main
==> default:     symlink=options.symlink)
==> default:   File "./vendor/virtualenv.py", line 992, in create_environment
==> default:     install_wheel(to_install, py_executable, search_dirs)
==> default:   File "./vendor/virtualenv.py", line 960, in install_wheel
==> default:     'PIP_NO_INDEX': '1'
==> default:   File "./vendor/virtualenv.py", line 902, in call_subprocess
==> default:     % (cmd_desc, proc.returncode))
==> default: OSError: Command /vagrant/env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1
==> default: make: *** [env] Error 1

Looks like pip upgrade is needed.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

Downloading pip and setuptools wheels helped. Going to commit later.

Contributor

techtonik commented Oct 29, 2014

Downloading pip and setuptools wheels helped. Going to commit later.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 29, 2014

Contributor

Well, another problem:

...
  File "distribute_setup.py", line 129, in _do_download
    _build_egg(egg, tarball, to_dir)
  File "distribute_setup.py", line 120, in _build_egg
    raise IOError('Could not build the egg.')
IOError: Could not build the egg.
Contributor

techtonik commented Oct 29, 2014

Well, another problem:

...
  File "distribute_setup.py", line 129, in _do_download
    _build_egg(egg, tarball, to_dir)
  File "distribute_setup.py", line 120, in _build_egg
    raise IOError('Could not build the egg.')
IOError: Could not build the egg.
@joeyespo

This comment has been minimized.

Show comment
Hide comment
@joeyespo

joeyespo Oct 29, 2014

Contributor

RE: Vagrant + Windows

Honestly, even after fixing the quirks, working within a Vagrant environment hasn't been a pleasant dev experience. It's terribly slow. And getting set up is still quite tedious (which is the whole point of Vagrant). This has kind of pushed me away from participating. I don't feel productive because of how much time it takes to get from "idea I want to test" or "issue I want to resolve" to "localhost".

Instead, I'd much rather work within a "native" Python and JavaScript dev environment since these technologies are already cross-platform. Striving for a single, cross-platform run step (and ideally, a single, cross-platform install step too) is a much better goal than "let's fix Vagrant."

I've applied this approach to other websites (both monolithic and microservices), and it's made both onboarding and day-to-day development much smoother. Contributors can continue to use the tools and the environments that make them maximally productive. And there's no time wasted with installation hand-holding. Requiring each machine to be exactly the same in order to make high-level changes shuts out a lot of people. (As for the dev/prod-parity as a bug-catching mechanism, which usually comes up, auto-running tests in a prod-like CI tool before merging PRs has been equivalently effective.) I'd be happy to talk more about the approaches I've used to make webapps more cross-platform if there's any interest here, but unfortunately, I don't think I'll be much help with Vagrant issues anymore.

Contributor

joeyespo commented Oct 29, 2014

RE: Vagrant + Windows

Honestly, even after fixing the quirks, working within a Vagrant environment hasn't been a pleasant dev experience. It's terribly slow. And getting set up is still quite tedious (which is the whole point of Vagrant). This has kind of pushed me away from participating. I don't feel productive because of how much time it takes to get from "idea I want to test" or "issue I want to resolve" to "localhost".

Instead, I'd much rather work within a "native" Python and JavaScript dev environment since these technologies are already cross-platform. Striving for a single, cross-platform run step (and ideally, a single, cross-platform install step too) is a much better goal than "let's fix Vagrant."

I've applied this approach to other websites (both monolithic and microservices), and it's made both onboarding and day-to-day development much smoother. Contributors can continue to use the tools and the environments that make them maximally productive. And there's no time wasted with installation hand-holding. Requiring each machine to be exactly the same in order to make high-level changes shuts out a lot of people. (As for the dev/prod-parity as a bug-catching mechanism, which usually comes up, auto-running tests in a prod-like CI tool before merging PRs has been equivalently effective.) I'd be happy to talk more about the approaches I've used to make webapps more cross-platform if there's any interest here, but unfortunately, I don't think I'll be much help with Vagrant issues anymore.

@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre Oct 29, 2014

Contributor

Thanks for checking in, @joeyespo! So are you saying Vagrant in general is a buzzkill for you lately, or Vagrant as we've implemented it? I think I'm hearing you say the former.

Contributor

chadwhitacre commented Oct 29, 2014

Thanks for checking in, @joeyespo! So are you saying Vagrant in general is a buzzkill for you lately, or Vagrant as we've implemented it? I think I'm hearing you say the former.

@joeyespo

This comment has been minimized.

Show comment
Hide comment
@joeyespo

joeyespo Oct 30, 2014

Contributor

@whit537 Sure! Glad things are going strong. I also really like the rename, it's catchy :-)

Yeah, the former.

Contributor

joeyespo commented Oct 30, 2014

@whit537 Sure! Glad things are going strong. I also really like the rename, it's catchy :-)

Yeah, the former.

@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre Oct 30, 2014

Contributor

@joeyespo So what's your secret formula? Let me guess: use SQLite? ;-)

Contributor

chadwhitacre commented Oct 30, 2014

@joeyespo So what's your secret formula? Let me guess: use SQLite? ;-)

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 30, 2014

Contributor

IOError: Could not build the egg. is caused by old distribute conflicting with new setuptools.
https://bitbucket.org/pypa/setuptools/issue/18/packages-that-use_setuptools-from-older

Contributor

techtonik commented Oct 30, 2014

IOError: Could not build the egg. is caused by old distribute conflicting with new setuptools.
https://bitbucket.org/pypa/setuptools/issue/18/packages-that-use_setuptools-from-older

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Nov 4, 2014

Contributor

This one is blocked by #2886 through #2867 ATM..

Contributor

techtonik commented Nov 4, 2014

This one is blocked by #2886 through #2867 ATM..

@joeyespo

This comment has been minimized.

Show comment
Hide comment
@joeyespo

joeyespo Nov 12, 2014

Contributor

@whit537 Haha, yeah, that's one option. A decent one too, since it's built-in to Python and is supported by popular ORMs.

The general idea is use cross-platform auto-installable components for dev, and production-optimized components for prod, staging, and tests running on CI.

Since Python/JS packages are already auto-installable and cross-platform, it's a natural fit to use them in place of the prod-ready counterparts. Of course, not every prod-ready component has a pure-Python implementation. In those cases, you have to substitute higher up. ORMs are a good option because the ability to plug in different backends is already built for you.

As for postgres.py, I'm not sure what the best approach would be. It highly couples your app to Postgres instead of to an object model. But I'm willing to talk through some ideas. Let's move the discussion to a new issue.

Contributor

joeyespo commented Nov 12, 2014

@whit537 Haha, yeah, that's one option. A decent one too, since it's built-in to Python and is supported by popular ORMs.

The general idea is use cross-platform auto-installable components for dev, and production-optimized components for prod, staging, and tests running on CI.

Since Python/JS packages are already auto-installable and cross-platform, it's a natural fit to use them in place of the prod-ready counterparts. Of course, not every prod-ready component has a pure-Python implementation. In those cases, you have to substitute higher up. ORMs are a good option because the ability to plug in different backends is already built for you.

As for postgres.py, I'm not sure what the best approach would be. It highly couples your app to Postgres instead of to an object model. But I'm willing to talk through some ideas. Let's move the discussion to a new issue.

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Nov 18, 2014

Contributor

Bootstrap fails with DB error:

==> basebox: env/bin/honcho -e defaults.env,local.env run env/bin/fake_data fake_data
==> basebox: Traceback (most recent call last):
==> basebox:   File "env/bin/fake_data", line 9, in <module>
==> basebox:     load_entry_point('gratipay==1736.post27', 'console_scripts', 'fake_data')()
...
==> basebox: psycopg2.IntegrityError: new row for relation "transfers" violates check constraint "positive"
==> basebox: DETAIL:  Failing row contains (578395790, 2014-08-20 12:00:37+00, Dominicbdp, Katharina92q, 0.00, tip).
==> basebox: make:
==> basebox: *** [data] Error 1

Moved to #2950

Contributor

techtonik commented Nov 18, 2014

Bootstrap fails with DB error:

==> basebox: env/bin/honcho -e defaults.env,local.env run env/bin/fake_data fake_data
==> basebox: Traceback (most recent call last):
==> basebox:   File "env/bin/fake_data", line 9, in <module>
==> basebox:     load_entry_point('gratipay==1736.post27', 'console_scripts', 'fake_data')()
...
==> basebox: psycopg2.IntegrityError: new row for relation "transfers" violates check constraint "positive"
==> basebox: DETAIL:  Failing row contains (578395790, 2014-08-20 12:00:37+00, Dominicbdp, Katharina92q, 0.00, tip).
==> basebox: make:
==> basebox: *** [data] Error 1

Moved to #2950

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Nov 19, 2014

Contributor

Please test new vagrant up and create tickets for any issues you may run into. I think now it is good on Windows.

Contributor

techtonik commented Nov 19, 2014

Please test new vagrant up and create tickets for any issues you may run into. I think now it is good on Windows.

pjfontillas added a commit to pjfontillas/dutchman that referenced this issue Jan 22, 2016

Tweaks to provisioning and make file to work in VM as Windows host.
Ran into a virtualization error that took a while to debug:
gratipay/gratipay.com#2327
pypa/virtualenv#565
Ultimate solution: http://stackoverflow.com/questions/24640819/protocol-error-setting-up-virtualenvironment-through-vagrant-on-ubuntu
Run console as admin, shared mounts permissions issue.
More evidence of issues on Windows: bash script cannot access vagrant shared mount, even when run as root.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.