VirtualEnv error with Windows + Vagrant #2327

Closed
clone1018 opened this Issue Apr 26, 2014 · 27 comments

Projects

None yet

6 participants

@clone1018
Member

No idea where to even start with this one. Fresh clone with fresh vagrant box.

Here's the weird part, at home my Windows 8.1 x64 computer experiences this problem but my Windows 7 x64 computer at work doesn't. This also happens to @abnor

vagrant@precise64:~/www.gittip.com$ make clean env
rm -rf env *.egg *.egg-info
find . -name \*.pyc -delete
"/usr/bin/python2.7"  "./vendor/virtualenv-1.9.1.py"\
                                --unzip-setuptools \
                                --prompt="[gittip] " \
                                --never-download \
                                --extra-search-dir=./vendor/ \
                                --distribute \
                                ./env/
Traceback (most recent call last):
  File "./vendor/virtualenv-1.9.1.py", line 2577, in <module>
    main()
  File "./vendor/virtualenv-1.9.1.py", line 979, in main
    no_pip=options.no_pip)
  File "./vendor/virtualenv-1.9.1.py", line 1081, in create_environment
    site_packages=site_packages, clear=clear))
  File "./vendor/virtualenv-1.9.1.py", line 1275, in install_python
    copyfile(join(stdlib_dir, fn), join(lib_dir, fn))
  File "./vendor/virtualenv-1.9.1.py", line 437, in copyfile
    copyfileordir(src, dest)
  File "./vendor/virtualenv-1.9.1.py", line 412, in copyfileordir
    shutil.copytree(src, dest, True)
  File "/usr/lib/python2.7/shutil.py", line 206, in copytree
    raise Error, errors
shutil.Error: [('/usr/lib/python2.7/config/libpython2.7.so', './env/lib/python2.
7/config/libpython2.7.so', '[Errno 71] Protocol error')]
make: *** [env] Error 1

Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

@patcon
Contributor
patcon commented Apr 26, 2014
@whit537
Member
whit537 commented Apr 29, 2014

Blech. I looked at this thinking it might be a simple Python thing but [Errno 71] Protocol error looks more virtualization-related.

@whit537 whit537 added the TeamX label May 8, 2014
@whit537
Member
whit537 commented May 8, 2014

TeamX

@patcon
Contributor
patcon commented May 9, 2014

Let's try this?
pypa/virtualenv#409

Can someone on a failing windows version give the --always-copy flag a shot?

@lukaszbanasiak

@patcon

For me it works. Without --always-copy flag:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ virtualenv --no-site-packages env
New python executable in /vagrant/env/bin/python
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 985, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 1405, in install_python
    os.symlink(py_executable_base, full_pth)
OSError: [Errno 71] Protocol error

With --always-copy flag:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ virtualenv --no-site-packages env --always-copy
New python executable in env/bin/python
Installing setuptools, pip...done.

Windows 7 64-bit, vagrant (Ubuntu 14.04 64-bit)

@whit537
Member
whit537 commented Sep 10, 2014

Thanks @lukaszbanasiak! So where do we need to add --always-copy? To the Makefile? Want to give us a PR for that?

@techtonik techtonik self-assigned this Oct 29, 2014
@techtonik
Member

Just run into this. I think I'll be able to tackle this. The problem is likely related to permissions for files mounted as VirtualBox share. Git also chokes on them, because all files are executable, permissions are immutable etc.

@techtonik
Member

Tracing the chain so far. This is invoked from /vagrant directory.

[vagrant provision] -> [scripts/vagrant-setup.sh] -> [make env] ->
  -> [virtualenv-1.9.1.py ...]
@techtonik
Member

virtualenv-1.9.1 calls failing copy tree two times until fault:

shutil.copytree(src, dst, symlinks=False):

==> default: Copying /usr/lib/python2.7/lib-dynload ./env/lib/python2.7/lib-dynload
==> default: Copying /usr/lib/python2.7/config ./env/lib/python2.7/config
@techtonik
Member

/usr/lib/python2.7/config appears to be a directory:

drwxr-xr-x  2 root root     4096 Apr 14  2014 .
drwxr-xr-x 26 root root    24576 Apr 14  2014 ..
...
-rw-r--r--  1 root root 18965158 Feb 27  2014 libpython2.7-pic.a
lrwxrwxrwx  1 root root       23 Feb 27  2014 libpython2.7.so -> ../../libpython2.7.so.1
...

This is a symlink that can not be symlinked to /vagrant share, because it is mounted from Windows which doesn't support this.

@techtonik
Member

virtualenv 1.11.6 doesn't fix this. --always-copy was added in 1.10, so we need to upgrade virtualenv anyway.

@whit537
Member
whit537 commented Oct 29, 2014

@techtonik If you want help ping @joeyespo. He's into Vagrant + Windows.

@techtonik
Member

Don't need it yet, but I'll keep that in mind. Thanks. The only help needed right now is to merge some virtualenv pull requests.

@techtonik
Member

Filled (pypa/virtualenv#663). Going to add --always-copy for now and upgrade virtualenv to 1.11.6 to see how it goes.

@techtonik
Member

Didn't get far. Hit mitchellh/vagrant#4745 on the way. Shot myself in the foot with this one, actually.

@techtonik
Member

Next fail with virtualenv 1.11.6:

...
==> default: Cannot find a wheel for setuptools
==> default: Cannot find a wheel for pip
==> default: Installing setuptools, pip...
==> default:   Complete output from command /vagrant/env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
==> default:   Traceback (most recent call last):
==> default:   File "<string>", line 1, in <module>
==> default: ImportError: No module named pip
==> default: ----------------------------------------
==> default: ...Installing setuptools, pip...done.
==> default: Traceback (most recent call last):
==> default:   File "./vendor/virtualenv.py", line 2338, in <module>
==> default:     main()
==> default:   File "./vendor/virtualenv.py", line 824, in main
==> default:     symlink=options.symlink)
==> default:   File "./vendor/virtualenv.py", line 992, in create_environment
==> default:     install_wheel(to_install, py_executable, search_dirs)
==> default:   File "./vendor/virtualenv.py", line 960, in install_wheel
==> default:     'PIP_NO_INDEX': '1'
==> default:   File "./vendor/virtualenv.py", line 902, in call_subprocess
==> default:     % (cmd_desc, proc.returncode))
==> default: OSError: Command /vagrant/env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1
==> default: make: *** [env] Error 1

Looks like pip upgrade is needed.

@techtonik
Member

Downloading pip and setuptools wheels helped. Going to commit later.

@techtonik
Member

Well, another problem:

...
  File "distribute_setup.py", line 129, in _do_download
    _build_egg(egg, tarball, to_dir)
  File "distribute_setup.py", line 120, in _build_egg
    raise IOError('Could not build the egg.')
IOError: Could not build the egg.
@joeyespo
Contributor

RE: Vagrant + Windows

Honestly, even after fixing the quirks, working within a Vagrant environment hasn't been a pleasant dev experience. It's terribly slow. And getting set up is still quite tedious (which is the whole point of Vagrant). This has kind of pushed me away from participating. I don't feel productive because of how much time it takes to get from "idea I want to test" or "issue I want to resolve" to "localhost".

Instead, I'd much rather work within a "native" Python and JavaScript dev environment since these technologies are already cross-platform. Striving for a single, cross-platform run step (and ideally, a single, cross-platform install step too) is a much better goal than "let's fix Vagrant."

I've applied this approach to other websites (both monolithic and microservices), and it's made both onboarding and day-to-day development much smoother. Contributors can continue to use the tools and the environments that make them maximally productive. And there's no time wasted with installation hand-holding. Requiring each machine to be exactly the same in order to make high-level changes shuts out a lot of people. (As for the dev/prod-parity as a bug-catching mechanism, which usually comes up, auto-running tests in a prod-like CI tool before merging PRs has been equivalently effective.) I'd be happy to talk more about the approaches I've used to make webapps more cross-platform if there's any interest here, but unfortunately, I don't think I'll be much help with Vagrant issues anymore.

@whit537
Member
whit537 commented Oct 29, 2014

Thanks for checking in, @joeyespo! So are you saying Vagrant in general is a buzzkill for you lately, or Vagrant as we've implemented it? I think I'm hearing you say the former.

@joeyespo
Contributor

@whit537 Sure! Glad things are going strong. I also really like the rename, it's catchy :-)

Yeah, the former.

@whit537
Member
whit537 commented Oct 30, 2014

@joeyespo So what's your secret formula? Let me guess: use SQLite? ;-)

@techtonik
Member

IOError: Could not build the egg. is caused by old distribute conflicting with new setuptools.
https://bitbucket.org/pypa/setuptools/issue/18/packages-that-use_setuptools-from-older

@techtonik techtonik added a commit to techtonik/gratipay.com that referenced this issue Nov 1, 2014
@techtonik techtonik Start rewriting Vagrantfile for Ubuntu 14.04 (#2327 and #2868) cf90d8e
@techtonik
Member

This one is blocked by #2886 through #2867 ATM..

@joeyespo
Contributor

@whit537 Haha, yeah, that's one option. A decent one too, since it's built-in to Python and is supported by popular ORMs.

The general idea is use cross-platform auto-installable components for dev, and production-optimized components for prod, staging, and tests running on CI.

Since Python/JS packages are already auto-installable and cross-platform, it's a natural fit to use them in place of the prod-ready counterparts. Of course, not every prod-ready component has a pure-Python implementation. In those cases, you have to substitute higher up. ORMs are a good option because the ability to plug in different backends is already built for you.

As for postgres.py, I'm not sure what the best approach would be. It highly couples your app to Postgres instead of to an object model. But I'm willing to talk through some ideas. Let's move the discussion to a new issue.

@techtonik
Member

Bootstrap fails with DB error:

==> basebox: env/bin/honcho -e defaults.env,local.env run env/bin/fake_data fake_data
==> basebox: Traceback (most recent call last):
==> basebox:   File "env/bin/fake_data", line 9, in <module>
==> basebox:     load_entry_point('gratipay==1736.post27', 'console_scripts', 'fake_data')()
...
==> basebox: psycopg2.IntegrityError: new row for relation "transfers" violates check constraint "positive"
==> basebox: DETAIL:  Failing row contains (578395790, 2014-08-20 12:00:37+00, Dominicbdp, Katharina92q, 0.00, tip).
==> basebox: make:
==> basebox: *** [data] Error 1

Moved to #2950

@whit537 whit537 closed this in fefe00d Nov 19, 2014
@techtonik
Member

Please test new vagrant up and create tickets for any issues you may run into. I think now it is good on Windows.

@pjfontillas pjfontillas added a commit to pjfontillas/dutchman that referenced this issue Jan 22, 2016
@pjfontillas pjfontillas Tweaks to provisioning and make file to work in VM as Windows host.
Ran into a virtualization error that took a while to debug:
gratipay/gratipay.com#2327
pypa/virtualenv#565
Ultimate solution: http://stackoverflow.com/questions/24640819/protocol-error-setting-up-virtualenvironment-through-vagrant-on-ubuntu
Run console as admin, shared mounts permissions issue.
More evidence of issues on Windows: bash script cannot access vagrant shared mount, even when run as root.
16e2fad
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment