Breakage in --always-copy - fails with: ImportError: No module named time #565

Open
ionelmc opened this Issue Feb 12, 2014 · 22 comments

Comments

Projects
None yet

ionelmc commented Feb 12, 2014

On CentOS 6.5 (python2.7 from SCL):

[dev@centbox core]$ virtualenv-2.7 --always-copy .break
sys.path (patched logging) = ['/opt/rh/python27/root/usr/bin', '/opt/rh/python27/root/usr/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg', '/opt/rh/python27/root/usr/lib/python2.7/site-packages/wheel-0.22.0-py2.7.egg', '/opt/rh/python27/root/usr/lib/python2.7/site-packages/virtualenv-1.11.2-py2.7.egg', '/opt/rh/python27/root/usr/lib/python2.7/site-packages/pip-1.5.2-py2.7.egg', '/opt/rh/python27/root/usr/lib64/python27.zip', '/opt/rh/python27/root/usr/lib64/python2.7', '/opt/rh/python27/root/usr/lib64/python2.7/plat-linux2', '/opt/rh/python27/root/usr/lib64/python2.7/lib-tk', '/opt/rh/python27/root/usr/lib64/python2.7/lib-old', '/opt/rh/python27/root/usr/lib64/python2.7/lib-dynload', '/opt/rh/python27/root/usr/lib64/python2.7/site-packages', '/opt/rh/python27/root/usr/lib/python2.7/site-packages']
Cannot find file lib (bad symlink)
New python executable in .break/bin/python2.7
Also creating executable in .break/bin/python
Installing setuptools, pip...
  Complete output from command /home/dev/core/.break/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/virtualenv-1.11.2-py2.7.egg/virtualenv_support/pip-1.5.2-py2.py3-none-any.whl/pip/__init__.py", line 9, in <module>
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/virtualenv-1.11.2-py2.7.egg/virtualenv_support/pip-1.5.2-py2.py3-none-any.whl/pip/log.py", line 6, in <module>
  File "/opt/rh/python27/root/usr/lib64/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref
ImportError: No module named time
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "/opt/rh/python27/root/usr/bin/virtualenv-2.7", line 9, in <module>
    load_entry_point('virtualenv==1.11.2', 'console_scripts', 'virtualenv-2.7')()
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/virtualenv-1.11.2-py2.7.egg/virtualenv.py", line 824, in main
    symlink=options.symlink)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/virtualenv-1.11.2-py2.7.egg/virtualenv.py", line 992, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/virtualenv-1.11.2-py2.7.egg/virtualenv.py", line 960, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/virtualenv-1.11.2-py2.7.egg/virtualenv.py", line 902, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/dev/core/.break/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

Works fine without it:

[dev@centbox core]$ virtualenv-2.7 .works
sys.path (patched logging) = ['/opt/rh/python27/root/usr/bin', '/opt/rh/python27/root/usr/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg', '/opt/rh/python27/root/usr/lib/python2.7/site-packages/wheel-0.22.0-py2.7.egg', '/opt/rh/python27/root/usr/lib/python2.7/site-packages/virtualenv-1.11.2-py2.7.egg', '/opt/rh/python27/root/usr/lib/python2.7/site-packages/pip-1.5.2-py2.7.egg', '/opt/rh/python27/root/usr/lib64/python27.zip', '/opt/rh/python27/root/usr/lib64/python2.7', '/opt/rh/python27/root/usr/lib64/python2.7/plat-linux2', '/opt/rh/python27/root/usr/lib64/python2.7/lib-tk', '/opt/rh/python27/root/usr/lib64/python2.7/lib-old', '/opt/rh/python27/root/usr/lib64/python2.7/lib-dynload', '/opt/rh/python27/root/usr/lib64/python2.7/site-packages', '/opt/rh/python27/root/usr/lib/python2.7/site-packages']
New python executable in .works/bin/python2.7
Also creating executable in .works/bin/python
Installing setuptools, pip...done.
[dev@centbox core]$

I'm hitting this issue too, but as a workaround you can add no-setuptools to not install pip in your virtualenv. This issue is related to installing pip/setuptools in the new virtualenv.

My version information:

  • Fedora 19 x86_64,
  • System install of Python 2.7.5
  • virtualenv 1.11.4
  • pip 1.5.4

Here is the full verbose output on failure:

[vagrant@localhost ~]$ virtualenv --always-copy --clear --verbose foobar
Do not need to delete foobar/lib/python2.7; already gone
Not deleting foobar/bin
Creating foobar/lib/python2.7
Cannot find file lib (bad symlink)
Symlinking Python bootstrap modules
  Copying to foobar/lib/python2.7/config
  Copying to foobar/lib/python2.7/lib-dynload
  Creating parent directories for foobar/lib64/python2.7
  Copying to foobar/lib64/python2.7/os.py
  Ignoring built-in bootstrap module: posix
  Copying to foobar/lib64/python2.7/posixpath.py
  Cannot import bootstrap module: nt
  Copying to foobar/lib64/python2.7/ntpath.py
  Copying to foobar/lib64/python2.7/genericpath.py
  Copying to foobar/lib64/python2.7/fnmatch.py
  Copying to foobar/lib64/python2.7/locale.py
  Copying to foobar/lib64/python2.7/encodings
  Copying to foobar/lib64/python2.7/codecs.py
  Copying to foobar/lib64/python2.7/stat.py
  Copying to foobar/lib64/python2.7/UserDict.py
  Creating parent directories for foobar/lib64/python2.7/lib-dynload
  Copying to foobar/lib64/python2.7/lib-dynload/readline.so
  Copying to foobar/lib64/python2.7/copy_reg.py
  Copying to foobar/lib64/python2.7/types.py
  Copying to foobar/lib64/python2.7/re.py
  Copying to foobar/lib64/python2.7/sre.py
  Copying to foobar/lib64/python2.7/sre_parse.py
  Copying to foobar/lib64/python2.7/sre_constants.py
  Copying to foobar/lib64/python2.7/sre_compile.py
  Copying to foobar/lib64/python2.7/lib-dynload/zlibmodule.so
  Copying to foobar/lib64/python2.7/warnings.py
  Copying to foobar/lib64/python2.7/linecache.py
  Copying to foobar/lib64/python2.7/_abcoll.py
  Copying to foobar/lib64/python2.7/abc.py
  Copying to foobar/lib64/python2.7/_weakrefset.py
Creating foobar/lib/python2.7/site-packages
Writing foobar/lib64/python2.7/site.py
Writing foobar/lib64/python2.7/orig-prefix.txt
Writing foobar/lib64/python2.7/no-global-site-packages.txt
Creating parent directories for foobar/include
Copying to foobar/include/python2.7
Creating foobar/bin
New python executable in foobar/bin/python
Changed mode of foobar/bin/python to 0775
Copying to foobar/bin/python2
Copying to foobar/bin/python2.7
Testing executable with foobar/bin/python -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
Got sys.prefix result: u'/home/vagrant/foobar'
Creating foobar/lib64/python2.7/distutils
Writing foobar/lib64/python2.7/distutils/__init__.py
Writing foobar/lib64/python2.7/distutils/distutils.cfg
Installing setuptools, pip...
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/__init__.py", line 9, in <module>
    File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/log.py", line 6, in <module>
    File "/usr/lib64/python2.7/logging/__init__.py", line 26, in <module>
      import sys, os, time, cStringIO, traceback, warnings, weakref
  ImportError: No module named time
  Complete output from command /home/vagrant/foobar/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/__init__.py", line 9, in <module>
  File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-1.5.4-py2.py3-none-any.whl/pip/log.py", line 6, in <module>
  File "/usr/lib64/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref
ImportError: No module named time
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 824, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 992, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 960, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 902, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/vagrant/foobar/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

Here is the full verbose output on success:

[vagrant@localhost ~]$ virtualenv --always-copy --clear --verbose --no-setuptools foobar
Do not need to delete foobar/lib/python2.7; already gone
Not deleting foobar/bin
Creating foobar/lib/python2.7
Cannot find file lib (bad symlink)
Symlinking Python bootstrap modules
  Copying to foobar/lib/python2.7/config
  Copying to foobar/lib/python2.7/lib-dynload
  Creating parent directories for foobar/lib64/python2.7
  Copying to foobar/lib64/python2.7/os.py
  Ignoring built-in bootstrap module: posix
  Copying to foobar/lib64/python2.7/posixpath.py
  Cannot import bootstrap module: nt
  Copying to foobar/lib64/python2.7/ntpath.py
  Copying to foobar/lib64/python2.7/genericpath.py
  Copying to foobar/lib64/python2.7/fnmatch.py
  Copying to foobar/lib64/python2.7/locale.py
  Copying to foobar/lib64/python2.7/encodings
  Copying to foobar/lib64/python2.7/codecs.py
  Copying to foobar/lib64/python2.7/stat.py
  Copying to foobar/lib64/python2.7/UserDict.py
  Creating parent directories for foobar/lib64/python2.7/lib-dynload
  Copying to foobar/lib64/python2.7/lib-dynload/readline.so
  Copying to foobar/lib64/python2.7/copy_reg.py
  Copying to foobar/lib64/python2.7/types.py
  Copying to foobar/lib64/python2.7/re.py
  Copying to foobar/lib64/python2.7/sre.py
  Copying to foobar/lib64/python2.7/sre_parse.py
  Copying to foobar/lib64/python2.7/sre_constants.py
  Copying to foobar/lib64/python2.7/sre_compile.py
  Copying to foobar/lib64/python2.7/lib-dynload/zlibmodule.so
  Copying to foobar/lib64/python2.7/warnings.py
  Copying to foobar/lib64/python2.7/linecache.py
  Copying to foobar/lib64/python2.7/_abcoll.py
  Copying to foobar/lib64/python2.7/abc.py
  Copying to foobar/lib64/python2.7/_weakrefset.py
Creating foobar/lib/python2.7/site-packages
Writing foobar/lib64/python2.7/site.py
Writing foobar/lib64/python2.7/orig-prefix.txt
Writing foobar/lib64/python2.7/no-global-site-packages.txt
Creating parent directories for foobar/include
Copying to foobar/include/python2.7
Creating foobar/bin
New python executable in foobar/bin/python
Changed mode of foobar/bin/python to 0775
Copying to foobar/bin/python2
Copying to foobar/bin/python2.7
Testing executable with foobar/bin/python -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
Got sys.prefix result: u'/home/vagrant/foobar'
Creating foobar/lib64/python2.7/distutils
Writing foobar/lib64/python2.7/distutils/__init__.py
Writing foobar/lib64/python2.7/distutils/distutils.cfg
Writing foobar/bin/activate
Writing foobar/bin/activate.fish
Writing foobar/bin/activate_this.py
Writing foobar/bin/activate.csh

In both the failing and successful cases the first error line is:

Cannot find file lib (bad symlink)

This log isn't useful because I don't have the fullpath to the file it can't find. I hence hacked virtualenv.py to print out os.path.abspath for this file and got:

Cannot find file /home/vagrant/lib (bad symlink)

It's looking for a lib folder in the current working directory.

dstufft referenced this issue Jan 5, 2015

Closed

Rewrite virtualenv #697

0 of 5 tasks complete

lhw365 commented Jan 19, 2015

the same here on Centos7 with stock python 2.7

$ virtualenv --always-copy /opt/
New python executable in /opt/bin/python
Installing setuptools, pip...
  Complete output from command /opt/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-6.0.6-py2.py3-none-any.whl/pip/__init__.py", line 4, in <module>
  File "/usr/lib64/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref
ImportError: No module named time
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 993, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 961, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 903, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /opt/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

Same here. Using --always-copy on CentOS 6.5, system Python 2.6 ends up with same ImportError: No module named time as mentioned.

lhw365 commented Feb 20, 2015

the solution is to forget about the --always-copy option on EL based distros

This was referenced May 14, 2015

there seems to be a typo in the fix_lib64 function that is currently running. I correct this and it works as expected. go to line:

   1579         if symlink:                              
   1580             os.symlink('lib', lib64_link)      
   1581         else:                                    
   1582             copyfile('lib', lib64_link)        

replace 'lib' with lib_dir

  1579         if symlink:                              
   1580             os.symlink(lib_dir, lib64_link)      
   1581         else:                                    
   1582             copyfile(lib_dir, lib64_link)        

Late, but hope this helps

jdelgad commented Jul 10, 2015

I'm getting the same problem here as well using CentOS 7.1 with flags --no-setup-tools and --always-copy. I'd love for this to be fixed since I'm packaging up the virtualenv and shipping them to customers (that are also on CentOS7+ so it's not a major risk).

Same here on Gentoo x86_64 with --always-copy.

@jdelgad and @selurvedu did you try what i recommended above?

@ikiini Yes, your patch solves the problem.

jdelgad commented Aug 23, 2015

@ikiini It worked! Thank you so much. Now to either get this into RHEL 7 or into the main branch.

+1

great. Sorry for the delayed response..but you're welcome @selurvedu and @jdelgad.

Member

Ivoz commented Oct 21, 2015

Only problem seems to be @ikiini 's patch would seem to regress back on #338 which is when it was actually changed from lib_dir to 'lib'

Problem still persists with CentOS 7 and python 2.7

[root@st2w-master-el7-5696a8a082 ~]# virtualenv no_copy
New python executable in no_copy/bin/python
Installing setuptools, pip, wheel...done.
[root@st2w-master-el7-5696a8a082 ~]# virtualenv --always-copy yes_copy
Cannot find file lib (bad symlink)
New python executable in yes_copy/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /root/yes_copy/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/virtualenv_support/pip-7.1.2-py2.py3-none-any.whl/pip/__init__.py", line 4, in <module>
  File "/usr/lib64/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref
ImportError: No module named time
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 832, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1004, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 969, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 910, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /root/yes_copy/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel failed with error code 1
[root@st2w-master-el7-5696a8a082 ~]# virtualenv --version
13.1.2
[root@st2w-master-el7-5696a8a082 ~]# ls -rlth yes_copy/
total 0
drwxr-xr-x. 3 root root 22 Jan 13 22:04 lib
drwxr-xr-x. 3 root root 22 Jan 13 22:04 lib64
drwxr-xr-x. 3 root root 22 Jan 13 22:04 include
drwxr-xr-x. 2 root root 49 Jan 13 22:04 bin
[root@st2w-master-el7-5696a8a082 ~]#

@pjfontillas pjfontillas added a commit to pjfontillas/dutchman that referenced this issue Jan 22, 2016

@pjfontillas pjfontillas Tweaks to provisioning and make file to work in VM as Windows host.
Ran into a virtualization error that took a while to debug:
gratipay/gratipay.com#2327
pypa/virtualenv#565
Ultimate solution: http://stackoverflow.com/questions/24640819/protocol-error-setting-up-virtualenvironment-through-vagrant-on-ubuntu
Run console as admin, shared mounts permissions issue.
More evidence of issues on Windows: bash script cannot access vagrant shared mount, even when run as root.
16e2fad

rohitpaulk referenced this issue in gratipay/inside.gratipay.com Jan 28, 2016

Closed

run Gratipay 191 #485

@bwilliams42 - Your change indeed fixed the problem for symlink-ing logic. However, things are fundamentally broken when not using --always-copy. Note that copyfile has a "symlink" kwarg that defaults to True. The following logic is effectively, if symlink, call os.symlink, else call copyfile w/o the "symlink" kwarg which results in copyfile defaulting symlink=True and a symlink is created by copyfile. Not useful at all. A symlink is the result no matter what.

Shouldn't the patch be something like the following?

   1579         if symlink:                              
   1580             os.symlink(lib_dir, lib64_link)      
   1581         else:                                    
   1582             copyfile(lib_dir, lib64_link, symlink=symlink)

jdelgad commented Feb 28, 2016

I had an issue whereby I did this and I had a symlink to /usr/lib64. When RH updated some of the openssl libraries this was broken (I was using ansible) with an error stating something like openssl_meth function. I fixed it by doing a deep copy of /usr/lib64. Not a complete fix @bwilliams42 but pointed me in the right direction.

pb-cdunn referenced this issue in PacificBiosciences/FALCON-integrate Apr 19, 2016

Closed

failed to install on CentOS 7.2 #39

bialix commented Oct 18, 2016 edited

Problem still persists on Centos 7.2. This ticket has been reported 2.5 years ago. It seems it's very serious problem. How can I help to fix it?

Owner

pfmoore commented Oct 18, 2016

Someone needs to provide a working fix. There's one fix that's been proposed here, but it apparently has a couple of problems. I'd suggest starting there.

bialix commented Oct 18, 2016

BTW, I'm trying to build and pack my python appliction into virtual env on special VM under Vagrant with help of fabric, so it's pretty easy to reproduce this bug with Vagrant + centos box + simple fabric script.

I don't have inner knowledge about how virtualenv is supposed to work, so I need some guidance.

Owner

pfmoore commented Oct 18, 2016

I can't help there sorry, as I'm on Windows, and the whole question of symlinking vs copying (or lib64 directories, which seem involved in this) isn't relevant there.

@Neurocinetics Neurocinetics added a commit to Neurocinetics/virtualenv that referenced this issue Feb 1, 2017

@Neurocinetics Neurocinetics Supposed typo: pypa#565 (comment), On CentOS 7 building certain proje…
…cts fails without this "fix". Also relevant, spotify/dh-virtualenv#150 and numpy/numpy#7570.
2e1319b

kika commented May 30, 2017

Interesting that (I'm on Amazon Linux) when I do virtualenv --always-copy app I get this error (Cannot find file lib (bad symlink)) but when I do mkdir app; cd app; virtualenv --always-copy . I do not. And it seems I understand why - when the app is created by the virtualenv it has lib64 as a directory, but when I create the virtualenv in the current directory it becomes a link to lib.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment