Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Installation fails in Cygwin 64 bit #1547

Closed
gregersn opened this Issue · 54 comments
@gregersn

When I try a pip install requests in Cygwin 64 bit the installation fails.

Here's the end of the log file:

  Downloading from URL https://pypi.python.org/packages/source/r/requests/requests-1.2.3.tar.gz#md5=adbd3f18445f7fe5e77f65c502e264fb (from https://pypi.python.org/simple/requests/)
  Running setup.py egg_info for package requests

Cleaning up...

  Removing temporary dir /cygdrive/d/home/greger/cygwin/projects/myproject/build...
No files/directories in /cygdrive/d/home/greger/cygwin/projects/myproject/build/requests/pip-egg-info (from PKG-INFO)

Exception information:
Traceback (most recent call last):
  File "/cygdrive/d/home/greger/cygwin/projects/myproject/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/cygdrive/d/home/greger/cygwin/projects/myproject/lib/python2.7/site-packages/pip/commands/install.py", line 236, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/cygdrive/d/home/greger/cygwin/projects/myproject/lib/python2.7/site-packages/pip/req.py", line 1139, in prepare_files
    req_to_install.assert_source_matches_version()
  File "/cygdrive/d/home/greger/cygwin/projects/myproject/lib/python2.7/site-packages/pip/req.py", line 394, in assert_source_matches_version
    version = self.installed_version
  File "/cygdrive/d/home/greger/cygwin/projects/myproject/lib/python2.7/site-packages/pip/req.py", line 390, in installed_version
    return self.pkg_info()['version']
  File "/cygdrive/d/home/greger/cygwin/projects/myproject/lib/python2.7/site-packages/pip/req.py", line 357, in pkg_info
    data = self.egg_info_data('PKG-INFO')
  File "/cygdrive/d/home/greger/cygwin/projects/myproject/lib/python2.7/site-packages/pip/req.py", line 293, in egg_info_data
    filename = self.egg_info_path(filename)
  File "/cygdrive/d/home/greger/cygwin/projects/myproject/lib/python2.7/site-packages/pip/req.py", line 330, in egg_info_path
    raise InstallationError('No files/directories in %s (from %s)' % (base, filename))
InstallationError: No files/directories in /cygdrive/d/home/greger/cygwin/projects/myproject/build/requests/pip-egg-info (from PKG-INFO)

Don't know if that is a problem with the installation package, or cygwin itself. But thought I'd try here first.

If I try to use easy_install it just leaves all the files in the tmp-dir. And if I from there try to do a python setup.py install, and then try to import requests from the python shell, it just exits without and output.

@Lukasa
Collaborator

Hmm, I can't reproduce this. =( But I wonder if you're using a Windows virtualenv with a Windows interpreter in Cygwin. It's not clear (to me) how well that would work.

@gregersn

You are right, I did try through VirtualEnv and with VirtualenvWrapper. Tried again now from just regular cygwin prompt, and here's the log, again:

  Downloading from URL https://pypi.python.org/packages/source/r/requests/requests-1.2.3.tar.gz#md5=adbd3f18445f7fe5e77f65c502e264fb (from https://pypi.python.org/simple/requests/)
  Running setup.py egg_info for package requests

Cleaning up...

  Removing temporary dir /tmp/pip_build_greger...
No files/directories in /tmp/pip_build_greger/requests/pip-egg-info (from PKG-INFO)

Exception information:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg/pip/commands/install.py", line 236, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg/pip/req.py", line 1139, in prepare_files
    req_to_install.assert_source_matches_version()
  File "/usr/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg/pip/req.py", line 394, in assert_source_matches_version
    version = self.installed_version
  File "/usr/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg/pip/req.py", line 390, in installed_version
    return self.pkg_info()['version']
  File "/usr/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg/pip/req.py", line 357, in pkg_info
    data = self.egg_info_data('PKG-INFO')
  File "/usr/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg/pip/req.py", line 293, in egg_info_data
    filename = self.egg_info_path(filename)
  File "/usr/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg/pip/req.py", line 330, in egg_info_path
    raise InstallationError('No files/directories in %s (from %s)' % (base, filename))
InstallationError: No files/directories in /tmp/pip_build_greger/requests/pip-egg-info (from PKG-INFO)
@gregersn

Tested again in a clean installed Cygwin64 bit, same error.
It does work in 32 bit version of Cygwin, though.

@Lukasa
Collaborator

That's truly bizarre. It looks like pip is being installed as part of Cygwin, so you shouldn't have any 32/64 bit problems, but I kinda have to believe you are.

@gregersn

I did install pip with easy_install , so the problem might lay there. Still a bit weird, since pip has no problem installing lots of other packages.

@sigmavirus24
Collaborator

I was under the impression pip was broken on Windows but I'm probably stuck in the past.

@Lukasa
Collaborator

Pip works fine on Windows, and in my experience is fine in Cygwin too. Except for this.

@zodman

same here the requests not install behind cygwin

@meeotch

I'm getting the same behavior as the OP: "pip install requests" works on 32 bit cygwin, but fails on 64 bit cygwin with messages about "no files/directories".

Possibly unrelated, but installation-relevant & equally mysterious: 64 bit cygwin wget fails to verify certs for https URLs, while 32 bit cygwin wget succeeds. (Certs are identical on each install, and running the 32 bit wget inside a 64 bit installation [including /usr/ssl/certs directory] works, too.) Anyway, what I'm saying is that installing python stuff under 64 bit cygwin has been a pain in the butt, and I'm keen to hear solutions.

@kennethreitz
Owner

This sounds like an issue with pip itself.

@sigmavirus24
Collaborator

I like the way you think @kennethreitz. :P

It sounds more to me, however, that 64 bit cygwin is broken (maybe).

@tanzaho

Same thing here. Very interested if someone finds a solution.

@sigmavirus24
Collaborator

Have any details you can add @tanzaho ?

@tanzaho

I get the following when run on cygwin64 and trying to pip install the module :
Downloading requests-1.2.3.tar.gz (348kB): 348kB downloaded
Running setup.py egg_info for package requests
Cleaning up...
No files/directories in /home/tanzaho/.virtualenvs/django_wordiz/build/requests/pip-egg-info (from PKG-INFO)

@mikewn

I'm a bit new to running python setup scripts, especially under 64 bit cygwin in Windows 8. But firewall issues forced me to download the requests-1.2.3.tar.gz file by hand (I couldn't use pip or easy_install without timeouts). I gunzipped and untarred the file and then went in to the directory and ran:

python setup.py install

by hand, and it doesn't output anything and returns to the prompt. Nothing is created in any of the cygwin /usr/lib/python*/site-packages directories. So I think that the process for running the above command is broken rather than it being a connectivity problem or a problem with pip itself. So, you probably don't need to reproduce any network problems to reproduce this problem. Just try to do the above and see if you get the same thing when installing from the command line in a cygwin terminal.

Would like to see this fixed too, as pip and easy_install aren't options for me to install libraries.

Also, I get core dumps if I try to execute the command above with "python3" or "python3.2" instead of "python".

@sigmavirus24
Collaborator

Also, I get core dumps if I try to execute the command above

@mikewn could you provide those? I'm wondering if this isn't an issue in the way Python is built for Windows now. I don't have a windows box to test with though. (Also I didn't know anyone was willing to subject themselves to the horror of Windows 8 ;))

@Lukasa
Collaborator

@sigmavirus24 I have Windows 8. =) Strongly considering moving to Linux on that machine though. Anyway, off-topic.

It's worth noting that this is absolutely not reproducible on 32-bit Cygwin on Windows 7, which strongly points to a problem with Cygwin.

@mikewn

@sigmavirus24 - here's the dump. Basically running python3 or python3.2 both yield stack dumps for python 3.2. And no, this isn't my personal box that I'm running Windows 8 on. :)

https://gist.github.com/mikewn/6688148

@ahmadia

Does the script, perchance, call a subprocess that uses the UUID module in Python? It turns out that the UUID module in Cygwin64 Python is currently segfaulting, which might be causing mysterious failures.

@sigmavirus24
Collaborator
@ahmadia

This is really an upstream problem for CPython and the Cygwin folks in my mind, so I'm not sure this is on you. I'll post a clean uuid patch against the current Python release on Cygwin on here for the self-starters.

@ahmadia

Just a confirmation that I can install requests correctly with my UUID-patch Python. I just successfully completed a python setup.py install and the test suite passes. I'm putting together a patch for the Cygwin folks, I'll post back here when it's ready.

@sigmavirus24
Collaborator

Thanks @ahmadia! I'll close this when you let us know the outcome. Otherwise I want it to be easy to find

@ahmadia

Until the patch lands in Cygwin's official repository, you will need to hotfix your existing Python. You can do with the following simple patch from Evgeny Sologubov which short-circuits the uuid module from attempting to load further libraries after the uuid routines have been located.

Apply the following patch to the uuid.py module in /usr/lib/python2.7

diff -r 4a318a45c4c3 Lib/uuid.py
--- a/Lib/uuid.py   Mon Aug 19 13:07:18 2013 -0400
+++ b/Lib/uuid.py   Mon Aug 19 21:41:08 2013 +0400
@@ -429,6 +429,8 @@
             _uuid_generate_random = lib.uuid_generate_random
         if hasattr(lib, 'uuid_generate_time'):
             _uuid_generate_time = lib.uuid_generate_time
+            if _uuid_generate_random is not None:
+                break  # found everything we were looking for

     # The uuid_generate_* functions are broken on MacOS X 10.5, as noted
     # in issue #8621 the function generates the same sequence of values
@sigmavirus24
Collaborator

Since there's a solution here, I'm actually tempted to close this. How do you feel @Lukasa ?

@ahmadia

Here's a two-liner to hotfix CPython from your Cygwin shell until the patch lands in Cygwin-apps (assuming you've got wget installed):

wget http://bugs.python.org/file31377/uuid.patch 
patch -d /usr/lib/python2.7 -p2 < uuid.patch 

+1 on closing this issue.

@sigmavirus24
Collaborator
@Lukasa
Collaborator

Oh man I meant to write on this and I didn't. I think we should keep it open until cygwin fixes itself. Based on the pain we had with header keys after that issue got closed, I suspect people don't look at closed issues.

@sigmavirus24
Collaborator

@Lukasa sometimes people don't even look at open ones =P. Either way you're correct. I wonder if we might want a special tag for this kind of issue (where there's a problem we cannot resolve but has some kind of fix available.

@gregersn

I'm amazed with this process/progress! Awesome work!

@sigmavirus24
Collaborator

It's all @ahmadia's fault. ;) Thank him since he found the relevant issues and possible solutions.

@joshfriend

I just tried @ahmadia's fix and got the following messages from the patch tool:

$ patch -d /usr/lib/python2.7 -p2 < uuid.patch
patching file uuid.py
Reversed (or previously applied) patch detected!  Assume -R? [n]
Apply anyway? [n]
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file uuid.py.rej

When I looked at /usr/lib/python2.7/uuid.py I noticed that the patch was already applied even though I had only just minutes before done a clean install of cygwin64. Importing uuid in python still segfaults even with the fix applied.

@ahmadia

Looks like the uuid patch landed on October 3rd

Not sure what's causing your problem, it sounds like something subtly wrong with the stack that we may not have caught upstream :(

Can you provide the backtrace?

@joshfriend

I was unable to get a Python stacktrace since the interpreter segfaults and exits immediately. I ran an import uuid statement with PDB and found where it died though:

> /usr/lib/python2.7/uuid.py(407)<module>()
-> if hasattr(lib, 'uuid_generate_random'):
(Pdb) s
--Call--
> /usr/lib/python2.7/ctypes/__init__.py(375)__getattr__()
-> def __getattr__(self, name):
(Pdb) s
> /usr/lib/python2.7/ctypes/__init__.py(376)__getattr__()
-> if name.startswith('__') and name.endswith('__'):
(Pdb) s
> /usr/lib/python2.7/ctypes/__init__.py(378)__getattr__()
-> func = self.__getitem__(name)
(Pdb) s
--Call--
> /usr/lib/python2.7/ctypes/__init__.py(382)__getitem__()
-> def __getitem__(self, name_or_ordinal):
(Pdb) s
> /usr/lib/python2.7/ctypes/__init__.py(383)__getitem__()
-> func = self._FuncPtr((name_or_ordinal, self))
(Pdb) s

And then it died.

Does that help?

@ahmadia

Sort of :) Can you check the hashes on uuid.py and delete the corresponding uuid.pyc and uuid.pyo files? Just want to make sure we're looking at the same code.

$ md5sum.exe /usr/lib/python2.7/uuid.py
3aa63cb0e74a2ce74d915616688233fd */usr/lib/python2.7/uuid.py
@ahmadia

A "clean install", you say? What's the output of:

cygcheck.exe -c libuuid1

If this is the problem, installing libuuid1 should fix this. Please report back so I can notify upstream.

@joshfriend

Looks like hashes match and libuuid1 is fine:

$ md5sum.exe /usr/lib/python2.7/uuid.py
3aa63cb0e74a2ce74d915616688233fd */usr/lib/python2.7/uuid.py
$ cygcheck.exe -c libuuid1
Cygwin Package Information
Package              Version        Status
libuuid1             2.21.2-1       OK

I've tried removing uuid.pyc/uuid.pyo before too and that didn't help either. Anything else you want me to try?

@sigmavirus24
Collaborator

@Creepr can you import uuid from the interactive prompt?

@joshfriend

@sigmavirus24 Nope. It just quits with no traceback.

@ahmadia

@Creepr It looks like something is wrong with your cyguuid-1.dll.

The following bit of code runs in uuid correctly for me. You can just execute directly in Python to test.

>>> import ctypes.util
>>> ctypes.util.find_library('uuid')
'cyguuid-1.dll'
>>> lib = ctypes.CDLL('cyguuid-1.dll')
>>> hasattr(lib, 'uuid_generate_random')
True
@ahmadia

Also, I hope this goes without saying, but you should have /usr/bin on your PATH and correct permissions or Python won't be able to load the DLL. If you get OSError: Permission denied, you need to make sure those are both right.

@joshfriend

My output looks slightly different when running find_library() (returns None):

>>> import ctypes.util
>>> ctypes.util.find_library('uuid')
>>> lib = ctypes.CDLL('cyguuid-1.dll')
>>> hasattr(lib, 'uuid_generate_random')
True

I tried re-installing the uuid package with the cygwin installer and also a rebaseall. Neither worked.

/usr/bin is in my PATH with 755 permissions.

Looks like I can hack the uuid module for now to have the DLL names hardcoded:

for libname in ['cyguuid-1.dll', 'cygwin1.dll']:
    try:
        lib = ctypes.CDLL(libname)
    except:
        continue
    if hasattr(lib, 'uuid_generate_random'):
        _uuid_generate_random = lib.uuid_generate_random

Doing that seems to have fixed my issues, though it is probably not an ideal solution.

@ahmadia

Weird. Is anybody else able to test with a Cygwin64 box?

@joshfriend

I just had a friend at work try it out. She had the same results as I did (find_library('uuid') returns None).

@ahmadia

I'm sorry, this isn't obvious at all, but I suspect you need libuuid-devel. Looking at the source code for find_library:

elif sys.platform == "cygwin":
    def find_library(name):
        for libdir in ['/usr/lib', '/usr/local/lib']:
            for libext in ['lib%s.dll.a' % name, 'lib%s.a' % name]:
                implib = os.path.join(libdir, libext)
                if not os.path.exists(implib):
                    continue
                cmd = "dlltool -I " + implib + " 2>/dev/null"
                res = os.popen(cmd).read().replace("\n","")
                if not res:
                    continue
                return res
        return None

It's looking for the stub DLL files needed for generating the actual libraries (that are populated by the -devel versions of the Cygwin packages).

@sigmavirus24
Collaborator

Hey guys,

I appreciate you trying to work this out but if you could work somewhere else and then post the steps for debugging and resolution in one concise comment. I'd rather not mute this thread but I'd also rather not have a ton of email to read when I get home tonight.

@joshfriend

Sorry @sigmavirus24, I promise to stop after this post. @ahmadia: you can email me at the address listed on my profile.

@sigmavirus24
Collaborator

No need to apologize @Creepr. I'm really interested in what the final result is that you two find. I just get a ton of email and this is more of a Cygwin issue than a requests issue.

@jvstein

I had the same problem and tracked it back to the usage of dlltool in find_library.

cmd = "dlltool -I " + implib + " 2>/dev/null"
res = os.popen(cmd).read().replace("\n","")
if not res:
    continue

Installing the 'binutils' cygwin package solved it for me.

@joshfriend

Like @jvstein I traced my issue back to dlltool:

$ dlltool -I /usr/lib/libuuid.dll.a
dlltool: Unable to open object file: cyguuid-1.dll: No such file or directory

However, I checked and did have binutils installed. I had several co-workers with the same issue as me run the latest cygwin setup.exe again and reinstall binutils. Unfortunately, this did not fix the issue for them (or me). I did also verify that we had the same versions of installed packages using cygcheck -s. I then removed cygwin and reinstalled from scratch, which fixed the issue for me. It may be worth noting that my previous cygwin installation was performed before the Oct 3rd patch to uuid.py.

I suppose the resolution steps are:

  1. Verify that python has the patch that was previously mentioned
  2. Update/Reinstall/Install binutils as needed
  3. As a last resort, reinstall cygwin64 from scratch
@mstave

Installing libuuid-devel on Win7 Cygwin64 fixed it for me.

@tinkerer

I encountered this error while trying to pip install django-allauth on Windows 7 in Cygwin64. libuuid-devel on Cygwin64 also fixed it for me

@Lukasa
Collaborator

Right, in that case I proclaim this issue "Not our fault" and am closing it. =)

Thank you all for your contributions and insight!

@Lukasa Lukasa closed this
@jdunn-big

Wow! Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.