RuntimeError: Compression requires the (missing) zlib module (mavericks) on python install

[ohardy]

$ brew install python
==> Downloading http://www.python.org/ftp/python/2.7.5/Python-2.7.5.tar.bz2
Already downloaded: /Library/Caches/Homebrew/python-2.7.5.tar.bz2
==> ./configure --prefix=/usr/local/Cellar/python/2.7.5 --enable-ipv6 --datarootdir=/usr/local/Cellar/python/2.7.5/share --datadir=/usr/local/Cellar/python/2
==> make
==> make install PYTHONAPPSDIR=/usr/local/Cellar/python/2.7.5
==> make frameworkinstallextras PYTHONAPPSDIR=/usr/local/Cellar/python/2.7.5/share/python
==> Downloading https://pypi.python.org/packages/source/d/distribute/distribute-0.6.45.tar.gz
Already downloaded: /Library/Caches/Homebrew/distribute-0.6.45.tar.gz
==> /usr/local/Cellar/python/2.7.5/bin/python2 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/2.7.5/bin --ins
File "/private/tmp/UNKNOWN-lV8C/distribute-0.6.45/setuptools/command/bdist_egg.py", line 540, in make_zipfile
z = zipfile.ZipFile(zip_filename, mode, compression=compression)
File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 732, in init
"Compression requires the (missing) zlib module"
RuntimeError: Compression requires the (missing) zlib module

[samueljohn]

Can you temporarily test to locally add depends_on 'xz' => :recommended # for the lzma module to the python formula by brew edit python?

I guess, we have to add this xz to the deps. I should do this for Python 3.3, too, but didn't know pyhton 2.7.5 also needed this.

[samueljohn]

Note on Mavericks: As of now, no homebrew core-member has enrolled for the Apple Developer Programm.

[ohardy]

No change with xz installed

[samueljohn]

Ok, then revert your local changes. Sorry, I can just do wild guessing.

Can you post the link to a gist.github.com with the output of brew install python --verbose, I want to look for some lines like these:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             dl                 imageop         
linuxaudiodev      ossaudiodev        spwd            
sunaudiodev                                           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The message as shown above for me on 10.8.3 is perfectly fine because these modules don't build on Mac but I assume on your system other bits are missing and later to install distribute, the required module to unpack distribute is missing. Perhaps on 10.9 some lib has been removed/renamed?

[toroidal-code]

To get python2/python3 and pip3 installed and working on my system (10.9) I had to install and brew link --force both zlib and openssl. I think this confirms samueljohn's idea that libraries are renamed, or at least now located somewhere the configure scripts for packages aren't able to find them normally.

[ohardy]

Python build finished, but the necessary bits to build these modules were not found:
_bsddb _ssl bsddb185
dbm dl imageop
linuxaudiodev nis ossaudiodev
spwd sunaudiodev zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

[samueljohn]

So zlib and _ssl and probably bsddb185 should be build but are not. Thanks for the info but I can't fix it right now because not having a Dev ID.

[mistydemeo]

Another user confirmed that the zlib dylib (/usr/lib/libz.1.2.5.dylib) exists on 10.9. What about the header? Is there a /usr/include/zlib.h?

[samueljohn]

Misty is doing a blind remote surgery here :-)

[sirn]

I can confirm that /usr/lib/libz.1.2.5.dylib is present, but I can't found /usr/include/zlib.h in my Mavericks installation. (In fact, the whole /usr/include is not present at all.)

[mistydemeo]

(In fact, the whole /usr/include is not present at all.)

Uh, hm. You have Xcode 5.0 installed? Can you retry installing?

[sirn]

Uh, hm. You have Xcode 5.0 installed? Can you retry installing?

Trashed prefs and reinstalled Xcode 5 DP. Same result.

[samueljohn]

perhaps Xcode 5.0 is still kind of broken in that regard ...

[mistydemeo]

Afraid we can't offer support for this right now as we don't have access to 10.9 or Xcode 5.0.

[adamv]

Reminder that problems in pre-release versions of OS X should be reported to Apple; there were some weird libs/headers issues in 10.8 previews that were fixed in the final release.

[samueljohn]

❗

[sirn]

Just some note, it appears that CLI tools is now installed as part of the OS itself rather than with Xcode. I've found that clang/clang++ binary (and many binaries/lib that were present in CLI tools, but not all of it) now being installed as part of Essentials.pkg inside Mavericks' InstallESD.dmg. The whole /usr/include is not present in InstallESD.

[mistydemeo]

Please report it to Apple.

[sirn]

That wasn't meant to be a bug report, just some note that CLI tools is now part of OS itself, rather than Xcode. Just to save someone's trouble if that person is trying to reinstall Xcode thinking something has gone wrong with his Xcode installation. :(

[mistydemeo]

Right, it's the /usr/include thing that should get reported.

Are there any release notes in Xcode or Mavericks that cover include locations, etc.?

[sirn]

No, I don't see any release notes in Xcode nor in Mavericks.
I've open a bug for this at rdar://14129396

Thanks.

[toroidal-code]

This is where I found zlib.h:

./Applications/Xcode5-DP.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
./Applications/Xcode5-DP.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/zlib.h

I can verify that this path does have the standard headers:

AssertMacros.h          ifaddrs.h               ranlib.h
Availability.h          inttypes.h              readline/
AvailabilityInternal.h  iso646.h                readpassphrase.h
AvailabilityMacros.h    kern/                   regex.h
Block.h                 krb5/                   removefile.h
CommonCrypto/           krb5.h                  resolv.h
ConditionalMacros.h     langinfo.h              rpc/
MacTypes.h              launch.h                rpcsvc/
NSSystemDirectories.h   lber.h                  rune.h
TargetConditionals.h    lber_types.h            runetype.h
Xplugin.h               ldap.h                  sandbox.h
_locale.h               ldap_cdefs.h            sasl/
_structs.h              ldap_features.h         sched.h
_types/                 ldap_schema.h           search.h
_types.h                ldap_utf8.h             secure/
_wctype.h               ldif.h                  security/
_xlocale.h              libc.h                  semaphore.h
aio.h                   libcharset.h            servers/
aliasdb.h               libexslt/               setjmp.h
alloca.h                libgen.h                sgtty.h
apache2/                libkern/                signal.h
apr-1/                  libproc.h               slapi-plugin.h
ar.h                    libunwind.h             spawn.h
architecture/           libxml2/                sqlite3.h
arpa/                   libxslt/                sqlite3ext.h
asl.h                   limits.h                stab.h
assert.h                localcharset.h          standards.h
bitstring.h             locale.h                stdbool.h
bootparams.h            mach/                   stddef.h
bsm/                    mach-o/                 stdint.h
bzlib.h                 mach_debug/             stdio.h
c++/                    machine/                stdlib.h
cache.h                 malloc/                 strhash.h
cache_callbacks.h       math.h                  string.h
checkint.h              membership.h            stringlist.h
com_err.h               memcached/              strings.h
complex.h               memory.h                struct.h
copyfile.h              menu.h                  sys/
cpio.h                  miscfs/                 sysexits.h
crt_externs.h           module.map              syslog.h
ctype.h                 monetary.h              tar.h
cups/                   monitor.h               tcl.h@
curl/                   mpool.h                 tclDecls.h@
curses.h                nameser.h               tclPlatDecls.h@
cxxabi.h                nc_tparm.h              tclTomMath.h@
db.h                    ncurses.h@              tclTomMathDecls.h@
default_pager/          ncurses_dll.h           term.h
device/                 ndbm.h                  term_entry.h
dirent.h                net/                    termcap.h
disktab.h               net-snmp/               termios.h
dispatch/               netdb.h                 tic.h
dlfcn.h                 netinet/                tidy/
dns.h                   netinet6/               time.h
dns_sd.h                netkey/                 timeconv.h
dns_util.h              nfs/                    tk.h@
dtrace.h                nl_types.h              tkDecls.h@
editline/               nlist.h                 tkIntXlibDecls.h@
err.h                   notify.h                tkMacOSX.h@
errno.h                 notify_keys.h           tkPlatDecls.h@
eti.h                   ntsid.h                 ttyent.h
execinfo.h              objc/                   tzfile.h
expat.h                 odmodule/               ucontext.h
expat_external.h        openssl/                ulimit.h
fcntl.h                 os/                     unctrl.h
fenv.h                  panel.h                 unistd.h
ffi/                    paths.h                 unwind.h
fmtmsg.h                pcap/                   util.h
fnmatch.h               pcap-bpf.h              utime.h
form.h                  pcap-namedb.h           utmp.h
fsproperties.h          pcap.h                  utmpx.h
fstab.h                 pexpert/                uuid/
fts.h                   php/                    vfs/
ftw.h                   poll.h                  vis.h
get_compat.h            printerdb.h             vproc.h
gethostuuid.h           printf.h                wchar.h
getopt.h                profile.h               wctype.h
glob.h                  protocols/              wordexp.h
grp.h                   pthread.h               xar/
gssapi/                 pthread_impl.h          xlocale/
gssapi.h                pthread_spis.h          xlocale.h
hfs/                    pwd.h                   xpc/
histedit.h              python2.5@              zconf.h
i386/                   python2.6@              zlib.h
iconv.h                 python2.7@

[maxbelanger]

Xcode 5 no longer installs headers in /usr/include. It seems we're supposed to use xcrun --show-sdk-path to find the active SDK.

[clemensg]

Does anybody know why Homebrew is able to find the correct include paths on 10.8 but not on 10.9 ?
The Xcode.prefix and the sdk_path method already give back the correct location of usr/include, but the compilation of cmake, etc. fails anyway..

Any ideas / guesses?

[mistydemeo]

If the user has the command line tools installed on 10.7 and 10.8, then the headers are installed to the standard /usr/include locations. On 10.9 that's different.

[clemensg]

Yes, but somebody mentioned, that even if the user did not install the command line tools, it works on 10.8.
Are you positive that Homebrew does only look in /usr/include and does not use the result of Xcode.prefix + sdk_path ?

[samueljohn]

That is because on 10.8 we worked hard to make homebrew (99%) work with either the CLT only and Xcode-only.
The 1% is due to some formulae hard-coding paths to certain includes.
Perhaps on 10.9 the "Xcode-only" code is not triggered correctly.

@clemensg you can brew irb to dig into MacOS.sdk_path and related methods.

[clemensg]

@samueljohn Thank you!

Should the same strategy, which is used for Xcode-only on 10.8, now be used as default on 10.9 instead of using /usr/include ?

[jacknagel]

Just to reiterate what Adam said earlier:

Reminder that problems in pre-release versions of OS X should be reported to Apple; there were some weird libs/headers issues in 10.8 previews that were fixed in the final release.

So if changes are made for 10.9 here, let's try to keep them (a) minimal and (b) well-documented, as it is not unlikely the situation will change by the time the final release is made.

[clemensg]

Until Homebrew pulls my pull request or fixes it in any other way, you could all use my mavericks branch or apply the last commit as patch to your local repository: https://github.com/clemensg/homebrew/tree/mavericks
Fixes this issue.

[theycallmeswift]

For anyone else who encounters this issue, the temporary fix is:

$ brew link --force zlib
$ brew link --force openssl

[sirn]

Apple developer has informed me that rdar://14129396 has been "fixed" as of Xcode 5 DP5. You will need to install CLI tools for Xcode 5 (which make /usr/include available again) with:

xcode-select --install

Python installation went through just fine after doing so (not sure if this is a result of any changes in Homebrew though...)

[joemaller]

I bumped into a similar (the same?) problem setting up a handful of clean 10.8.4 (current Mountain Lion) macs. Homebrew can't find zlib.h with just Xcode 4.6.3 installed. Command line tools must be installed to populate /usr/include. Without that, python fails here:

building dbm using ndbm
error: /usr/include/zlib.h: No such file or directory
make: *** [sharedmods] Error 1

As noted above, there are several copies of zlib.h littered about inside the Xcode bundle SDKs, before installing the command line tools, I copied the last of the following into /usr/include/ and python installed correctly.

Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.1.sdk/usr/include/zlib.h
Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator6.1.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator6.1.sdk/usr/include/zlib.h
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/usr/include/zlib.h
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/zlib.h

[samueljohn]

@joemaller that is strange, as I can install python with the zlib module on 10.8.4 with Xcode-only (no CLTs).
Only difference I see is that I tested in a homebrew not at /usr/local.

[adamv]

@joemaller if this is happening on Mountain Lion, please open a separate issue, with full troubleshooting information; this issue is specifically about 10.9 and though the zlib error may seem the same, it almost certainly has a different cause (since Python is known to compile on 10.8).

[adamv]

Closing since @sirn reports this may be fixed in DP5. Certainly after 10.9 is released we'll update Hombrew to work well with it.

[johngoodleaf]

I don't think it's necessary to reopen this issue, but I encountered the same problem using GM seeds of both os 10.9 and Xcode (5.0.1). I ran: xcode-select --install manually (per sirn's comment above) and that fixed the problem.

[zellux]

That works for me too, thanks @johngoodleaf !

[AdamCanady]

xcode-select --install works for me on 10.9 as well, thanks!

[endyul]

xcode-select --install works for me on 10.9, thanks

[johngoodleaf]

Maybe we should reopen this. Looks like the basic problem occurs with some frequency. Could it warrant at least an advisory note when running brew?

[MikeMcQuaid]

xcode-select --install is recommended by brew install and brew doctor. Glad this is helping people but we don't need more comments saying it worked, thanks.

[johngoodleaf]

Sorry! I didn't check before sounding off.

[MikeMcQuaid]

@johngoodleaf No worries.

[gabrielstuff]

Well, sorry to bump here, but finding it was really hard and brew doctor never mentioned something easy cake as :
xcode-select --install
So first thanks ! and second, we might want fix this and make people find it easily !

Thanks !

[msaffitz]

👍 another bump here-- this took quite a bit to track down. Thanks for the fix, and it'd be awesome to have it in brew doctor. (My issue manifested as an error installing MacVim, which then was traced back to Python, which in turn hit this issue)

[pjohalloran]

I can also confirm installing latest command line tools for xcode on maverick fixes this issue.

[jcampbell05]

Issue crops up again in Yosemite though :) I am trying the force link workaround as mentioned above.

[jcampbell05]

force link solution works in Yosemite :)

[tnguyen14]

I just installed Yosemite and use the force link solution, but that didn't fix the issue for me. does anyone know what else could help?

[verkholantsev]

brew link --force zlib
brew link --force openssl

This does not work in Yosemite for me too.

[qipeng]

Yosemite users, if your brew doctor complains about missing Xcode command line tools, try xcode-select --install to install it before trying any other solution (it doesn't automatically come with Xcode, and this might not have worked in Mavericks but surprisingly it works now again in Yosemite).