fatal error: 'leveldb/db.h' file not found

[edsu]

I'm running into this install error with pip on OSX 10.14.3 (Mojave). It looks like the compile step can't find leveldb/db.h but I brew installed leveldb with Homebrew.

% pip install plyvel
Collecting plyvel
  Downloading https://files.pythonhosted.org/packages/b3/1d/05e3555abddb49b9a2f3aa1268bec4933bf377ffcad3c1b2630fcfa597eb/plyvel-1.0.5.tar.gz (143kB)
     |████████████████████████████████| 153kB 4.9MB/s 
Building wheels for collected packages: plyvel
  Building wheel for plyvel (setup.py) ... error
  ERROR: Complete output from command /Users/edsu/.local/share/virtualenvs/twarc-kn8bqGwd/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/6s/clpsqb7n5y37xtscsn39kygw0000gp/T/pip-install-5z0oxmk7/plyvel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/6s/clpsqb7n5y37xtscsn39kygw0000gp/T/pip-wheel-7dfy2jjs --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.14-x86_64-3.7
  creating build/lib.macosx-10.14-x86_64-3.7/plyvel
  copying plyvel/_version.py -> build/lib.macosx-10.14-x86_64-3.7/plyvel
  copying plyvel/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/plyvel
  warning: build_py: byte-compiling is disabled, skipping.
  
  running build_ext
  building 'plyvel._plyvel' extension
  creating build/temp.macosx-10.14-x86_64-3.7
  creating build/temp.macosx-10.14-x86_64-3.7/plyvel
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c plyvel/_plyvel.cpp -o build/temp.macosx-10.14-x86_64-3.7/plyvel/_plyvel.o -Wall -g -mmacosx-version-min=10.7 -stdlib=libc++
  plyvel/_plyvel.cpp:589:10: fatal error: 'leveldb/db.h' file not found
  #include "leveldb/db.h"
           ^~~~~~~~~~~~~~
  1 error generated.
  error: command 'clang' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for plyvel
  Running setup.py clean for plyvel
Failed to build plyvel
Installing collected packages: plyvel
  Running setup.py install for plyvel ... error
    ERROR: Complete output from command /Users/edsu/.local/share/virtualenvs/twarc-kn8bqGwd/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/6s/clpsqb7n5y37xtscsn39kygw0000gp/T/pip-install-5z0oxmk7/plyvel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6s/clpsqb7n5y37xtscsn39kygw0000gp/T/pip-record-funxcc5v/install-record.txt --single-version-externally-managed --compile --install-headers /Users/edsu/.local/share/virtualenvs/twarc-kn8bqGwd/bin/../include/site/python3.7/plyvel:
    ERROR: running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14-x86_64-3.7
    creating build/lib.macosx-10.14-x86_64-3.7/plyvel
    copying plyvel/_version.py -> build/lib.macosx-10.14-x86_64-3.7/plyvel
    copying plyvel/__init__.py -> build/lib.macosx-10.14-x86_64-3.7/plyvel
    warning: build_py: byte-compiling is disabled, skipping.
    
    running build_ext
    building 'plyvel._plyvel' extension
    creating build/temp.macosx-10.14-x86_64-3.7
    creating build/temp.macosx-10.14-x86_64-3.7/plyvel
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c plyvel/_plyvel.cpp -o build/temp.macosx-10.14-x86_64-3.7/plyvel/_plyvel.o -Wall -g -mmacosx-version-min=10.7 -stdlib=libc++
    plyvel/_plyvel.cpp:589:10: fatal error: 'leveldb/db.h' file not found
    #include "leveldb/db.h"
             ^~~~~~~~~~~~~~
    1 error generated.
    error: command 'clang' failed with exit status 1
    ----------------------------------------
ERROR: Command "/Users/edsu/.local/share/virtualenvs/twarc-kn8bqGwd/bin/python3.7 -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/6s/clpsqb7n5y37xtscsn39kygw0000gp/T/pip-install-5z0oxmk7/plyvel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6s/clpsqb7n5y37xtscsn39kygw0000gp/T/pip-record-funxcc5v/install-record.txt --single-version-externally-managed --compile --install-headers /Users/edsu/.local/share/virtualenvs/twarc-kn8bqGwd/bin/../include/site/python3.7/plyvel" failed with error code 1 in /private/var/folders/6s/clpsqb7n5y37xtscsn39kygw0000gp/T/pip-install-5z0oxmk7/plyvel/

Here are the Homebrew details:

(twarc) bash-3.2$ brew info leveldb
leveldb: stable 1.21 (bottled)
Key-value storage library with ordered mapping
https://github.com/google/leveldb/
/usr/local/Cellar/leveldb/1.21 (29 files, 912.0KB) *
  Poured from bottle on 2019-04-30 at 17:34:18
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/leveldb.rb
==> Dependencies
Build: cmake ✔
Required: gperftools ✔, snappy ✔
==> Analytics
install: 4,341 (30 days), 7,979 (90 days), 27,704 (365 days)
install_on_request: 3,125 (30 days), 6,078 (90 days), 22,528 (365 days)
build_error: 0 (30 days)

And I can see the header files are installed, I guess setup.py isn't looking for them in /usr/local/include/leveldb/?

(twarc) bash-3.2$ ls -l /usr/local/include/leveldb/
total 184
-rw-r--r--  1 edsu  staff  12162 Mar 29 17:37 c.h
-rw-r--r--  1 edsu  staff   4134 Mar 29 17:37 cache.h
-rw-r--r--  1 edsu  staff   2400 Mar 29 17:37 comparator.h
-rw-r--r--  1 edsu  staff   6856 Mar 29 17:37 db.h
-rw-r--r--  1 edsu  staff    950 Mar 29 17:37 dumpfile.h
-rw-r--r--  1 edsu  staff  14512 Mar 29 17:37 env.h
-rw-r--r--  1 edsu  staff    911 Mar 29 17:37 export.h
-rw-r--r--  1 edsu  staff   2998 Mar 29 17:37 filter_policy.h
-rw-r--r--  1 edsu  staff   3985 Mar 29 17:37 iterator.h
-rw-r--r--  1 edsu  staff   7394 Mar 29 17:37 options.h
-rw-r--r--  1 edsu  staff   3252 Mar 29 17:37 slice.h
-rw-r--r--  1 edsu  staff   3961 Mar 29 17:37 status.h
-rw-r--r--  1 edsu  staff   3003 Mar 29 17:37 table.h
-rw-r--r--  1 edsu  staff   3432 Mar 29 17:37 table_builder.h
-rw-r--r--  1 edsu  staff   2519 Mar 29 17:37 write_batch.h

[schmty]

Hey @edsu this PR #99 resolves the issue, it will be available on pypi soon 😄

[trungtin]

for people with M1 Mac and leveldb is installed, add homebrew path (to .bashrc or .zshrc) so clang linker can find the header

export LIBRARY_PATH="$LIBRARY_PATH:$(brew --prefix)/lib"
export CPATH="$CPATH:$(brew --prefix)/include"

[SatoriHoshiAiko]

This last comment was a game changer for ElectrumX on macOS. I'm adding these CPATH and LIBRARY_PATH for brew will just fix so many other things. Thank-you AAA+

[abhi-jha]

How do you provide a cutom path for leveldb headers? I am compiling leveldb from source on a custom path as well.

[maxrys]

Worked for me:

brew install leveldb
python3 -m pip install plyvel

[eccstartup]

for people with M1 Mac and leveldb is installed, add homebrew path (to .bashrc or .zshrc) so clang linker can find the header

export LIBRARY_PATH="$LIBRARY_PATH:$(brew --prefix)/lib"
export CPATH="$CPATH:$(brew --prefix)/include"

This works when installing, but not for Python import this library.

[abhi-jha]

You need to provide DYLD_LIBRARY_PATH=<path_to_lib> python my_scipt.py