Unable to pip3 install ratarmount on MacOS 11.4 using a MacBook Air M1

[shawwn]

The problem seems to be something related to indexed-zstd.

I git clone'd ratarmount, then ran pip3 install -e .. Here's the output:

:~/ml/ratarmount-clean$ pip3 install -e .
Obtaining file:///Users/shawn/ml/ratarmount-clean
Collecting fusepy
  Using cached fusepy-3.0.1-py3-none-any.whl
Collecting indexed_bzip2>=1.3.0
  Using cached indexed_bzip2-1.3.0-cp39-cp39-macosx_11_0_arm64.whl
Collecting indexed_gzip>=1.6.3
  Using cached indexed_gzip-1.6.3-cp39-cp39-macosx_11_0_arm64.whl (584 kB)
Requirement already satisfied: python-xz>=0.1.2 in /opt/homebrew/lib/python3.9/site-packages (from ratarmount==0.9.0) (0.1.2)
Requirement already satisfied: rarfile>=4.0 in /opt/homebrew/lib/python3.9/site-packages (from ratarmount==0.9.0) (4.0)
Collecting indexed_zstd>=1.2.2
  Using cached indexed_zstd-1.3.1.tar.gz (60 kB)
Building wheels for collected packages: indexed-zstd
  Building wheel for indexed-zstd (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/homebrew/opt/python@3.9/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-install-7wfd6b92/indexed-zstd_a2f537afce334a818672cb3d4034280a/setup.py'"'"'; __file__='"'"'/private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-install-7wfd6b92/indexed-zstd_a2f537afce334a818672cb3d4034280a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-wheel-jdfu418w
       cwd: /private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-install-7wfd6b92/indexed-zstd_a2f537afce334a818672cb3d4034280a/
  Complete output (18 lines):
  running bdist_wheel
  running build
  running build_py
  file indexed_zstd.py (for module indexed_zstd) not found
  file indexed_zstd.py (for module indexed_zstd) not found
  running build_clib
  building 'zstd_zeek' library
  creating build
  creating build/temp.macosx-11-arm64-3.9
  creating build/temp.macosx-11-arm64-3.9/indexed_zstd
  creating build/temp.macosx-11-arm64-3.9/indexed_zstd/libzstd-seek
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -c indexed_zstd/libzstd-seek/zstd-seek.c -o build/temp.macosx-11-arm64-3.9/indexed_zstd/libzstd-seek/zstd-seek.o
  In file included from indexed_zstd/libzstd-seek/zstd-seek.c:19:
  indexed_zstd/libzstd-seek/zstd-seek.h:20:10: fatal error: 'zstd.h' file not found
  #include <zstd.h>
           ^~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for indexed-zstd
  Running setup.py clean for indexed-zstd
Failed to build indexed-zstd
Installing collected packages: indexed-zstd, indexed-gzip, indexed-bzip2, fusepy, ratarmount
    Running setup.py install for indexed-zstd ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/homebrew/opt/python@3.9/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-install-7wfd6b92/indexed-zstd_a2f537afce334a818672cb3d4034280a/setup.py'"'"'; __file__='"'"'/private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-install-7wfd6b92/indexed-zstd_a2f537afce334a818672cb3d4034280a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-record-qyq_93tp/install-record.txt --single-version-externally-managed --compile --install-headers /opt/homebrew/include/python3.9/indexed-zstd
         cwd: /private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-install-7wfd6b92/indexed-zstd_a2f537afce334a818672cb3d4034280a/
    Complete output (18 lines):
    running install
    running build
    running build_py
    file indexed_zstd.py (for module indexed_zstd) not found
    file indexed_zstd.py (for module indexed_zstd) not found
    running build_clib
    building 'zstd_zeek' library
    creating build
    creating build/temp.macosx-11-arm64-3.9
    creating build/temp.macosx-11-arm64-3.9/indexed_zstd
    creating build/temp.macosx-11-arm64-3.9/indexed_zstd/libzstd-seek
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -c indexed_zstd/libzstd-seek/zstd-seek.c -o build/temp.macosx-11-arm64-3.9/indexed_zstd/libzstd-seek/zstd-seek.o
    In file included from indexed_zstd/libzstd-seek/zstd-seek.c:19:
    indexed_zstd/libzstd-seek/zstd-seek.h:20:10: fatal error: 'zstd.h' file not found
    #include <zstd.h>
             ^~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/homebrew/opt/python@3.9/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-install-7wfd6b92/indexed-zstd_a2f537afce334a818672cb3d4034280a/setup.py'"'"'; __file__='"'"'/private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-install-7wfd6b92/indexed-zstd_a2f537afce334a818672cb3d4034280a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/cz/4fmt40y176988qbnggwg9rc00000gn/T/pip-record-qyq_93tp/install-record.txt --single-version-externally-managed --compile --install-headers /opt/homebrew/include/python3.9/indexed-zstd Check the logs for full command output.
WARNING: You are using pip version 21.1.1; however, version 21.2.4 is available.
You should consider upgrading via the '/opt/homebrew/opt/python@3.9/bin/python3.9 -m pip install --upgrade pip' command.

Then I ran git clone https://github.com/martinellimarco/indexed_zstd and tried python3 setup.py develop:

$ python3 setup.py develop
running develop
running egg_info
writing indexed_zstd.egg-info/PKG-INFO
writing dependency_links to indexed_zstd.egg-info/dependency_links.txt
writing top-level names to indexed_zstd.egg-info/top_level.txt
file indexed_zstd.py (for module indexed_zstd) not found
adding license file 'LICENSE' (matched pattern 'LICEN[CS]E*')
reading manifest file 'indexed_zstd.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.py' under directory 'indexed_zstd'
writing manifest file 'indexed_zstd.egg-info/SOURCES.txt'
running build_ext
building 'indexed_zstd' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I. -I/opt/homebrew/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c indexed_zstd/indexed_zstd.cpp -o build/temp.macosx-11-arm64-3.9/indexed_zstd/indexed_zstd.o -std=c++11 -O3 -DNDEBUG
In file included from indexed_zstd/indexed_zstd.cpp:673:
indexed_zstd/ZSTDReader.hpp:175:5: warning: 'seek' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
    seek( long long int offset,
    ^
indexed_zstd/FileReader.hpp:33:5: note: overridden virtual function is here
    seek( long long int offset,
    ^
1 warning generated.
clang++ -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk build/temp.macosx-11-arm64-3.9/indexed_zstd/indexed_zstd.o -L/opt/homebrew/lib -L/opt/homebrew/opt/openssl@1.1/lib -L/opt/homebrew/opt/sqlite/lib -Lbuild/temp.macosx-11-arm64-3.9 -lm -lzstd -lzstd_zeek -o build/lib.macosx-11-arm64-3.9/indexed_zstd.cpython-39-darwin.so
ld: library not found for -lzstd_zeek
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang++' failed with exit code

Then I tried:

cd indexed_zstd/libzstd-seek
mkdir build
cd build
cmake ..
make

and got:

$ make
[ 16%] Building C object CMakeFiles/zstd-seek.dir/zstd-seek.c.o
In file included from /Users/shawn/ml/indexed_zstd/indexed_zstd/libzstd-seek/zstd-seek.c:19:
/Users/shawn/ml/indexed_zstd/indexed_zstd/libzstd-seek/zstd-seek.h:20:10: fatal error: 'zstd.h' file not found
#include <zstd.h>
         ^~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/zstd-seek.dir/zstd-seek.c.o] Error 1
make[1]: *** [CMakeFiles/zstd-seek.dir/all] Error 2
make: *** [all] Error 2

So then I decided to open this issue and go to sleep. :)

[shawwn]

By the way, brew install zstd reports that it's already installed:

Warning: zstd 1.5.0 is already installed and up-to-date.
To reinstall 1.5.0, run:
  brew reinstall zstd

So I'm not sure if something is failing to find the proper paths, or what to try next.

[mxmlnkn]

That's weird. I have a macOS CI for ratarmount and I'm also just installing zstd via brew there. Maybe zstd.h is not in a default include path on your system? I don't see indexed_zstd adding any specific include paths for zstd.

I would quick "fix" this by removing zstd from the required dependencies of ratarmount on macOS until there are wheels for indexed_zstd on macOS. Or do you want to use ratarmount with zstd support?

[mxmlnkn]

As for what you can try to get ratarmount to work in the meantime. You can install it without dependencies: pip3 install --user --no-deps ratarmount. And if you need specific compressions install those as needed, e.g., pip3 install --user fusepy indexed_gzip indexed_bzip2 rarfile.

[mxmlnkn]

I made the indexed_zstd dependency optional on macOS and Windows in 0.9.1. So, pip install should work for now. I'll give making it a required dependency another try when there are wheels for those systems available.