bug(python): unable to install on mac Darwin 22.4.0 arm64 due to dynamic linking to 'liblzma'

[universalmind303]

I believe this was incorrectly closed in lancedb/lancedb#919 as it's still an issue on main in the rust binary and they python dylib

LanceDB version

latest e022688ac7f23a99320a6ff9b56ba97c38f29c06

What happened?

# create a fresh venv
> python3 -m virtualenv .venv
# activate the venv
> overlay use .venv/bin/activate.nu   

# install lancedb
> pip install pandas lancedb

# try to import it 
> python -c "import lancedb"

ImportError: dlopen(/Users/me/Development/glaredb/lance-test/.venv/lib/python3.11/site-packages/lance/lance.abi3.so, 0x0002): Library not loaded: /opt/homebrew/opt/xz/lib/liblzma.5.dylib

Reason: tried: '/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file), '/opt/homebrew/opt/xz/lib/liblzma.5.dylib' (no such file)

Are there known steps to reproduce?

> git clone https://github.com/lancedb/lance
> cd python
> maturin develop
>  otool -L target/debug/liblance.dylib | grep 'liblzma'   
        /usr/local/lib/liblzma.5.dylib (compatibility version 10.0.0, current version 10.5.0)

[eddyxu]

Thanks for reporting @universalmind303 . Lemme take a look, i don't recall that we use xz as dependency. Lemme check where does it come from.

Btw, do you use homebrew python, or this is system installed python3?

[eddyxu]

Update:

• The release build picks up xz from the homebrew installed on the GHA.

A workaround could be brew install xz to get around this issue. I am looking to fix the build now.

[universalmind303]

@eddyxu I tried installing xz but it seems that it is locked to that specific path /opt/homebrew/opt/xz/lib/liblzma.5.dylib. My default homebrew doesn't install it there, so it still fails.

I even tried manually moving the dylib to that path and it still fails to run

 ls /opt/homebrew/opt/xz/lib
liblzma.5.dylib

[fcakyon]

I am also having the same issue. This issue starts after lancedb==0.5.2 I believe. I'm having the issue on an MBA M2 with the Python 3.10 miniconda environment.

brew install xz did not work.

[fcakyon]

Is there any updates on this issue? @westonpace

[wjones127]

Perhaps we can add the static feature and then it should dynamically link? https://github.com/alexcrichton/xz2-rs/blob/1a82c40d6d80171b7df328aea43b7054acd10c44/Cargo.toml#L33

[westonpace]

Thanks @wjones127