datasketches cannot be imported on Apple M1

[davideanastasia]

I am currently experiencing a "wrong architecture" problem when I tried to use datasketches on my M1 macbook pro:

davide-anastasia@Davides-MBP ~ % mkdir datasketches
davide-anastasia@Davides-MBP ~ % cd datasketches
davide-anastasia@Davides-MBP datasketches % pipenv shell
Creating a virtualenv for this project...
Pipfile: /Users/davide-anastasia/datasketches/Pipfile
Using /opt/homebrew/bin/python3 (3.9.7) to create virtualenv...
⠸ Creating virtual environment...created virtual environment CPython3.9.7.final.0-64 in 169ms
  creator CPython3Posix(dest=/Users/davide-anastasia/.local/share/virtualenvs/datasketches-yWoxoifi, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/davide-anastasia/Library/Application Support/virtualenv)
    added seed packages: pip==21.3.1, setuptools==60.2.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment!
Virtualenv location: /Users/davide-anastasia/.local/share/virtualenvs/datasketches-yWoxoifi
Creating a Pipfile for this project...
Launching subshell in virtual environment...
 . /Users/davide-anastasia/.local/share/virtualenvs/datasketches-yWoxoifi/bin/activate
davide-anastasia@Davides-MBP datasketches %  . /Users/davide-anastasia/.local/share/virtualenvs/datasketches-yWoxoifi
/bin/activate
(datasketches) davide-anastasia@Davides-MBP datasketches % pipenv install datasketches
Installing datasketches...
Adding datasketches to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (99e97b)!
Installing dependencies from Pipfile.lock (99e97b)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
(datasketches) davide-anastasia@Davides-MBP datasketches % python3 -c 'import datasketches'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/davide-anastasia/.local/share/virtualenvs/datasketches-yWoxoifi/lib/python3.9/site-packages/datasketches.so, 2): no suitable image found.  Did find:
	/Users/davide-anastasia/.local/share/virtualenvs/datasketches-yWoxoifi/lib/python3.9/site-packages/datasketches.so: mach-o, but wrong architecture
	/Users/davide-anastasia/.local/share/virtualenvs/datasketches-yWoxoifi/lib/python3.9/site-packages/datasketches.so: mach-o, but wrong architecture

[jmalkin]

I have unfortunately not yet been able to find an M1 Mac to test this on locally :(

[davideanastasia]

I'd be happy to test it on my machine if you can point me towards some documentation on how to build the project.

[jmalkin]

My apologies for the slow reply.

I have to admit that I'm far from an expert with the cibuildwheel package this relies on. In theory it's set up to build a universal binary that'll work on intel or apple silicon. That's pretty clearly not working as intended for you.

What happens if you clone the repo and, in a pipenv in that directory, run pipenv install .? On my intel-based mac it compiled and installed the dev version, 3.4.0.dev0.

[davideanastasia]

davide-anastasia@Davides-MBP ~ % git clone https://github.com/apache/datasketches-cpp.git
Cloning into 'datasketches-cpp'...
remote: Enumerating objects: 8444, done.
remote: Counting objects: 100% (1303/1303), done.
remote: Compressing objects: 100% (797/797), done.
remote: Total 8444 (delta 706), reused 962 (delta 494), pack-reused 7141
Receiving objects: 100% (8444/8444), 3.16 MiB | 8.21 MiB/s, done.
Resolving deltas: 100% (5452/5452), done.
davide-anastasia@Davides-MBP ~ % mkdir datasketches-venv
davide-anastasia@Davides-MBP ~ % cd datasketches-venv
davide-anastasia@Davides-MBP datasketches-venv % pipenv shell
Creating a virtualenv for this project...
Pipfile: /Users/davide-anastasia/datasketches-venv/Pipfile
Using /opt/homebrew/bin/python3 (3.9.7) to create virtualenv...
⠼ Creating virtual environment...created virtual environment CPython3.9.7.final.0-64 in 260ms
  creator CPython3Posix(dest=/Users/davide-anastasia/.local/share/virtualenvs/datasketches-venv-2e2c-NOz, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/davide-anastasia/Library/Application Support/virtualenv)
    added seed packages: pip==21.3.1, setuptools==60.5.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment!
Virtualenv location: /Users/davide-anastasia/.local/share/virtualenvs/datasketches-venv-2e2c-NOz
Creating a Pipfile for this project...
Launching subshell in virtual environment...
 . /Users/davide-anastasia/.local/share/virtualenvs/datasketches-venv-2e2c-NOz/bin/activate
davide-anastasia@Davides-MBP datasketches-venv %  . /Users/davide-anastasia/.local/share/virtualenvs/datasketches-venv-2e2c-NOz/bin/activate
(datasketches-venv) davide-anastasia@Davides-MBP datasketches-venv % pipenv install ../datasketches-cpp
Installing ../datasketches-cpp...
Adding datasketches to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (40ae47)!
Installing dependencies from Pipfile.lock (40ae47)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
(datasketches-venv) davide-anastasia@Davides-MBP datasketches-venv % python3 -c 'import datasketches'

It seems to be working when installed from the master branch.

[jmalkin]

Well, that's encouraging at least.

Looking around for info briefly last night, this seems to be a common issue for projects that rely on compiled C code. I saw mentions of Apple Silicon devices using python via rosetta as a translation layer to intel for compatibility but didn't have a chance to understand that. In theory we have a universal binary that has both intel and apple compiled libraries bundled together, but that obviously didn't work for you.

Apparently we generated an artifact for the 3.3.0 release but it wasn't pushed anywhere. I'm not sure if you can see the github actions run results, but the most recent successful run of the build_wheels workflow produced this archive: https://github.com/apache/datasketches-cpp/suites/4750101462/artifacts/131517924
If you know the correct architecture, can you try installing the relevant wheel directly?

The other thing I'll try is creating an explicit arm64 mac build in that workflow. But that'll have to wait until household weekend activities are done.

[davideanastasia]

Hi @jmalkin , I managed to download the artifact, but I can only see intel architectures and universal in there. The build I did on my machine is an arm64 instead. I am not sure why the universal didn't work though.

[jmalkin]

Hi @davideanastasia , can you try the right arm64 variant from this artifact? https://github.com/apache/datasketches-cpp/suites/5294530751/artifacts/164184871

It should match what you did locally (although almost certainly not a byte-level match). I suspect github cross-compiles for that architecture, so I want to make sure that works properly for you.

[TheMellyBee]

Hello, I am also running into problems on the M1 architecture although I'm not sure if it's from apache data sketches or spark.

@davideanastasia did you have a chance to on the last note?

Otherwise I may try this out.

[davideanastasia]

@TheMellyBee I had to put aside my project for a few weeks, so I haven't managed to do further tests. Feel free to give it a go.

[jmalkin]

I think there should be an architecture-specific wheel with the latest changes, although we'll need to do a release cycle before we can get a pre-built convenience binary. Aiming to do that after the classic quantiles sketch is merged (and then a little reconciliation/cleanup after that).

[jmalkin]

Well, I never got any takers on testing things. But I believe the 3.4.0 release should address this. That is now live on pypi.,

[jmalkin]

3.4.0 did not actually work, but 3.5.1 does. I was able to confirm that this is now properly resolved.