Linux ARM support

[thomasaarholt]

This is same as #237, but for linux.

Polars added linux arm support a few days ago (issue) (PR), which will also be wanted here, since Docker images running on M1 macs will be running arm architecture as well.

connectorx successfully installs in my arm docker image when I run pip install connectorx, but then fails when I try to import it:

import connectorx
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/fabrica/projects/forecast-compare/Untitled.ipynb Cell [1](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f666f7265636173742d636f6d70617265227d/fabrica/projects/forecast-compare/Untitled.ipynb#ch0000052vscode-remote?line=0)8' in <module>
----> 1[ import connectorx

File /usr/local/lib/python3.9/site-packages/connectorx/__init__.py:3, in <module>
      ]()[1](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=0)[ from typing import Optional, Tuple, Union, List, Dict, Any
----> ]()[3](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=2)[ from .connectorx_python import read_sql as _read_sql
      ]()[5](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=4)[ try:
      ]()[6](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=5)[     from importlib.metadata import version

ModuleNotFoundError: No module named 'connectorx.connectorx_python']()

If I require it to use the binary from pypi, then it does not find a match:

pip install -U connectorx --only-binary connectorx 
ERROR: Could not find a version that satisfies the requirement connectorx (from versions: none)
ERROR: No matching distribution found for connectorx

[wangxiaoying]

Hi @thomasaarholt , thanks for the issue. The source file on pypi is not ready yet so we have removed it from our new alpha version. We are working on it.

[messense]

Feel free to ping me if you need any help.

[wangxiaoying]

Hi @messense , can you please help to check this error (build wheel for arm linux): https://github.com/sfu-db/connector-x/runs/5337702956?check_suite_focus=true ? I have no idea how to fix it. The yaml file can be found here. Thank you!

[messense]

@wangxiaoying Try add manylinux: 2014 option to maturin-action.

[messense]

This is harder than we thought since you have system dependencies which aren't present in maturin-action's cross compilation docker images(they are based on Ubuntu 20.04).

connector-x/.github/workflows/release.yml

Line 46 in fd7a61d

yum install -y openssl11-devel mysql-devel postgresql-devel freetds-devel

And

connector-x/.github/workflows/release.yml

Line 12 in fd7a61d

container: quay.io/pypa/manylinux2014_x86_64

forces x86_64 so you may need to change to use QEMU and switch to the official aarch64 manylinux docker image instead, but it's gonna be at least 6x slow on CI.

It will be much easier if we're statically link system dependencies.

[messense]

For now the easiest way forward is to use a CI with native ARM support for example https://www.drone.io/

https://docs.drone.io/pipeline/docker/syntax/platform/

[wangxiaoying]

For now the easiest way forward is to use a CI with native ARM support for example https://www.drone.io/

https://docs.drone.io/pipeline/docker/syntax/platform/

Thank you so much! I will take a look at this.

[thomasaarholt]

Has anyone taken a look at this since February? I just met this again now - installing connectorx succeeds on aarch64, but it is not importable, resulting in the same error as in the first post.

[wangxiaoying]

Hi @thomasaarholt , we are now successfully supporting apple m1 with wheel files. But still having issue to cross compile for linux arm. Currently the only way to install connector-x on linux arm is to build it from source: https://sfu-db.github.io/connector-x/install.html#build-from-source-code

[viniciusd]

Sorry for bumping this, but I find it more adequate than creating yet another issue (in addition to #186, #386, and #240).

I understand there was an attempt to build ARM releases, but it was removed at 739b8b5, but I can't really tell why.

Do you need help figuring out the ARM gh action @wangxiaoying? Is there something we can do?

[wangxiaoying]

Sorry for bumping this, but I find it more adequate than creating yet another issue (in addition to #186, #386, and #240).

I understand there was an attempt to build ARM releases, but it was removed at 739b8b5, but I can't really tell why.

Do you need help figuring out the ARM gh action @wangxiaoying? Is there something we can do?

Yes, the previous PR for supporting aarch platform was failed (more details here). We are still looking for help on this one. Unfortunately, the logs for the previous actions are already expired. But I think you can fork the repo and test the gihub action (.github/workflows/release.yml) locally. Please feel free to submit a PR.