[ARM64] ERROR: No matching distribution found for connectorx==0.2.2 #186
Comments
|
Hi @Matthieu-Tinycoaching, unfortunately we do not have a way to build wheel for ARM architecture for now. And the only way to use connector-x on it is build the project manually. To manually build connector-x, you need to:
|
|
@wangxiaoying thanks for your feedback. If I build it manually for arm architecture, where do I specify that I want an arm package? |
|
@Matthieu-Tinycoaching no need to specify the arch. The script only builds the package for the host platform. Basically this command The wheel will be under |
|
@dovahcrow thanks for your feedback. Does this mean I have to be on arm64 architecture to build a version of connectorx for this host? |
Correct. That's the main reason why we don't provide an arm64 pip package - because GitHub action does not have arm64 yet. |
|
@dovahcrow @wangxiaoying when trying to install tools and dependencies on ARM64 AWs instance : I got the following error: Do you have any idea on how to deal with? |
|
@dovahcrow @wangxiaoying I manage to deal with above error by installing git and cloning the repo. Now, when running the command : |
|
I managed to pass the previous error by installing a pyenv environment with python3 instead of the python2 system version. Now I have the following error: |
|
Hi @Matthieu-Tinycoaching , |
|
@wangxiaoying thanks it worked. However, when running: I got the following error now: |
|
Solved by replacing But got a new error: |
|
Hi @Matthieu-Tinycoaching , I'm not sure which rust version you are using. It should be the same with our release compilation environment: https://github.com/sfu-db/connector-x/blob/main/.github/workflows/release.yml#L34, which is |
|
@wangxiaoying I installed the However, when running |
|
@wangxiaoying I managed to pass the error for the However, now I still got an error with the polars-core compilation: Could this still be due to a lack of RAM? |
|
@wangxiaoying I managed to build it all with a 4Go RAM graviton AWs instance. One strange thing when I tested it on arm64 instance was that loading a small DB table with ConnectorX was slower than pandas (360 vs. 70 ms, respectively). Could you help me understand why? |
|
Hi @Matthieu-Tinycoaching , great that it successfully compiled.
May I ask which database are you using and what does the query look like? Also it would be great if you could show the log so we can see which step is consuming the time. You can set the RUST_LOG environment variable before importing the library like the following example: import os
os.environ["RUST_LOG"]="connectorx=debug,connectorx_python=debug"
import connectorx as cx
df = cx.read_sql(conn_url, query, return_type="pandas") |
|
I'm running into Rust compile errors when trying to compile the latest release branch. I'm using brew for most of the dependent packages and python. Below are my settings. Any pointers would be appreciated. Thanks for the fantastic software! brew packagesgit branchrust nightlyBuild logLog for just the failed section. It compiles all steps fine up until the last connector-x component. |
|
@lonestriker can you try this nightly version: #186 (comment) ? |
Perfect, that fixed up the build issue on my M1 mac! Edit: but when I check the generated library but it's arm64: Steps to build for those who also hit the same issue:Fresh clone of sourceInstall nightly-2021-10-21Override default project toolchainBuild |
|
Hi @lonestriker , thanks so much for sharing the building procedure!
You are right. We build and publish the release using github-hosted environment. So our script is hard coded for the macOS environment it has. You may need to manually change the file name for now. Sorry for the inconvenience. |
I use a postgreSQL database on AWS and the query looks like:
Please find below the log: |
|
Hi @Matthieu-Tinycoaching , thanks for the log. Looks like your query is very small (only 9 rows). For pandas, ConnectorX will issue a COUNT query ( This overhead of this extra COUNT query can be ignored when the query result is relatively large. But when the result is very small as in your test case, its overhead becomes obvious. |
|
@wangxiaoying thanks for the detailed explanation. Could you estimate from which size of query result would ConnectorX be better than pandas? |
|
Hi @Matthieu-Tinycoaching it will be affected by factors such as network condition so I cannot give you an estimation. Another thing is we also provide the "arrow" destination, and arrow provides this to_pandas api to efficiently convert arrow to pandas. Since arrow does not require continuous memory buffer, we do not issue the COUNT query for it. You can try whether it can reach the speed you want. Here is an example: |
|
@wangxiaoying I tried what you suggested with arrow: But got the following error: |
|
@Matthieu-Tinycoaching Looks like the there is a type that is not supported. What are the types of your query result columns? |
|
@wangxiaoying I have a postgres AWS database and please find below the types of my query result columns: |
|
Hi @Matthieu-Tinycoaching , we are actually currently working on moving from |
|
Hi @wangxiaoying I tried your suggestion but got the following error: I supposed this is because I don't have the right version of ConnectorX. Where could I find versions >= |
|
Hi @Matthieu-Tinycoaching , sorry I forgot you need to compile the code manually. I think for now you can directly download our newest
Yes, I believe if you need to install it on ARM instance then you need to compile it on ARM instance |
|
Thanks @wangxiaoying but the latest release of the main branch is 0.2.3 not the one you mentioned? |
I think the current main branch is under the development of |
|
@wangxiaoying and how to clone this last development version? |
Can you use the methods provided by github? like |
|
@wangxiaoying I checked the version I built ConnectorX from and it's already And still got the issue: Is there anything wrong I have done when building it manually? |
|
Hi @Matthieu-Tinycoaching , looks like the transport rule is missing for TimestampTz from postgres to arrow. I just added it 6db7187. Can you pull the main branch and try again? Thank you! |
|
Hi @wangxiaoying thanks for the update! I finally managed to build a new version of ConnectorX including |
|
I'm trying to compile the new 0.2.4 version of connectorx on my M1 Mac and am getting a different error now. Using the old toolchain:I'm getting this compile error:Switching to latest nightly toolchain:Gives us our old error with lifetimes: |
|
Hi @lonestriker , can you try |
Thank you, worked perfectly. I'll know where to look for the next update as well :) |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
I would like to use connector-x on ARM64 architecture. However, it seems that this library has no ARM architecture version when trying to
pip installit viadocker buildxon this type of architecture.Would it be possible to release an ARM64 version of the library?
Thanks!
The text was updated successfully, but these errors were encountered: