Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] Problem importing PyArrow on a cluster #24341

Closed
asfimport opened this issue Mar 17, 2020 · 5 comments
Closed

[Python] Problem importing PyArrow on a cluster #24341

asfimport opened this issue Mar 17, 2020 · 5 comments

Comments

@asfimport
Copy link

Hi, when I am trying to import pyarrow in python, I get the following error:

File "", line 1, in
File "/services/matejm/anaconda3/lib/python3.7/site-packages/pyarrow/init.py", line 49, in
from pyarrow.lib import cpu_count, set_cpu_count
ImportError: libaws-cpp-sdk-s3.so: cannot open shared object file: No such file or directory
What can this be related to? I have searched wherever i could've and could not find any reason for it, so I figured i might as well try in here.

Thank you very much

Note: I have installed py-arrow and its dependencies off-line, since our cluster has a company firewall that does not allow pip nor conda installation

Environment: Linux, RedHat CentOS 7
Reporter: Matej Murin

Note: This issue was originally created as ARROW-8135. Please see the migration documentation for further details.

@asfimport
Copy link
Author

Joris Van den Bossche / @jorisvandenbossche:
Can you give more details on how you install pyarrow and its dependencies?

@asfimport
Copy link
Author

Matej Murin:
I have done:

**conda info pyarrow**

Locally on my machine.
Found my version, and checked if I have all the dependencies listed there in my conda list command.
The ones i didn't, I found the dependencies on conda-forge website, downloaded the tar.bz2 files manually and transferred them, then Installed offline.

@asfimport
Copy link
Author

Wes McKinney / @wesm:
It looks like you're missing transitive dependencies of pyarrow. You also need the runtime dependencies of arrow-cpp, the C++ library

https://github.com/conda-forge/arrow-cpp-feedstock/blob/master/recipe/meta.yaml#L50

Recommend closing the issue

@asfimport
Copy link
Author

Matej Murin:
@wesm yes, thank you for the link ![ it was the aws-sdk-cpp package ]( it was the aws-sdk-cpp package )

@asfimport
Copy link
Author

Matej Murin:
Had been missing dependencies of the package

@asfimport asfimport added this to the 0.16.0 milestone Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant