-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make pyarrow optional #186
Comments
Could you tell me any actual use case behind your request where you can't install? PFIO is originally a file system wrapper that does not support POSIX. And it supports HDFS as a primary file system to wrap. We assumed that PFIO users always needed pyarrow. I also want which feature of PFIO you're using, without need for HDFS. Also, pyarrow itself has an x86 wheel release - it can be installed without Hadoop or even Java installation. So does boto3. In case when you don't need them, they don't do any harm unless you use them. Also, #187 would be a huge change that changes the usage of PFIO. I don't have good idea to implement pyarrow and boto3 as |
Thank you for a comment. This is the result of
The error is because pyarrow 3.0.0 does not provide a wheel for pfio's README says it's a file system abstraction. We use it in our library for that purpose. The library intends to support both HDFS and POSIX file system. Users (like me in this case) should be able to run it both with and without HDFS. |
Got it. Thank you for detail. How about updating pyarrow to 4.0? (Now it has 4.0.1). That should work for you if you're using Linux, because pyarrow itself does no harm (once it's installed :P ), even without HDFS. That fix would also be easy for me, too. For I understand your concern that a dependency that's not needed is installed by opt-out, but the complexity introduced by naive resolution, like install options such as |
I've released 1.5, which depends on pyarrow 4.0.1. I hope it works for you, but otherwise would welcome further discussion. |
Thank you for the suggestion, but closing. Feel free to reopen, to resume the discussion. |
I'm trying to install pfio in an environment where I can't install arrow.
I have no plan to use HDFS either.
I think it's very helpful if pfio can be installed without HDFS support.
The text was updated successfully, but these errors were encountered: