Import apach-arrow from spark overlay #1192
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Tests passed on amd64 Signed-off-by: Yiyang Wu <xgreenlandforwyy@gmail.com>
|
/cc @heroxbd |
littlewu2508
force-pushed
the
apache-arrow
branch
from
5e1a249
to
e47ffb4
Compare
|
@Berrysoft Currently I do not include https://github.com/6-6-6/spark-overlay/blob/e575584fc60afe93316316f5e3b8d4cd7ecab0e0/dev-libs/apache-arrow/files/apache-arrow-9.0.0-thrift-limit.patch you introduced. Can you give some explanations on this patch, and should it be included for generic usage? |
Turns on all use flags, tests all passed: 100% tests passed, 0 tests failed out of 59 Label Time Summary: arrow-tests = 18.62 sec*proc (31 tests) arrow_compute = 15.69 sec*proc (12 tests) arrow_dataset = 5.38 sec*proc (9 tests) filesystem = 1.02 sec*proc (1 test) parquet-tests = 3.46 sec*proc (7 tests) unittest = 43.15 sec*proc (59 tests) Signed-off-by: Yiyang Wu <xgreenlandforwyy@gmail.com>
Signed-off-by: Yiyang Wu <xgreenlandforwyy@gmail.com>
littlewu2508
force-pushed
the
apache-arrow
branch
from
e47ffb4
to
a3bfedc
Compare
|
Thanks 👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These commits pulls
dev-libs/apache-arrow,dev-python/pyarrowand their depedency from spark overlayThey are initially introduced into spark overlay because they are runtime dependencies of pyspark, responsible for parquet data format.
Nowadays we found the parquet data format is suitable for generic scientific computing, and pyspark is not the only use case, so I picked up these ebuilds, bumped its version, did some QA enhancements, and created this PR at ::science.
There are many optional features apache-arrow provided, and I do not have time to turn all the options into use flags. So I just add some useful ones for me (mainly for parquet IO). Also, there are many programming languages supported, these two ebuilds provides support for c++ and python. So, further contributions are always welcome.