FIX-#4636: allows read_parquet to detect column partitioning in non-local filesystems
#5192
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
billiam-wang
force-pushed
the
bill-4636
branch
from
b9e1902
to
1ae5c98
Compare
anmyachev
reviewed
Nov 5, 2022
| path_generator = os.walk(path) | ||
| else: | ||
| storage_options = kwargs.get("storage_options") | ||
| fs, fs_path = url_to_fs(path, **storage_options) |
There was a problem hiding this comment.
storage_options variable can be None, so this construction breaks several tests in CI.
mvashishtha
reviewed
Nov 9, 2022
…ning in non-local filesystems Signed-off-by: Bill Wang <billiam@ponder.io>
Signed-off-by: Bill Wang <billiam@ponder.io>
…ndas version Signed-off-by: Bill Wang <billiam@ponder.io>
billiam-wang
force-pushed
the
bill-4636
branch
from
2c1b8c6
to
152a7a3
Compare
Signed-off-by: Bill Wang <billiam@ponder.io>
pyrito
approved these changes
Nov 16, 2022
mvashishtha
approved these changes
Nov 16, 2022
dchigarev
pushed a commit
that referenced
this pull request
Nov 25, 2022
…-local filesystems (#5192) Signed-off-by: Bill Wang <billiam@ponder.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Bill Wang billiam@ponder.io
What do these changes do?
Added support for
read_parqueton column partitioned parquet files in non-local filesystems. Includes tests on non-local S3 parquet files.flake8 modin/ asv_bench/benchmarks scripts/doc_checker.pyblack --check modin/ asv_bench/benchmarks scripts/doc_checker.pygit commit -sdocs/development/architecture.rstis up-to-date