[WIP][SPARK-41367][SQL] Enable V2 file tables in read paths in session catalog #38885
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @cloud-fan |
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
from
38bfc2a
to
b531c04
Compare
peter-toth
changed the title
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
8 times, most recently
from
ac301e3
to
d9154d6
Compare
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
6 times, most recently
from
fc1d46b
to
6f8b0e7
Compare
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
4 times, most recently
from
c65c7ac
to
1bc52a8
Compare
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
4 times, most recently
from
5649882
to
02e973f
Compare
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
from
02e973f
to
e80b737
Compare
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
from
e80b737
to
7744619
Compare
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
2 times, most recently
from
7e1dce5
to
db5cde8
Compare
peter-toth
force-pushed
the
SPARK-enable-dsv2-file-source-read-in-session-catalog
branch
from
db5cde8
to
7bcf665
Compare
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently the config
spark.sql.sources.useV1SourceListdoesn't work with V2 file tables in session catalog, it is always the V1 path that is used. This PR enables V2 file tables (if they are not inspark.sql.sources.useV1SourceList) in read paths via session catalog and fixes a few issues where V2 behaves differently to V1.Why are the changes needed?
It would be good if we could use the already available V2 file source implmenentaions with the session catalog. We ran into a few problems with V2 optimization paths that want to fix in the future. But, currently Spark don't have built-in catalog support for any of the V2 file table implementations. As a first step this PR enables V2 controlled by
spark.sql.sources.useV1SourceListfor the select query plans only. All commands andInsertIntoStatementremain using V1 implementations.The PR also contains some test changes:
SQLQuerySuiteis splitted into V1 and V2 versions.OrcPartitionDiscoverySuiteandParquetPartitionDiscoverySuiteare modified to behave like the V1 versions do. Basically the order of output columns changed in the edge case when partitioning and data columns overlap.Does this PR introduce any user-facing change?
Yes, see order of output columns when partitioning and data columns overlap.
How was this patch tested?
Existing and new UTs.