feat: Add V2 scan support for native_iceberg_compat
#3272
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR adds support for native_iceberg_compat scan implementation with V2 data sources (BatchScanExec with ParquetScan). ## Changes ### New Files - `CometNativeParquetPartitionReaderFactory`: A V2 partition reader factory that uses NativeBatchReader (DataFusion-based Parquet reader) - `CometNativeParquetScan`: A V2 scan trait that creates the new reader factory ### Behavior - V2 scans with `auto` or `native_iceberg_compat`: Use CometNativeParquetScan (DataFusion-based reader) - V2 scans with `native_comet`: Use existing CometParquetScan (legacy JNI-based BatchReader) - V2 scans with `native_datafusion`: Fall back to Spark (not yet supported for V2) This is consistent with the goal of deprecating the legacy BatchReader code - V2 scans can now use the DataFusion-based reader when using the recommended scan implementations. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3272 +/- ##
============================================
+ Coverage 56.12% 60.32% +4.19%
- Complexity 976 1490 +514
============================================
Files 119 177 +58
Lines 11743 16246 +4503
Branches 2251 2682 +431
============================================
+ Hits 6591 9800 +3209
- Misses 4012 5082 +1070
- Partials 1140 1364 +224 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
andygrove
changed the title
native_iceberg_compat support for V2 Parquet scans [WIP]
…scan impls The test was failing when the default scan impl was set to native_datafusion because V2 scans fall back to Spark in that case. This fix ensures the test explicitly sets SCAN_AUTO to test the V2 scan replacement behavior. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
andygrove
changed the title
native_iceberg_compat support for V2 Parquet scans [WIP]native_iceberg_compat support for V2 Parquet scans
3 tasks
andygrove
changed the title
native_iceberg_compat support for V2 Parquet scansnative_iceberg_compat
Add ParquetReadV2NativeIcebergCompatSuite that runs all ParquetReadSuite tests (278+ tests) with V2 scans and explicit native_iceberg_compat setting. Note: ParquetReadV2Suite already provides implicit coverage via SCAN_AUTO, but this suite explicitly sets SCAN_NATIVE_ICEBERG_COMPAT for clarity and consistency with ParquetReadV2NativeDataFusionSuite. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
This PR adds support for
native_iceberg_compatscan implementation with V2 data sources (BatchScanExec with ParquetScan).Rationale for this change
Previously, V2 Parquet scans always used the legacy
BatchReader(thenative_cometapproach) regardless of theCOMET_NATIVE_SCAN_IMPLsetting. This was inconsistent with V1 scans which respect the scan impl configuration.This change enables V2 scans to use the DataFusion-based
NativeBatchReaderwhennative_iceberg_compatorautois specified, which is important for deprecating the legacy mutable buffer code.What changes are included in this PR?
New Files
CometNativeParquetPartitionReaderFactory: A V2 partition reader factory that usesNativeBatchReader(DataFusion-based Parquet reader)CometNativeParquetScan: A V2 scan trait that creates the new reader factoryBehavior
autoornative_iceberg_compat: UseCometNativeParquetScan(DataFusion-based reader)native_comet: Use existingCometParquetScan(legacy JNI-based BatchReader)native_datafusion: Fall back to Spark (not yet supported for V2)How are these changes tested?
CometScanRuleSuitewith tests for V2 scan behaviorParquetReadV2Suiteto verify V2 scans work with the new implementation