[improvement](be) Write Variant V2 directly to Parquet Variant - #67424
Draft
eldenmoon wants to merge 1 commit into
Draft
[improvement](be) Write Variant V2 directly to Parquet Variant#67424eldenmoon wants to merge 1 commit into
eldenmoon wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
eldenmoon
force-pushed
the
branch-variant-v2-direct-parquet
branch
from
September 2, 2026 04:28
92ee243 to
df621fe
Compare
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 16948 ms |
Contributor
TPC-DS: Total hot run time: 82200 ms |
Contributor
ClickBench: Total hot run time: 14.7 s |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Variant V2 Parquet export renders existing metadata/value encoding through JSON and UTF-8 before writing Parquet. When return_object_data_as_binary is enabled, select Arrow's Parquet Variant extension in the Parquet transformer and let DataTypeVariantV2SerDe append the existing metadata/value bytes with exact buffer reservation. Keep the default UTF-8 export for compatibility. The common Arrow converter handles extension arrays through the standard ExtensionType storage contract without Parquet-specific type-name knowledge. On the fixed 10 GB workload, the enabled direct path decreases wall median from 77.7376 to 53.0676 seconds and BE CPU from 245.04 to 213.88 core-seconds, improvements of 31.73% and 12.72% respectively. DuckDB recognizes and decodes the output as VARIANT; Doris legacy Parquet schema inference and Stream Load do not yet consume this logical type end to end.
### Release note
When return_object_data_as_binary is enabled, Variant V2 Parquet export writes the Parquet Variant logical type directly. The default remains the legacy UTF-8 representation.
### Check List (For Author)
- Test: Unit Test, Release build, and manual performance/format validation
- ASAN BE UT before the final generic ExtensionType cleanup: DataTypeSerDeArrowTest.* and VParquetTransformerTest.*, 15/15 passed
- Final cleanup validation: clang-format and check-format passed; the modified source compiled in the ASAN UT build. The focused UT rerun did not reach test execution because the ldb toolchain compatibility library duplicated getrandom; the GLIBC_COMPATIBILITY=OFF retry was blocked by isolated-worktree submodule metadata permissions
- Unit coverage: actual Parquet write, footer inspection, Arrow readback, object, array, string, integers, floating point, booleans, Variant null, and SQL null canonical equality
- Release build before the final generic ExtensionType cleanup: env BUILD_TYPE=RELEASE GLIBC_COMPATIBILITY=OFF ./build.sh --be --hive-udf passed
- Manual test: DuckDB 1.5.2 recognized the full 2,906,455-row export as VARIANT, reported no null values, and decoded sampled objects
- Performance benchmark: one warmup and three trials on the fixed 10 GB workload
- Behavior changed: Yes, opt-in Variant V2 Parquet export uses the Parquet Variant logical type
- Does this need documentation: No, this remains experimental pending Doris reader/load integration
eldenmoon
force-pushed
the
branch-variant-v2-direct-parquet
branch
from
September 2, 2026 08:22
df621fe to
225cff7
Compare
Member
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 16868 ms |
Contributor
TPC-DS: Total hot run time: 82340 ms |
Contributor
ClickBench: Total hot run time: 14.51 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Variant V2 Parquet export renders the existing metadata/value encoding through JSON and UTF-8 before writing Parquet. When
return_object_data_as_binaryis enabled, this change selects Arrow's Parquet Variant extension inVParquetTransformerand letsDataTypeVariantV2SerDeappend the existing metadata/value bytes with exact buffer reservation. The shared Arrow converter handles extension arrays through the standardExtensionTypestorage contract without Parquet-specific type-name knowledge. The default remains the legacy UTF-8 representation, preserving current Doris export-to-Stream-Load behavior. On the fixed 10 GB workload, the enabled direct path decreases wall median from 77.7376 to 53.0676 seconds and BE CPU from 245.04 to 213.88 core-seconds, improvements of 31.73% and 12.72% respectively.DuckDB 1.5.2 recognizes and decodes the direct output as VARIANT. Doris legacy Parquet schema inference and Stream Load do not yet consume the Parquet VARIANT logical type end to end, so direct encoding is explicitly opt-in in this PR.
Release note
When
return_object_data_as_binaryis enabled, Variant V2 Parquet export writes the Parquet Variant logical type directly. The default remains the legacy UTF-8 representation.Check List (For Author)
Test
ExtensionTypecleanup:DataTypeSerDeArrowTest.*andVParquetTransformerTest.*, 15/15 passed.getrandom; theGLIBC_COMPATIBILITY=OFFretry was blocked by isolated-worktree submodule metadata permissions.VParquetTransformer, inspect the Parquet footer, read the file through Arrow Parquet, and compare every non-SQL-null row with the source Variant canonical value.ExtensionTypecleanup:env BUILD_TYPE=RELEASE GLIBC_COMPATIBILITY=OFF ./build.sh --be --hive-udfpassed.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)