feat: Phase 3c — ParquetMergeExecutor + full downloader#6358
Open
g-talbot wants to merge 1 commit intogtt/parquet-merge-pipeline-3bfrom
Open
feat: Phase 3c — ParquetMergeExecutor + full downloader#6358g-talbot wants to merge 1 commit intogtt/parquet-merge-pipeline-3bfrom
g-talbot wants to merge 1 commit intogtt/parquet-merge-pipeline-3bfrom
Conversation
4 tasks
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
3b171a0 to
84c6dd3
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
ceba410 to
5937440
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
84c6dd3 to
a23011c
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
5937440 to
0b1c9cc
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
a23011c to
86fd55a
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
2 times, most recently
from
66b97e0 to
0f051bc
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
86fd55a to
ef391c9
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
0f051bc to
16b46d7
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
ef391c9 to
90c5589
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
16b46d7 to
8e19b6b
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
90c5589 to
49c6c19
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
8e19b6b to
f32bd64
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
49c6c19 to
93a0a20
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
f32bd64 to
de17c0e
Compare
…ase 3c) Phase 3 pipeline integration, third PR: - ParquetMergeSplitDownloader: downloads each input split's Parquet file from object storage to a local temp directory, forwards ParquetMergeScratch to the executor. Replaces the stub from PR 3b. - ParquetMergeExecutor: runs merge_sorted_parquet_files via run_cpu_intensive, builds output ParquetSplitMetadata via merge_parquet_split_metadata, renames output files to match generated split IDs, sends ParquetSplitBatch with replaced_split_ids to the uploader. - ParquetSplitBatch.checkpoint_delta -> checkpoint_delta_opt: now Option to support merge operations (no checkpoint delta for data reorganization). Ingest path passes Some(delta), merge path passes None. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
93a0a20 to
3a91a31
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
de17c0e to
1f6512e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #6357 (Phase 3b).
storage.copy_to_file(), forwardsParquetMergeScratchto the executor.merge_sorted_parquet_filesviarun_cpu_intensive, builds outputParquetSplitMetadataviamerge_parquet_split_metadata, renames output files to match generated split IDs, sendsParquetSplitBatchwithreplaced_split_idsto the uploader.checkpoint_delta_opt:ParquetSplitBatch.checkpoint_deltachanged toOption<IndexCheckpointDelta>to support merge operations (no checkpoint delta for data reorganization). Ingest path passesSome(delta), merge path passesNone.Test plan
metricsfeaturecargo clippyclean🤖 Generated with Claude Code