[feature](cloud) Support Cloud Table Stream control plane#65859
Draft
Yukang-Lian wants to merge 9 commits into
Draft
[feature](cloud) Support Cloud Table Stream control plane#65859Yukang-Lian wants to merge 9 commits into
Yukang-Lian wants to merge 9 commits into
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Add the typed index, table stream offset, read-state, transaction update protocol messages, and latest/versioned offset key layouts required by the Cloud Table Stream control plane. Reserve CommitTxnRequest field 13 for the related Cloud Row Binlog data-plane change and use field 14 for stream updates. ### Release note Add metadata protocol and key support for Cloud Table Stream. ### Check List (For Author) - Test: Unit Test - Cloud key encoding tests - Cloud protobuf and MetaStore compilation - Behavior changed: No; protocol and key definitions are not wired into runtime flows yet - Does this need documentation: Yes; covered by the approved Cloud Table Stream design
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Cloud Table Stream needs durable metadata, statement-level read snapshots, exactly-once offset advancement, clone-aware reads, lifecycle recycling, and operational diagnostics in MetaService. This change models streams as typed indexes, initializes and reads partition offsets through latest and versioned key spaces, commits target-table data and offset CAS updates in one immediate transaction, and extends recycling and metadata tooling for the new keys. ### Release note Add the MetaService control plane for Cloud Table Stream. ### Check List (For Author) - Test: Unit Test - Cloud focused unit tests for metadata, reads, transaction updates, recycling, checking, metrics, and operational interfaces - Behavior changed: Yes; MetaService supports Cloud Table Stream metadata and offset lifecycle - Does this need documentation: Yes; covered by the Cloud Table Stream design document
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Cloud Table Stream requires FE to persist Stream creation through MetaService, resolve one statement-level offset and version snapshot after partition pruning, pass expected offsets into target-table commits, and create typed recycle work on FORCE DROP. This change adds the capability gate, Cloud create/query/update/delete integration, fixed scan versions, partition-batched consumption limits, SHOW consumption support, and preserves the existing storage-compute integrated path. Scans eliminated before execution do not advance offsets. ### Release note Add FE integration for the Cloud Table Stream control plane. ### Check List (For Author) - Test: Unit Test - Focused FE Table Stream, planner, transaction, capability, create, and drop tests - FE build with checkstyle - Behavior changed: Yes; Cloud Table Stream statements use MetaService-managed offsets and FORCE-only drop semantics - Does this need documentation: Yes; covered by the Cloud Table Stream design document
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Cloud Table Stream resolves one MetaService read snapshot per statement after partition pruning. The MV temporary rewrite path must keep unresolved Cloud Stream scans intact, avoid pruning them with FE-local offsets, and reuse the installed read state when the full statement is rewritten again. This change preserves the non-Cloud normalization path, prevents duplicate MetaService RPCs, and verifies empty-partition CTE plans and unordered partition requests. ### Release note None ### Check List (For Author) - Test: Unit Test and Regression test - 73 FE Table Stream, planner, transaction, drop, and MV rewrite tests - FE build with checkstyle - table_stream_p0 and row_binlog_p0 regression suites (15 suites) - Behavior changed: No; this preserves the designed statement-level snapshot and existing non-Cloud behavior - Does this need documentation: Yes; implementation and test design documents were updated
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Simplify the Cloud Table Stream control plane before review. Store the authoritative current Stream mapping in the 0x01 key space, keep versioned projections only for versioned-write modes, share metadata reads across Create, Query, and Update, and reuse the ordinary Index state machine for Stream creation. Remove the feature-specific capability gate, Operation Log HTTP endpoints, and global FDB range metric changes that are outside this feature's scope. Extend recycler and checker paths for the current Stream mapping lifecycle. ### Release note None ### Check List (For Author) - Test: Not run; code was pushed first for review as requested. Clang format, format check, and git diff check passed. - Behavior changed: Yes. Single-version Cloud Table Streams use current 0x01 mappings, while WRITE_ONLY and READ_WRITE additionally maintain versioned mappings. - Does this need documentation: Yes. The local implementation design has been updated.
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: The generic set_value handler always converted the saved value through a protobuf JSON parser. Empty-value metadata keys do not have a JSON parser, so a successful update could assert while constructing the HTTP response. Return the serialized empty value for those keys, and reset reused decode buffers in the mapping-key unit test because decode_bytes appends to its output. ### Release note None ### Check List (For Author) - Test: Unit Test and Regression test - Cloud unit tests: keys_test, http_encode_key_test, MetaServiceTableStreamTest, TableStreamRecyclerTest, TableStreamCheckerTest, clone_chain_reader_test, meta_reader_test - FE unit tests: 110 focused Table Stream tests - Regression test: table_stream_p0 and row_binlog_p0, 15 suites - Behavior changed: Yes. Generic set_value now returns empty-value metadata without invoking a nonexistent JSON parser. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#65859 Problem Summary: Consolidate duplicated Cloud Table Stream RPC retry and read-state validation code, keep shared-nothing transaction commits on their existing listener-based path, and remove unused MetaReader and CloneChainReader overloads. This preserves the Create, Query, Update, and Delete semantics while reducing the production implementation surface. ### Release note None ### Check List (For Author) - Test: Unit Test - run-fe-ut.sh: 33 focused Cloud and shared-nothing Table Stream tests - run-cloud-ut.sh: MetaReaderTest.GetTableStreamOffset and CloneChainReaderTest.GetTableStreamOffsets - clang-format 16 and clang-tidy 16 on modified Cloud C++ files - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#65859 Problem Summary: Merge the latest Apache Doris master and resolve the MetaServiceProxy conflict by retaining the upstream RPC rate limiting changes and the Cloud Table Stream read-state RPC. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.cloud.rpc.MetaServiceProxyTest - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#65810 Problem Summary: Simplify Cloud Table Stream persistence by making the FE Catalog authoritative for Stream existence and binding. Remove dedicated current and versioned Stream mapping metadata, use per-partition Offset keys for idempotent Create retries, and carry Stream identities through partition recycling so Offset cleanup does not require a reverse mapping. ### Release note None ### Check List (For Author) - Test: Unit Test and Manual test - Cloud Table Stream MetaService and Recycler unit tests - FE Table Stream catalog unit tests - Cloud and FE builds - C++ format and changed production-file clang-tidy checks - Behavior changed: Yes. MetaService no longer persists Table Stream mapping metadata. - Does this need documentation: Yes. The implementation design is updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #65810
Problem Summary:
Table Stream already provides incremental consumption in storage-compute integrated deployments, where FE owns each partition offset. In Cloud mode, the target-table commit is finalized by MetaService, so keeping offsets in FE cannot atomically commit target data and advance Stream offsets.
This PR adds the Cloud Table Stream control plane:
This draft depends on #65810 for the Cloud Row Binlog data plane. Before Cloud E2E is enabled, the integration must confirm that every visible partition, including an empty partition and a newly added partition, has an authoritative
VersionPB.visible_tsoand empty-boundary TSO. The control plane intentionally fails when that metadata is missing instead of treating metadata loss as an empty partition.Rolling upgrade rule: upgrade all MetaService nodes before enabling the FE Cloud Table Stream path. Once enabled, MetaService rollback requires stopping Stream consumption first because an older MetaService does not understand Stream offset updates.
Release note
Add the storage-compute separated Table Stream control plane. End-to-end availability depends on the Cloud Row Binlog data plane in #65810.
Check List (For Author)
Test
table_stream_p0androw_binlog_p0: 15 suites passed on the branch baseline../build.sh --cloudpassed.NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fepassed with Checkstyle.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)