Skip to content

[feature](cloud) Support Cloud Table Stream control plane#65859

Draft
Yukang-Lian wants to merge 9 commits into
apache:masterfrom
Yukang-Lian:codex/cloud-table-stream-control-plane
Draft

[feature](cloud) Support Cloud Table Stream control plane#65859
Yukang-Lian wants to merge 9 commits into
apache:masterfrom
Yukang-Lian:codex/cloud-table-stream-control-plane

Conversation

@Yukang-Lian

@Yukang-Lian Yukang-Lian commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: None

Related PR: #65810

Problem Summary:

Table Stream already provides incremental consumption in storage-compute integrated deployments, where FE owns each partition offset. In Cloud mode, the target-table commit is finalized by MetaService, so keeping offsets in FE cannot atomically commit target data and advance Stream offsets.

This PR adds the Cloud Table Stream control plane:

  • treats FE Catalog as the authority for Stream existence and binding, and stores only Latest and Versioned per-partition offsets in MetaService;
  • reuses the existing Index prepare, commit, Drop Index Log, Recycle Index, and InstanceRecycler lifecycle without creating Stream Mapping keys or physical tablets;
  • resolves one statement-level read snapshot, including clone-chain Offset inheritance;
  • performs expected-offset CAS and target-table commit in the same immediate FDB transaction, with no lazy-commit fallback;
  • keeps Create retries idempotent without overwriting offsets that consumption has already advanced;
  • carries affected Stream identities through DROP PARTITION so the partition recycler removes the corresponding Latest and Versioned offsets;
  • preserves the existing non-Cloud Table Stream path.

This draft depends on #65810 for the Cloud Row Binlog data plane. Before Cloud E2E is enabled, the integration must confirm that every visible partition, including an empty partition and a newly added partition, has an authoritative VersionPB.visible_tso and empty-boundary TSO. The control plane intentionally fails when that metadata is missing instead of treating metadata loss as an empty partition.

Rolling upgrade rule: upgrade all MetaService nodes before enabling the FE Cloud Table Stream path. Once enabled, MetaService rollback requires stopping Stream consumption first because an older MetaService does not understand Stream offset updates.

Release note

Add the storage-compute separated Table Stream control plane. End-to-end availability depends on the Cloud Row Binlog data plane in #65810.

Check List (For Author)

  • Test

    • Regression test
      • table_stream_p0 and row_binlog_p0: 15 suites passed on the branch baseline.
    • Unit Test
      • Cloud MetaService Create, Query, Update, key, clone-chain, recycler, checker, and operation-log tests passed.
      • FE Cloud catalog and Table Stream manager tests: 8 tests passed after the Offset-only metadata change.
      • The earlier full FE Table Stream, transaction, drop, planner, and MV rewrite baseline passed.
    • Manual test
      • ./build.sh --cloud passed.
      • NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe passed with Checkstyle.
      • Cloud C++ formatting passed, and changed-line clang-tidy passed for all modified production C++ files. The existing Cloud HTTP test target still cannot be analyzed because its compile command exposes pre-existing private-API diagnostics.
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. Cloud Table Stream uses MetaService-managed offsets, immediate atomic consumption commits, and FORCE-only drop semantics. Non-Cloud behavior is unchanged.
  • Does this need documentation?

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65810

Problem Summary: Add the typed index, table stream offset, read-state, transaction update protocol messages, and latest/versioned offset key layouts required by the Cloud Table Stream control plane. Reserve CommitTxnRequest field 13 for the related Cloud Row Binlog data-plane change and use field 14 for stream updates.

### Release note

Add metadata protocol and key support for Cloud Table Stream.

### Check List (For Author)

- Test: Unit Test
    - Cloud key encoding tests
    - Cloud protobuf and MetaStore compilation
- Behavior changed: No; protocol and key definitions are not wired into runtime flows yet
- Does this need documentation: Yes; covered by the approved Cloud Table Stream design
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65810

Problem Summary: Cloud Table Stream needs durable metadata, statement-level read snapshots, exactly-once offset advancement, clone-aware reads, lifecycle recycling, and operational diagnostics in MetaService. This change models streams as typed indexes, initializes and reads partition offsets through latest and versioned key spaces, commits target-table data and offset CAS updates in one immediate transaction, and extends recycling and metadata tooling for the new keys.

### Release note

Add the MetaService control plane for Cloud Table Stream.

### Check List (For Author)

- Test: Unit Test
    - Cloud focused unit tests for metadata, reads, transaction updates, recycling, checking, metrics, and operational interfaces
- Behavior changed: Yes; MetaService supports Cloud Table Stream metadata and offset lifecycle
- Does this need documentation: Yes; covered by the Cloud Table Stream design document
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65810

Problem Summary: Cloud Table Stream requires FE to persist Stream creation through MetaService, resolve one statement-level offset and version snapshot after partition pruning, pass expected offsets into target-table commits, and create typed recycle work on FORCE DROP. This change adds the capability gate, Cloud create/query/update/delete integration, fixed scan versions, partition-batched consumption limits, SHOW consumption support, and preserves the existing storage-compute integrated path. Scans eliminated before execution do not advance offsets.

### Release note

Add FE integration for the Cloud Table Stream control plane.

### Check List (For Author)

- Test: Unit Test
    - Focused FE Table Stream, planner, transaction, capability, create, and drop tests
    - FE build with checkstyle
- Behavior changed: Yes; Cloud Table Stream statements use MetaService-managed offsets and FORCE-only drop semantics
- Does this need documentation: Yes; covered by the Cloud Table Stream design document
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65810

Problem Summary: Cloud Table Stream resolves one MetaService read snapshot per statement after partition pruning. The MV temporary rewrite path must keep unresolved Cloud Stream scans intact, avoid pruning them with FE-local offsets, and reuse the installed read state when the full statement is rewritten again. This change preserves the non-Cloud normalization path, prevents duplicate MetaService RPCs, and verifies empty-partition CTE plans and unordered partition requests.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Regression test
    - 73 FE Table Stream, planner, transaction, drop, and MV rewrite tests
    - FE build with checkstyle
    - table_stream_p0 and row_binlog_p0 regression suites (15 suites)
- Behavior changed: No; this preserves the designed statement-level snapshot and existing non-Cloud behavior
- Does this need documentation: Yes; implementation and test design documents were updated
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65810

Problem Summary: Simplify the Cloud Table Stream control plane before review. Store the authoritative current Stream mapping in the 0x01 key space, keep versioned projections only for versioned-write modes, share metadata reads across Create, Query, and Update, and reuse the ordinary Index state machine for Stream creation. Remove the feature-specific capability gate, Operation Log HTTP endpoints, and global FDB range metric changes that are outside this feature's scope. Extend recycler and checker paths for the current Stream mapping lifecycle.

### Release note

None

### Check List (For Author)

- Test: Not run; code was pushed first for review as requested. Clang format, format check, and git diff check passed.
- Behavior changed: Yes. Single-version Cloud Table Streams use current 0x01 mappings, while WRITE_ONLY and READ_WRITE additionally maintain versioned mappings.
- Does this need documentation: Yes. The local implementation design has been updated.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65810

Problem Summary: The generic set_value handler always converted the saved value through a protobuf JSON parser. Empty-value metadata keys do not have a JSON parser, so a successful update could assert while constructing the HTTP response. Return the serialized empty value for those keys, and reset reused decode buffers in the mapping-key unit test because decode_bytes appends to its output.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Regression test
    - Cloud unit tests: keys_test, http_encode_key_test, MetaServiceTableStreamTest, TableStreamRecyclerTest, TableStreamCheckerTest, clone_chain_reader_test, meta_reader_test
    - FE unit tests: 110 focused Table Stream tests
    - Regression test: table_stream_p0 and row_binlog_p0, 15 suites
- Behavior changed: Yes. Generic set_value now returns empty-value metadata without invoking a nonexistent JSON parser.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65859

Problem Summary: Consolidate duplicated Cloud Table Stream RPC retry and read-state validation code, keep shared-nothing transaction commits on their existing listener-based path, and remove unused MetaReader and CloneChainReader overloads. This preserves the Create, Query, Update, and Delete semantics while reducing the production implementation surface.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - run-fe-ut.sh: 33 focused Cloud and shared-nothing Table Stream tests
    - run-cloud-ut.sh: MetaReaderTest.GetTableStreamOffset and CloneChainReaderTest.GetTableStreamOffsets
    - clang-format 16 and clang-tidy 16 on modified Cloud C++ files
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65859

Problem Summary: Merge the latest Apache Doris master and resolve the MetaServiceProxy conflict by retaining the upstream RPC rate limiting changes and the Cloud Table Stream read-state RPC.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.cloud.rpc.MetaServiceProxyTest
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65810

Problem Summary: Simplify Cloud Table Stream persistence by making the FE Catalog authoritative for Stream existence and binding. Remove dedicated current and versioned Stream mapping metadata, use per-partition Offset keys for idempotent Create retries, and carry Stream identities through partition recycling so Offset cleanup does not require a reverse mapping.

### Release note

None

### Check List (For Author)

- Test: Unit Test and Manual test
    - Cloud Table Stream MetaService and Recycler unit tests
    - FE Table Stream catalog unit tests
    - Cloud and FE builds
    - C++ format and changed production-file clang-tidy checks
- Behavior changed: Yes. MetaService no longer persists Table Stream mapping metadata.
- Does this need documentation: Yes. The implementation design is updated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants