[Feature](cloud) Table-level event-driven warmup with progress observation

[bobhan1]

What problem does this PR solve?

Problem Summary:

The current event-driven warmup only supports cluster-level granularity — once enabled, all table writes on the source cluster are warmed up to the target cluster. For scenarios where only specific tables need warming (e.g., core business tables, high-frequency query tables), cluster-level warmup leads to unnecessary bandwidth consumption and cache space waste. Additionally, existing bvar metrics are global accumulators that cannot distinguish between different Jobs or tables, and BE crashes cause permanent gap residuals.

This PR implements two core features:

1. Table-Level Event-Driven Warmup (ON TABLES)

Specify the warmup scope via ON TABLES (INCLUDE/EXCLUDE) SQL syntax with * and ? wildcard support. Key features:

• INCLUDE / EXCLUDE rule composition: INCLUDE declares the warmup scope, EXCLUDE removes specific tables from the INCLUDE result set
• Dynamic table change tracking: Automatically updates matched table list after CREATE TABLE / DROP TABLE / RENAME TABLE
• Async materialized view support: ON TABLES patterns match both regular tables and async materialized views (MTMV)
• Multi-destination clusters: The same table can be warmed up to multiple target clusters simultaneously with independent filters
• Rule canonicalization and deduplication: Rules are sorted before comparison to prevent duplicate jobs with semantically identical but differently ordered rules
• SQL syntax extensions: New ON TABLES clause; SHOW WARM UP JOB adds TableFilter/MatchedTables columns

2. Warmup Progress Observation

Per-job windowed metrics based on bvar::MultiDimension + bvar::Window, solving the problems of global accumulators being unable to distinguish Jobs/tables and permanent gap residuals after BE crashes:

• BE side: Per-job windowed metrics (5min/30min/1h) automatically maintaining requested/finish/fail statistics
• BE HTTP API: /api/warmup_event_driven_stats exposes per-job JSON statistics
• FE on-demand collection: SHOW WARM UP JOB concurrently collects from all BEs and displays aggregated SyncStats JSON column
• Gap calculation: gap = requested - finished; windowed metrics cause crash-induced gaps to expire automatically

SQL Syntax Examples

-- Create a table-level event-driven warmup job
WARM UP COMPUTE GROUP query_cg WITH COMPUTE GROUP write_cg
ON TABLES (
    INCLUDE 'core_db.config',
    INCLUDE 'core_db.metadata',
    INCLUDE 'report_db.monthly_*',
    INCLUDE '*.sales_*',
    INCLUDE 'log_db.log_?',
    EXCLUDE '*.*_archive'
)
PROPERTIES (
    "sync_mode" = "event_driven",
    "sync_event" = "load"
);
-- View warmup job status and sync progress
SHOW WARM UP JOB;
+-------+-----------------+-----------------+---------+---------+---------------------+---------------------------+-----------+--------+-----------------------------------------------+-----------------------------------------+-----------+
| JobId | SrcComputeGroup | DstComputeGroup | Status  | Type    | SyncMode            | CreateTime                | ...       | Tables | TableFilter                                   | MatchedTables                           | SyncStats |
+-------+-----------------+-----------------+---------+---------+---------------------+---------------------------+-----------+--------+-----------------------------------------------+-----------------------------------------+-----------+
| 13419 | ingestion_cg    | analytics_cg    | RUNNING | TABLES  | EVENT_DRIVEN (LOAD) | 2024-01-01 10:05:00.000   | ...       |        | {"include":["ods.*"],"exclude":["ods.tmp_*"]} | ods.orders, ods.payments, ods.users     | {...}     |
+-------+-----------------+-----------------+---------+---------+---------------------+---------------------------+-----------+--------+-----------------------------------------------+-----------------------------------------+-----------+
-- SyncStats column shows per-job aggregated windowed sync progress in JSON

{
  "seg_num": {
    "requested_5m": 42, "finish_5m": 40, "gap_5m": 2, "fail_5m": 0,
    "requested_30m": 180, "finish_30m": 178, "gap_30m": 2, "fail_30m": 0,
    "requested_1h": 320, "finish_1h": 318, "gap_1h": 2, "fail_1h": 0
  },
  "seg_size": {
    "requested_5m": "12.5mb", "finish_5m": "11.8mb", "gap_5m": "716kb", "fail_5m": "0b",
    "requested_30m": "58.2mb", "finish_30m": "57.5mb", "gap_30m": "716kb", "fail_30m": "0b",
    "requested_1h": "102.3mb", "finish_1h": "101.6mb", "gap_1h": "716kb", "fail_1h": "0b"
  },
  "idx_num": { "requested_5m": 10, "finish_5m": 10, "gap_5m": 0, "fail_5m": 0, "..." : "..." },
  "idx_size": { "requested_5m": "2.1mb", "finish_5m": "2.1mb", "gap_5m": "0b", "fail_5m": "0b", "..." : "..." },
  "last_trigger_ts": "14:32:15",
  "last_finish_ts": "14:32:18"
}

Code Change Summary

FE Core Changes:

• CacheHotspotManager.java: Table ID resolution, dynamic refresh, Auth-Token authentication, on-demand metrics collection
• CloudWarmUpJob.java: JobType.TABLES enum, table filter rule persistence, SyncStats display
• OnTablesFilter.java (new): Glob wildcard matching engine (INCLUDE/EXCLUDE)
• TableWarmUpWindowedStats.java / JobWarmUpStats.java (new): Windowed statistics data models
• WarmUpClusterCommand.java / LogicalPlanBuilder.java: ON TABLES SQL parsing

BE Core Changes:

• bvar_windowed_adder.h (new): Multi-dimension windowed adder wrapper
• cloud_warm_up_manager.cpp/h: JobReplicaInfo struct, per-job requested metrics
• cloud_internal_service.cpp: Per-job finish/fail metrics
• warmup_stats_action.cpp/h (new): HTTP API /api/warmup_event_driven_stats
• internal_service.proto: Added job_id field to PWarmUpRowsetRequest

Regression Tests (Docker multi-cluster): 8 independent test files

• INCLUDE wildcard, INCLUDE+EXCLUDE combination, multi-INCLUDE cross-database, rule canonicalization
• Dynamic table change tracking, multi-destination clusters, error handling & lifecycle, SyncStats progress observation

FE Unit Tests: 7 test classes, ~70 test methods

• OnTablesFilterTest, CloudWarmUpJobTableFilterTest, CacheHotspotManagerTableFilterTest
• WarmUpClusterOnTablesParseTest, WarmUpStatsTest, etc.

Design Documents

• table-level-event-driven-warmup-design.md: Table-level warmup feature design
• table-level-event-driven-warmup-user-guide.md: User guide
• warmup-progress-observation-design.md: Progress observation design
• warmup-progress-observation-user-guide.md: Progress observation user guide
• table-level-event-driven-warmup-test-doc.md: Complete test documentation

Release note

Support table-level Event-Driven Warmup: specify warmup table scope via ON TABLES (INCLUDE/EXCLUDE) syntax (supports */? wildcards), with automatic table change tracking. Added per-job windowed sync progress observation (SHOW WARM UP JOB SyncStats column + BE HTTP API /api/warmup_event_driven_stats).

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes. Added ON TABLES SQL clause; SHOW WARM UP JOB adds TableFilter/MatchedTables/SyncStats columns; new BE HTTP API /api/warmup_event_driven_stats; CloudWarmUpJob.JobType adds TABLES enum value

• Does this need documentation?

  • No.
  • Yes. Design docs and user guides are included in this PR

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[bobhan1]

/review

[github-actions]

OpenCode automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24560471071

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

[bobhan1]

run buildall

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.02% (1842/2361)
Line Coverage	64.65% (32946/50958)
Region Coverage	65.26% (16338/25034)
Branch Coverage	55.81% (8716/15616)

[hello-stephen]

FE UT Coverage Report

Increment line coverage 77.34% (454/587) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 3.55% (21/592) 🎉
Increment coverage report
Complete coverage report

[bobhan1]

run buildall

[bobhan1]

/review

[github-actions]

I found blocking correctness issues in the new ON TABLES / progress-observation path, so I can't approve this PR yet.

Findings:

• FE image/checkpoint load does not rebuild transient ON TABLES state, so restored jobs can stop warming tables after FE restart.
• WITH TABLE ... ON TABLES (...) is accepted but ON TABLES is silently ignored; the statement becomes a one-shot explicit-table warmup.
• resolveTableIds() matches non-writable objects such as VIEWs, so a job can be accepted with only view matches and then never trigger.
• Rolling-upgrade compatibility is not handled for new thrift field table_ids; old BEs will ignore it and treat filtered jobs as cluster-level warmup.
• /api/warmup_event_driven_stats drops index-only jobs because it discovers job ids only from segment metrics.
• The per-destination fan-out path still aborts on the first transport failure, so one bad destination can block later destinations in a multi-destination setup.
• cloud_warm_up_table_filter_refresh_interval_ms is marked mutable but the daemon only snapshots it at startup.
• One new regression test still expects Type = CLUSTER for an ON TABLES job even though the implementation now returns TABLES.

Critical checkpoints:

• Goal of current task: Partially accomplished. Table-level event-driven warmup and per-job progress observation are implemented on the happy path, but the issues above break restart safety, compatibility, object selection, and independent fan-out.
• Modification size/focus: Reasonably focused on warmup, but it spans FE parser/persistence/runtime, BE protocol/metrics, HTTP, tests, and docs, so the regression surface is large.
• Concurrency: Applicable. FE adds a refresh daemon and concurrent BE HTTP collection; BE updates metrics from async download callbacks. I did not find a new lock-order/deadlock issue in the touched paths, but correctness still fails in concurrent execution because one RPC failure can abort later destinations.
• Lifecycle / static initialization: Applicable. Journal replay rebuilds transient table-filter state, but image/checkpoint load does not. I did not find a blocking cross-TU static-init issue in the new globals.
• Configuration items: Applicable. The new refresh-interval config is declared mutable, but runtime changes are not observed without FE restart.
• Incompatible changes / rolling upgrade: Applicable. New FE->BE table_ids and BE->BE job_id fields are introduced. job_id is backward-safe as an optional protobuf field, but table_ids is not rolling-upgrade safe because old BEs will ignore it and warm all tables.
• Parallel code paths: Applicable. Journal replay and image load are not equivalent for filtered jobs, and the explicit-table path conflicts with the new ON TABLES path when both syntaxes are combined.
• Special conditions / validation: Applicable. ON TABLES validation misses the WITH TABLE combination, and table resolution misses the restriction to writable object types.
• Test coverage: Broad but incomplete. The PR adds substantial FE/BE/regression coverage, but it misses image/checkpoint restore, mixed-version BE rollout, and non-writable-object matching; one new regression assertion is also stale.
• Test result changes: I reviewed the new tests only; I did not run the suite in this runner.
• Observability: Applicable. The new per-job stats are useful, but index-only warmup can disappear from the HTTP/SHOW path.
• Persistence / failover: Applicable. FE metadata persistence is touched; EditLog replay is handled, but image/checkpoint load is not equivalent.
• Data writes / transactionality: Not a primary area of change. I did not find a new visible-version / MoW correctness regression in the touched storage paths.
• FE/BE variable propagation: Applicable. table_ids propagation is not safe across mixed BE versions; job_id propagation itself looks consistent on the new-code path.
• Performance: No new blocking hot-path performance issue stood out beyond the correctness issues above.

Requesting changes.

[github-actions]

read() only reconstructs tableFilterExpr. During image/checkpoint load CloudEnv.loadCloudWarmUpJob() calls addCloudWarmUpJob() rather than replayCloudWarmUpJob(), so the restored job never gets rebuildOnTablesFilter() / currentTableIdNames. After an FE restart an ON TABLES job comes back with a null filter and starts sending an empty table_ids list (or the refresh daemon keeps failing if Gson left the transient map null), so the job silently stops warming any tables until it is recreated. We need to rebuild the transient filter state on image load too, not just on journal replay.

[github-actions]

getTableNamesOrEmptyWithLock() returns every catalog entry name here, and we never filter the resolved object type. That means ordinary VIEWs can satisfy the initial ON TABLES match check and show up in MatchedTables even though load-driven warmup is only triggered from BE rowset writes. A job like ON TABLES (INCLUDE 'db.v_*') can be accepted with only views matched and then never warm anything. We should restrict this to writable types (OlapTable / MTMV) before adding the ID.

[github-actions]

This starts sending filtered jobs to every BE unconditionally, but table_ids is only a new thrift field. Pre-upgrade BEs will drop the unknown field and register the job as cluster-level warmup, so an ON TABLES job silently warms all tables during an FE-first / mixed-BE rollout. We need a version gate / feature check before allowing filtered jobs, or some compatibility fallback that refuses to install the job on old BEs.

[bobhan1]

run buildall

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.06% (1843/2361)
Line Coverage	64.78% (33001/50947)
Region Coverage	65.29% (16379/25085)
Branch Coverage	55.86% (8743/15652)