disagg: Add O11y on object store usage summary of each tiflash store (#10764)

[ti-chi-bot]

This is an automated cherry-pick of #10764

What problem does this PR solve?

Issue Number: ref #10763

Problem Summary:

• We need continuous and low-noise observability for remote object-store usage per TiFlash store.
• Existing S3GC visibility is insufficient to quickly compare expected valid data size vs real object-store footprint.
• We also need this summary collection to be configurable and safe for production overhead.

What is changed and how it works?

disagg: add configurable owner-only S3 storage summary and per-store usage metrics

• Add periodic S3 storage summary in S3GCManagerService:

  • Runs only on S3GC owner.
  • Interval is configurable via summary_interval_seconds.
  • If summary_interval_seconds <= 0, the periodic task is not registered.
  • Warn when interval is too small (< 12h) to avoid high S3 listing overhead.
  • Disabled by default until we ensure the cost is acceptable.
  • If we need to check this metrics, we can trigger the HTTP API to get a metrics and inspect on it. http://${TIFLASH_IP}:${TIFLASH_STATUS_PORT}/tiflash/remote/info https://github.com/pingcap/tiflash/blob/master/docs/tiflash_http_api.md#fetch-the-remote-storage-summary-from-tiflash-write-node

• Add graceful shutdown behavior during summary scan:

  • getStoreStorageSummary checks shutdown_called in listPrefix loop and exits early.

• Add per-store summary metrics:

  • New gauge family:
    • tiflash_storage_s3_store_summary_bytes{store_id, type}
    • type in {data_file_bytes, dt_file_bytes}
  • Metrics are updated after each getStoreStorageSummary completion.

• Wire configuration from settings to S3GC config:

  • New setting:
    • profiles.default.remote_summary_interval_seconds (default 0, disabled)
    • Related docs: tiflash: Add S3 GC related configs docs-cn#21455
  • Set in TMTContext and passed into S3GCManagerService.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

# Run chbenchmark workload and check the metrics of `prelock_keys` and OSS usage
tiup bench ch --host 10.2.12.81 -P 8081 --warehouses 8000 run -D chbenchmark8k -T 50 -t 0 --time 30m --ignore-error --queries q1
# Before the fix, from 23:29 to 00:00, the number of prelock_keys in memory would accumulate and increase with the write load; after the fix, from 02:00 to 02:30, there was no longer any persistent residue of prelock_keys in memory.
# Also can check the new added grafana panel "Remote Store Summary (Disagg arch)"

- [ ] No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Add configurable owner-only periodic S3 storage summary and per-store summary metrics (`tiflash_storage_s3_store_summary_bytes`) for TiFlash disaggregated S3GC observability.

Summary by CodeRabbit

• New Features

  • Added per-store S3 storage summary metrics for data and index file sizes.
  • Added configurable periodic S3 summary collection (default 24h; ≤0 disables).
  • New Grafana panels: Remote Store Summary, Local Lock Manager status, and Local Lock Manager QPS.

• Documentation

  • Updated dashboard layout and panel organization for improved metric visibility.

[ti-chi-bot]

@JaySon-Huang This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (3)

• release-8.5
• release-7.5
• release-8.1

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 46fa9777-84c9-4b91-8601-2291a9fafb89

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[JaySon-Huang]

/unhold

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-03-25 01:09:42.985840621 +0000 UTC m=+316979.021910871: ☑️ agreed by JaySon-Huang.
• 2026-03-25 01:28:54.342561527 +0000 UTC m=+318130.378631788: ☑️ agreed by JinheLin.

[ti-chi-bot]

@kolafish: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, JinheLin, kolafish, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,JinheLin]
• dbms/src/Interpreters/OWNERS [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment