disagg: Fix TiFlash crash randomly when fails connect to s3 (#10676)

[ti-chi-bot]

This is an automated cherry-pick of #10676

What problem does this PR solve?

Issue Number: close #10674

Problem Summary:

The crash caused by a the ptr to s3_client that passed to task is released when one of the task throw an exception, but other task still keep the reference to the dangling s3_client

Thread A: DataStoreS3::putDMFileLocalFiles
  |
  |-- create s3_client (local)
  |-- spawn Task-1 (upload key K1)  -- fails quickly (throw)
  |-- spawn Task-2 (upload key K2)  -- blocked by SyncPoint
  |
  |-- wait/get Task-1 -> throw -> early return from function
  |-- s3_client destroyed here
  |
  +--> Task-2 resumes later
        |
        |-- S3::uploadFile(*s3_client, ...)
        |-- TiFlashS3Client::setBucketAndKeyWithRoot
        |
        +-- crash: s3_client is dangling (NULL/invalid this)

Similar problem and fix for MPPTask #3889

What is changed and how it works?

* Updates the async upload paths to capture shared resources by value and wait for all tasks to finish before returning, preventing early‑return lifetimes from leaving in‑flight tasks with dangling s3_client/file_provider references.
* Propagates the same safety pattern to `putCheckpointFiles`, `getDataFilesInfo`, `copyToLocal`, `setTaggingsForKeys`

And check whether there are similar problems in the codebase when using IOThreadPool. The answer is no.

In production code, most other IOThreadPool usages don’t have the same “by‑ref + early return” UAF risk; there are a couple of weaker patterns, but they’re either already guarded or don’t capture locals by reference.

Safe patterns already in place

* StorageDisaggregatedRemote.cpp uses scheduleWithFuture plus IOPoolHelper::FutureContainer::getAllResults() for BuildReadTask* pools, so all tasks are waited even when one throws.
* FlashService.cpp (line 995) uses WNEstablishDisaggTaskPool and waits on the future immediately, keeping request/response references valid.

Weaker patterns (not the same UAF, but could be hardened)

* SegmentReadTask.cpp (line 630) (RNWritePageCachePool): uses a vector of futures and sequential f.get(). If a write throws, remaining tasks may still run while the function unwinds. Those tasks only capture WritePageTask by value, so no dangling local refs, but you could switch to FutureContainer for symmetry.
* FileCache.cpp (line 1106) (S3FileCachePool): background downloads capture this. FileCache::shutdown waits for the pool before destruction (FileCache.h (line 287)), so it’s safe if shutdown is respected; otherwise it’s a lifetime dependency.

Tests/benchmarks only

* gtest_filecache.cpp (line 138) and MainEntry.cpp (line 570) use the old “vector futures + f.get” pattern but are non‑prod.
If you want, I can harden SegmentReadTask and the test/benchmark helpers to use FutureContainer for consistency.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Fixed an issue that cause TiFlash panic after timeouts occurred when connecting to the S3

Summary by CodeRabbit

Summary by CodeRabbit

• Bug Fixes

  • Fixed a documentation typo in task execution reporting.

• Tests

  • Added tests covering S3 upload synchronization and concurrent upload handling.

• Code Quality

  • Improved asynchronous task management for S3 uploads to strengthen robustness and resource lifecycle safety.

• New Features

  • Introduced a runtime synchronization control to allow forcing and testing synchronized behavior during S3 uploads.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (3)

• release-8.5
• release-7.5
• release-8.1

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ti-chi-bot]

@yinshuangfei: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

@kolafish: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, JinheLin, kolafish, yinshuangfei

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,JinheLin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-01-26 05:31:25.738546985 +0000 UTC m=+990313.352503841: ☑️ agreed by JaySon-Huang.
• 2026-01-26 07:51:11.588867183 +0000 UTC m=+998699.202824039: ☑️ agreed by JinheLin.