Skip to content

Storages: use-after-free race between SegmentReadTaskScheduler and SegmentReaderPoolManager shutdown (#10917)#10922

Merged
ti-chi-bot[bot] merged 2 commits into
pingcap:release-nextgen-202603from
ti-chi-bot:cherry-pick-10917-to-release-nextgen-202603
Jun 25, 2026
Merged

Storages: use-after-free race between SegmentReadTaskScheduler and SegmentReaderPoolManager shutdown (#10917)#10922
ti-chi-bot[bot] merged 2 commits into
pingcap:release-nextgen-202603from
ti-chi-bot:cherry-pick-10917-to-release-nextgen-202603

Conversation

@ti-chi-bot

Copy link
Copy Markdown
Member

This is an automated cherry-pick of #10917

What problem does this PR solve?

Issue Number: close #10903

Problem Summary:

During shutdown, SegmentReaderPoolManager::stop() destroys all reader_pools and their WorkQueue mutexes, but SegmentReadTaskScheduler's schedLoop background thread is still running. When schedLoop wakes from its 2ms sleep, it calls scheduleOneRound()SegmentReaderPoolManager::addTask() → accesses already-destroyed reader_pools, causing SIGSEGV (use-after-free).

What is changed and how it works?

Add a public `stop()` method to `SegmentReadTaskScheduler` that sets the stop flag and joins the background thread. Call it **before** `SegmentReaderPoolManager::stop()` in both production (`Server.cpp`) and test (`gtests_dbms_main.cpp`) shutdown paths.

- `SegmentReadTaskScheduler.h`: expose `stop()` as public method; rename `std::atomic<bool> stop` → `stop_flag` to avoid name collision
- `SegmentReadTaskScheduler.cpp`: implement `stop()`, destructor delegates to it
- `Server.cpp`: call `SegmentReadTaskScheduler::instance().stop()` before `SegmentReaderPoolManager::instance().stop()`
- `gtests_dbms_main.cpp`: same reorder

Also fix two minor typos along the way:
- `reapPeningPools` → `reapPendingPools` in class comment
- `max_free_threds` → `max_free_threads` in test main

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Manual test: ./gtests_dbms --gtest_filter="DeltaMergeStoreTest.ReadLegacyStringDataCFTiny" — test passes and process exits cleanly without SIGSEGV.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

Release Notes

  • Bug Fixes
    • Improved shutdown stability by ensuring the segment read task scheduler stops before dependent reader pools are torn down, reducing risk of invalid task enqueuing during shutdown.
  • Tests
    • Corrected a thread-pool initialization parameter name in the test harness to ensure consistent setup and teardown behavior.

…ReaderPoolManager shutdown (pingcap#10903)

The schedLoop background thread could still be running when
SegmentReaderPoolManager::stop() destroyed all reader_pools and their
WorkQueue mutexes. Subsequent scheduleOneRound() calls would then
access already-destroyed objects, causing SIGSEGV.

Fix by adding a public stop() method to SegmentReadTaskScheduler
and calling it before SegmentReaderPoolManager::stop() in both
production (Server.cpp) and test (gtests_dbms_main.cpp) shutdown paths.

Also fix:
- rename std::atomic<bool> stop -> stop_flag to avoid name collision
- fix typo: reapPeningPools -> reapPendingPools in header comment
- fix typo: max_free_threds -> max_free_threads in test main
SegmentReader threads may call pushMergedTask() for unfinished work
after stop() has joined schedLoop. Since no scheduler remains to drain
merged_task_pool, guard the method with isStop() to discard tasks
instead of leaking them until singleton teardown.
@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. type/cherry-pick-for-release-nextgen-202603 labels Jun 24, 2026
@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (3)
  • release-8.5
  • release-7.5
  • release-8.1

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: d1e06cac-0862-48e8-b40a-25265f3f4c1e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 24, 2026
@JaySon-Huang

Copy link
Copy Markdown
Contributor

/test pull-unit-next-gen

@ti-chi-bot

ti-chi-bot Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, yongman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [JaySon-Huang,yongman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 25, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-06-24 14:00:48.179171032 +0000 UTC m=+128931.842396545: ☑️ agreed by JaySon-Huang.
  • 2026-06-25 03:44:20.626100078 +0000 UTC m=+178344.289325581: ☑️ agreed by yongman.

@JaySon-Huang

Copy link
Copy Markdown
Contributor

/test pull-unit-next-gen

@ti-chi-bot ti-chi-bot Bot merged commit e25e749 into pingcap:release-nextgen-202603 Jun 25, 2026
6 checks passed
@ti-chi-bot ti-chi-bot Bot deleted the cherry-pick-10917-to-release-nextgen-202603 branch June 25, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. type/cherry-pick-for-release-nextgen-202603

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants