planner: flaky test TestBatchDropBindings (#66559)

[ti-chi-bot]

This is an automated cherry-pick of #66559

What problem does this PR solve?

Issue Number: close #66371

Problem Summary:

What changed and how does it work?

Decision was made here to fix the test rather than the underlying behavior. The test does expose a race condition - but the likelihood that a customer sees this as a critical issue, or that it has any significant impact to their operations - is arguably low.

The issue is - if the binding cache reload (which occurs every 3 seconds) had begun before the drop completed, then the dropped binding could be reloaded - and exist for 3 seconds longer than the user intended. The solution for that would be to add a mutex to the bindingCacheUpdater - but that would then be executed every 3 seconds. However, if a customer exposes this problem - then it may be necessary to add this fix.

Analysis of the issue:

Root Cause: Race Between Background Binding Loader and DROP

The test does not set bindinfo.Lease = 0, so the background goroutine globalBindHandleWorkerLoop (started in domain.go:1571) runs every 3 seconds, calling LoadFromStorageToCache(false, false). This races with the DropBinding operation's own deferred LoadFromStorageToCache call.

The Race in Detail

LoadFromStorageToCache (binding_cache.go:64) is not atomic — it first reads bindings from storage via SQL, then iterates over them updating the cache one-by-one. There is no mutex protecting the entire read-process cycle. Two concurrent calls can interleave.

Here's the problematic sequence:

┌──────┬─────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────┐
  │ Step │                    Background goroutine                     │                         Test goroutine                         │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │ 1    │ LoadFromStorageToCache starts, reads storage snapshot →     │                                                                │
  │      │ sees enabled bindings (update_time=T0)                      │                                                                │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │ 2    │                                                             │ DROP binding commits → marks bindings as deleted               │
  │      │                                                             │ (update_time=T1 > T0) in mysql.bind_info                       │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │      │                                                             │ DROP's deferred LoadFromStorageToCache reads storage → sees    │
  │ 3    │                                                             │ deleted bindings → pickCachedBinding(enabled@T0, deleted@T1) → │
  │      │                                                             │  nil → RemoveBinding() → cache is now empty                    │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │      │ Continues processing stale snapshot: GetBinding() returns   │                                                                │
  │ 4    │ nil (just removed). pickCachedBinding(nil, enabled@T0) →    │                                                                │
  │      │ returns enabled@T0 → SetBinding() re-adds the binding to    │                                                                │
  │      │ cache                                                       │                                                                │
  ├──────┼─────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┤
  │ 5    │                                                             │ SHOW BINDINGS → sees the re-added binding → assertion fails    │
  └──────┴─────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────┘

Why the stale data persists

At step 4, the background goroutine also stores lastUpdateTime = T0 (from its stale snapshot), potentially overwriting the DROP's stored T1. This makes lastUpdateTime go backwards. The 10-second timeLagTolerance (binding_cache.go:94) means the next incremental load will likely catch the T1 deleted record again, but there's a transient window (up to the next 3-second tick) where the cache has stale data.

Key code locations

• Background loop: pkg/domain/domain.go:1571-1626 — ticks every bindinfo.Lease (default 3s)
• Lease default: pkg/bindinfo/binding_handle.go:26 — var Lease = 3 * time.Second
• Lease=0 check: pkg/domain/domain.go:1560 — if bindinfo.Lease == 0 { return } (skips starting the background goroutine)
• DropBinding deferred load: pkg/bindinfo/binding_operator.go:164-167
• Non-atomic LoadFromStorageToCache: pkg/bindinfo/binding_cache.go:64-136 — reads from storage, then processes into cache without a mutex
• Mock owner succeeds: pkg/owner/mock.go:135 — CampaignOwner() returns nil, so the background loop does start in mock test environments

Evidence: other tests avoid this

TestGCBindRecord (bind_test.go:371-377) explicitly sets bindinfo.Lease = 0 before creating the mock store/domain, preventing the background goroutine from starting. TestBatchDropBindings does not.

Session bindings are not affected

removeAllBindings(tk, false) for session bindings uses DropSessionBinding (session_handle.go:90), which is a simple in-memory map delete with no background reload — no race possible there.

Summary

The flakiness is caused by the background globalBindHandleWorkerLoop goroutine's LoadFromStorageToCache interleaving with the DropBinding deferred LoadFromStorageToCache, re-adding stale (enabled) bindings to the cache after the DROP already removed them. The fix would be to set bindinfo.Lease = 0 at the start of the test (as TestGCBindRecord does), or to add mutual exclusion around LoadFromStorageToCache.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Summary by CodeRabbit

• Tests
  • Improved test isolation for batch binding cleanup scenarios.
  • Ensured temporary lease settings are restored after the test completes.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign time-and-fate for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/bindinfo/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0c512d6e-f744-4fee-aeae-bd64a4124f20

📥 Commits

Reviewing files that changed from the base of the PR and between 10431d8 and 494677e.

📒 Files selected for processing (1)

• pkg/bindinfo/tests/bind_test.go

---

📝 Walkthrough

Walkthrough

TestBatchDropBindings now temporarily sets bindinfo.Lease to 0 and restores the original value after the test.

Changes

Binding test stability

Layer / File(s)	Summary
Scoped lease override pkg/bindinfo/tests/bind_test.go	TestBatchDropBindings saves the original lease, sets it to 0, and restores it with defer.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested labels: approved, lgtm

Suggested reviewers: ailinkid, time-and-fate

Poem

I’m a bunny guarding tests tonight,
Lease set to zero, everything right.
I save the old value, then hop away,
Restoring it neatly at end of day.
No flaky carrots in this test array!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title names the flaky TestBatchDropBindings fix and is clearly related to the change.
Description check	✅ Passed	The description includes the issue number, problem summary, change details, checklist, side effects, and release note template.
Linked Issues check	✅ Passed	The change addresses #66371 by disabling the background binding reload that causes the flaky TestBatchDropBindings failure.
Out of Scope Changes check	✅ Passed	The only change is a test-only lease override and restore, which stays within the flaky test fix scope.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (release-8.5@10431d8). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff                @@
##             release-8.5     #70013   +/-   ##
================================================
  Coverage               ?   55.1686%           
================================================
  Files                  ?       1849           
  Lines                  ?     666288           
  Branches               ?          0           
================================================
  Hits                   ?     367582           
  Misses                 ?     271267           
  Partials               ?      27439           

Flag	Coverage Δ
integration	38.2105% <ø> (?)
unit	65.1937% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	55.3108% <0.0000%> (?)
parser	∅ <0.0000%> (?)
br	54.7802% <0.0000%> (?)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.