Skip to content

fix: add backoff to retry loops - #24653

Merged
mergify[bot] merged 5 commits into
matrixorigin:mainfrom
LeftHandCold:fix/pr-24107-retry-backoff-main
May 28, 2026
Merged

fix: add backoff to retry loops#24653
mergify[bot] merged 5 commits into
matrixorigin:mainfrom
LeftHandCold:fix/pr-24107-retry-backoff-main

Conversation

@LeftHandCold

@LeftHandCold LeftHandCold commented May 27, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue #24654

What this PR does / why we need it:

This ports the still-needed retry-loop fixes from the closed PR #24107 to current main.

  • add capped exponential backoff to remote lock table unlock/getLock retry loops to avoid CPU spinning while preserving the existing unlock cleanup contract
  • add context-aware capped exponential backoff to txn parallelSendWithRetry send failures and response TxnError retries so cancellation exits promptly
  • add targeted tests for the backoff paths and cancellation behavior

The lockop budget=0 change from #24107 is intentionally not reapplied because current main already treats ErrLockTableBindChanged as a bounded retry error.

Tests:

  • GOCACHE=/private/tmp/mo-gocache-pr24107-4 go test -mod=mod ./pkg/lockservice -run 'TestUnlockRemoteWithRetry|TestRemoteRetryBackoffCapped|TestGetLockRemoteWithRetry' -count=1
  • GOCACHE=/private/tmp/mo-gocache-pr24652-review-main go test -mod=mod ./pkg/txn/service -run 'Test_parallelSendWithRetry' -count=1

Note: the lockservice test was run outside the sandbox because it needs to bind unix sockets.

Copilot AI review requested due to automatic review settings May 27, 2026 11:02
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds capped exponential backoff to three previously busy-spin retry loops: remote lock table unlock/getLock and txn service parallelSendWithRetry. The parallelSendWithRetry path is also made context-aware so cancellation exits promptly. Targeted unit tests cover the backoff doubling/cap and cancellation behavior. As noted in the PR description, the lockop budget=0 change from #24107 is intentionally not reapplied because main already classifies ErrLockTableBindChanged as a bounded retry error.

Changes:

  • Add exponential backoff (100ms → 5s) to remoteLockTable.unlock and getLock retry loops, preserving the no-error unlock contract.
  • Add context-aware exponential backoff (100ms → 1s) to service.parallelSendWithRetry, returning nil promptly on ctx cancel.
  • Add unit tests for backoff doubling/cap, get-lock retry path, and cancellation-during-backoff behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
pkg/lockservice/lock_table_remote.go Adds package-level backoff knobs and helpers; inserts sleep+next-backoff into the unlock and getLock retry loops.
pkg/lockservice/lock_table_remote_test.go New tests verifying nextRemoteRetryBackoff doubling/cap and a successful retry path for getLock after an RPC timeout.
pkg/txn/service/service.go Adds context-aware backoff wait and next-backoff helpers; applies them to parallelSendWithRetry send failures; resets backoff after a successful send.
pkg/txn/service/service_test.go Adds a stub retryTestSender and tests that parallelSendWithRetry exits during backoff on cancel and returns nil when ctx is already canceled.

@mergify mergify Bot added the queued label May 28, 2026
@mergify

mergify Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

  • Entered queue2026-05-28 06:22 UTC · Rule: main
  • Checks passed · in-place
  • Merged2026-05-28 07:26 UTC · at 6ce21ab7ff6083d8824390a2321cc2391829be38 · squash

This pull request spent 1 hour 3 minutes 31 seconds in the queue, including 1 hour 2 minutes 22 seconds running CI.

Required conditions to merge
  • #approved-reviews-by >= 1 [🛡 GitHub branch protection]
  • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • github-review-decision = APPROVED [🛡 GitHub branch protection]
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
    • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
    • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
    • check-neutral = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
    • check-skipped = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
    • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
    • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone CI / SCA Test on Ubuntu/x86
    • check-neutral = Matrixone CI / SCA Test on Ubuntu/x86
    • check-skipped = Matrixone CI / SCA Test on Ubuntu/x86
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone CI / UT Test on Ubuntu/x86
    • check-neutral = Matrixone CI / UT Test on Ubuntu/x86
    • check-skipped = Matrixone CI / UT Test on Ubuntu/x86
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
    • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
    • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
    • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
    • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
    • check-neutral = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
    • check-skipped = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Utils CI / Coverage
    • check-neutral = Matrixone Utils CI / Coverage
    • check-skipped = Matrixone Utils CI / Coverage

@mergify
mergify Bot merged commit 3407df9 into matrixorigin:main May 28, 2026
23 of 24 checks passed
@mergify mergify Bot removed the queued label May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working kind/enhancement kind/test-ci size/M Denotes a PR that changes [100,499] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants