Skip to content

fix: fail closed when draining CN pipelines - #617

Merged
loveRhythm1990 merged 2 commits into
matrixorigin:mainfrom
loveRhythm1990:lr90/fix-cn-drain-pipeline-safety
Sep 2, 2026
Merged

fix: fail closed when draining CN pipelines#617
loveRhythm1990 merged 2 commits into
matrixorigin:mainfrom
loveRhythm1990:lr90/fix-cn-drain-pipeline-safety

Conversation

@loveRhythm1990

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

Fixes #616

Related to matrixorigin/matrixone#27776

What this PR does / why we need it:

During CNClaim migration, a source CN can still be executing remote pipelines after its ordinary frontend sessions reach zero. The operator did not enable GetPipelineInfo for verified MO 3.x and 4.x releases, so those pipelines could be treated as zero and the source CN could be reclaimed early.

This change:

  • enables PipelineInfo for the verified MO 3.x and 4.x release lines;
  • records whether session, pipeline, and replica counts were successfully observed in the current refresh;
  • invalidates observation evidence before each refresh so failed or partial queries cannot reuse an old zero;
  • treats legacy score annotations as unsafe until a fresh observation succeeds;
  • requires all observations needed by the current MO version to be both current and zero before normal reclaim;
  • preserves the existing force-timeout, unsupported-version, and CN-absent-from-HAKeeper behavior.

Special notes for your reviewer:

The feature gate intentionally remains explicit per MO major release. Unknown or unverified major versions retain the existing compatibility behavior and do not enable version-specific RPCs.

No end-to-end test was run locally. Static diff validation passed, and 62 tests in api/core/v1alpha1 and pkg/controllers/common passed. The cnstore test binary cannot be linked on the local macOS environment because libmo is unavailable; CI should run the full suite in the project build environment.

Additional documentation:

The incident mechanism and acceptance criteria are documented in #616.

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@loveRhythm1990
loveRhythm1990 merged commit 2d55b5b into matrixorigin:main Sep 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[S0][CN Draining] Prevent reclaiming source CNs with active remote pipelines

2 participants