fix(research): preserve Campaign checkpoints across interruptions - #1100
Merged
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
Recover a Campaign controller interrupted after learning publication or before completion cleanup without changing immutable evidence or entering an unapproved child. Previously, rerunning the learner could change
reused_existingin an already-published report, while deleting the request before the completion marker could make the finalized generation impossible to resume.Persist and validate the first complete learn report and plan before publication. Commit a versioned generation completion record before sensitive-file cleanup, rebuild the cycle summary from that record, and bind completed ACK retries to their original Pod. Status and resume reject corrupted or unsupported completion records. The Monorepo CI control-plane check now invokes the controller syntax, lint, and recovery contract checks so these regressions are exercised remotely.
Issue relationship
None
Validation
At
b28d39aaccc65d6de213fbd7849c221136db181b, including main8c7d05cb617c9516fbde571c5676285e887f0ed5:deployment/aliyun/research/test-campaign-cycle-controller.sh: existing contract flow and 13 added recovery/conflict scenarios passed, including process termination at cleanup/completion boundaries, lost PUT responses, summary-write failure, late parent retry, bounded/no-improvement stops, and local/remote evidence corruption.bash -n deployment/aliyun/research/scripts/campaign-cycle-controller.sh deployment/aliyun/research/test-campaign-cycle-controller.sh: passed.shellcheck -e SC2154 deployment/aliyun/research/scripts/campaign-cycle-controller.sh deployment/aliyun/research/test-campaign-cycle-controller.sh: passed, retaining the existing ERR-trap exclusion.git diff --check: passed..github/workflows/ci.ymlpassed; the controller-script path selectscontrol=true, activating the new CI step. Current-head remote CI is pending.Runtime impact and rollback
Research control-plane checkpoint behavior only. No cloud resource, model, fee, Gate, holdout, order, risk, or paused runtime was changed.
The completion checkpoint format intentionally rejects old empty markers. Preserve historical evidence and use a fresh cycle for acceptance; resuming an older checkpoint requires a separately reviewed migration. Roll back source before adopting the new format, or retain the matching controller version for in-progress new-format cycles; an old controller must not resume them.
Local fault injection uses a local filesystem and fake external ports. It does not prove node-power-loss durability, OSS CSI rename atomicity, or multi-writer fencing. New-image publication, the frozen 31-hour inventory at 12 GiB, and a real parent/child ACK readback remain separate pending acceptance.