Skip to content

Extract rollback-safety workflow and add human-override + stale-label cleanup #35396

@erickgonzalez

Description

@erickgonzalez

Description

Three improvements to the Claude AI rollback-safety workflow.

1. Split into a dedicated workflow file

The rollback-safety job used to live inside .github/workflows/ai_claude-orchestrator.yml, where it ran serially alongside the other Claude jobs. It has been extracted into its own standalone workflow file, .github/workflows/ai_claude-rollback-safety.yml, so it can run in parallel with the other Claude workflows (interactive, automatic review, backend reviewer). The new file follows the same pattern as .github/workflows/ai_claude-backend-reviewer.yml and gates on ./.github/actions/security/org-membership-check. The original job was removed from the orchestrator to prevent double execution.

2. Clear stale AI: labels on every new push

Added a new preflight-clear-stale-labels job that runs before the AI evaluation. On every PR push, any pre-existing AI: Safe To Rollback / AI: Not Safe To Rollback label is removed via gh pr edit --remove-label … || true. This fixes the issue where a PR could end up carrying both labels after a subsequent push flipped the verdict — each run now starts from a clean slate.

3. Skip AI evaluation when a human has already classified the PR

The same preflight job is gated off when the PR carries Human: Safe To Rollback or Human: Not Safe To Rollback, meaning the author has already acknowledged the rollback safety of the change. Implemented via contains(github.event.pull_request.labels.*.name, …) in the preflight if:; the skip cascades to claude-rollback-safety-check through needs: + default success() gating. Avoids redundant Claude API calls when a human has already weighed in.

Acceptance Criteria

  • .github/workflows/ai_claude-rollback-safety.yml created with its own security-check, preflight-clear-stale-labels, and claude-rollback-safety-check jobs
  • Original claude-rollback-safety-check job removed from .github/workflows/ai_claude-orchestrator.yml
  • Preflight removes stale AI: labels on every new push so each run is evaluated from scratch
  • Preflight skips AI evaluation when a Human: label is present on the PR
  • actionlint passes on the new workflow file

Test Plan

Validate each improvement with a dummy PR against main. Each PR can be a trivial change (e.g., a README typo) — the behavior being verified is the workflow wiring, not the diff content.

Test 1 — Parallel execution (Improvement 1)

  1. Open a dummy PR with any trivial change.
  2. Open the Actions tab → the triggered run.
  3. Expected: Claude AI Orchestrator and Claude AI Rollback Safety Check appear as separate workflow runs, starting at roughly the same time. The rollback-safety workflow no longer blocks or serializes with the orchestrator's other jobs.

Test 2 — Stale AI: label cleanup (Improvement 2)

  1. On an open dummy PR, manually add the label AI: Not Safe To Rollback.
  2. Push a trivial commit that should clearly be classified as safe (e.g., a comment-only change).
  3. Expected:
    • preflight-clear-stale-labels runs and removes AI: Not Safe To Rollback before the AI job starts.
    • After the workflow finishes, the PR carries only AI: Safe To Rollback — never both labels.
  4. Repeat with the opposite stale label to cover the symmetric case.

Test 3 — Human override (Improvement 3)

  1. On an open dummy PR, add the label Human: Safe To Rollback (and verify no AI: label is present).
  2. Push a trivial commit.
  3. Expected:
    • Both preflight-clear-stale-labels and claude-rollback-safety-check show status Skipped in the Actions tab.
    • No AI: label is added to the PR.
    • No Claude API call is logged for this run.
  4. Repeat with Human: Not Safe To Rollback to cover the symmetric case.

Cleanup

Close the dummy PRs without merging once the three scenarios pass.

Priority

Low

Additional Context

Files touched

  • .github/workflows/ai_claude-rollback-safety.yml — new
  • .github/workflows/ai_claude-orchestrator.yml — removed claude-rollback-safety-check job

Metadata

Metadata

Assignees

Type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions