Skip to content

ci(release): schedule nonblocking daily tags - #8483

Closed
cjagwani wants to merge 8 commits into
mainfrom
codex/scheduled-nonblocking-tag
Closed

ci(release): schedule nonblocking daily tags#8483
cjagwani wants to merge 8 commits into
mainfrom
codex/scheduled-nonblocking-tag

Conversation

@cjagwani

@cjagwani cjagwani commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Before this change, the release skill required a candidate-bound E2E ledger before a maintainer-confirmed tag. After this change, Carlos's #8477 runs complete E2E asynchronously on every push to main, while a separate workflow tags the current main at 4 PM America/Los_Angeles regardless of E2E state and merging continues normally.

This PR is intentionally stacked on #8477 so that PR remains the sole owner of E2E-on-main behavior.

Changes

  • Add a DST-aware 4 PM Los Angeles workflow that skips empty releases, verifies the dated changelog, creates a signed patch tag through a protected signer environment, and directly invokes release promotion and label housekeeping.
  • Add a restricted scheduled mode to the existing tag script. It permits later merges while still rejecting an unreachable target, a changed latest semver, an existing tag, or an untrusted workflow context.
  • Remove the obsolete pre-tag E2E ledger command, implementation, and tests.
  • Update maintainer skills and policy to make E2E asynchronous and advisory: agents stabilize failures from 4 PM through 8 AM while merges remain open.
  • Add release workflow, tag-race, policy, and opaque-input watch-trigger coverage.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: This changes maintainer automation and policy only; no public CLI, configuration, runtime default, or support claim changes.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Trusted workflow identity is enforced in the tag script, the signing key is confined to the protected release-tag environment and runner temp storage, credentials are not persisted by checkout, and the boundaries are covered by security source-shape and executable tag tests.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: No docs/ page owns the daily tag, merge cadence, or E2E authorization policy. Carlos's ci(e2e): run every workflow E2E on main #8477 already covers E2E-on-main in contributor E2E READMEs; the next normal pre-tag changelog pass can record the infrastructure change.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npm exec vitest run -- test/release-latest-tag.test.ts test/release-latest-tag-workflow.test.ts test/release-daily-tag-workflow.test.ts test/maintainer-skills-policy.test.ts test/maintainer-e2e-skill.test.ts test/vitest-watch-triggers.test.ts --testTimeout=30000 (65 passed)
  • Applicable broad gate passed — Not applicable; normal hooks and the focused release, workflow, skill, policy, and watch-trigger suites cover this change.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

Summary by CodeRabbit

  • New Features

    • Added an automated daily release-tag workflow scheduled for 4 PM Pacific time.
    • Release tags are signed and automatically promoted to latest.
    • Added manual recovery support for scheduled workflow failures and release exceptions.
    • Release tags can proceed independently of asynchronous E2E results.
  • Improvements

    • E2E testing is now diagnostic and handled after tagging, with results tied to specific commits.
    • Improved validation for release plans, signing, tag selection, and workflow-triggered promotions.

@cjagwani cjagwani self-assigned this Aug 6, 2026
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The release process now uses a scheduled 4 PM Los Angeles tag workflow. E2E runs are asynchronous diagnostics and do not block merges or tags. Manual recovery, signed tagging, changelog validation, exact-tag promotion, and workflow contract tests were added or updated.

Changes

Release automation

Layer / File(s) Summary
Nonblocking release policy
.agents/skills/nemoclaw-maintainer-*/**, .agents/skills/nemoclaw-skills-guide/SKILL.md
Maintainer guidance now describes scheduled tagging, manual recovery, changelog requirements, label handling, and overnight E2E triage. The pre-tag E2E evidence flow was removed.
Scheduled daily tag workflow
.github/workflows/release-daily-tag.yaml, ci/source-shape-test-budget.json
A scheduled workflow prepares patch releases, validates dated changelogs, creates signed tags, uploads artifacts, and promotes the generated tag.
Scheduled cut and tag selection
scripts/release-cut-tag.sh, .github/workflows/release-latest-tag.yaml
Scheduled tag cuts require canonical workflow context and current remote semver state. The reusable latest-tag workflow accepts an explicit tag input.
E2E and workflow validation
test/*release*, test/maintainer-*.test.ts, test/helpers/vitest-watch-triggers.ts
Tests now verify nonblocking E2E behavior, scheduled-cut safeguards, workflow contracts, exact-tag promotion, and watch triggers.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler
  participant DailyTagWorkflow
  participant ReleaseCutScript
  participant LatestTagWorkflow
  Scheduler->>DailyTagWorkflow: activate the 4 PM Los Angeles schedule
  DailyTagWorkflow->>ReleaseCutScript: create the planned signed tag
  ReleaseCutScript-->>DailyTagWorkflow: return the release receipt
  DailyTagWorkflow->>LatestTagWorkflow: promote the generated semver tag
Loading

Possibly related PRs

Suggested labels: area: ci, chore

Suggested reviewers: cv, ericksoa

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: scheduled daily release tags that do not depend on E2E results.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/scheduled-nonblocking-tag

Comment @coderabbitai help to get the list of available commands.

@cjagwani

cjagwani commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

Deployment prerequisite: the repository does not currently have a release-tag environment or release-signing secret/identity variables. Before this workflow can cut its first scheduled tag, a release admin must create that protected environment and configure NEMOCLAW_RELEASE_TAG_SIGNING_KEY, NEMOCLAW_RELEASE_TAG_SIGNER_NAME, and NEMOCLAW_RELEASE_TAG_SIGNER_EMAIL. The workflow fails closed when any value is absent.

@cjagwani

cjagwani commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Base automatically changed from ci/e2e-all-workflow-main to main August 6, 2026 16:17

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md:
- Line 60: Update the release instructions near the QA summary to direct
maintainers to monitor the “Release / Daily Tag” workflow rather than wait for
confirmation or manually create the tag. Restrict manual tag creation to the
documented recovery cases, preserving asynchronous E2E triage without blocking
the protected scheduled release path.

In `@test/release-latest-tag.test.ts`:
- Around line 556-559: Extend the rejected scheduled-cut assertions in the test
cases around cutResult and localTagObject to verify that origin does not contain
refs/tags/v0.0.2 after each rejection. Use the existing public
remote-observation mechanism rather than relying only on localTagObject, and
preserve the current status, error-message, and local-tag assertions.

In `@test/vitest-watch-triggers.test.ts`:
- Around line 71-72: Remove the two release workflow fixture entries from the
OPAQUE_INPUTS collection in the watch-trigger test, while preserving the focused
mappings in the related watch-trigger helper and all other opaque inputs.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2dcc2791-d687-4c9d-93dc-70ce10910287

📥 Commits

Reviewing files that changed from the base of the PR and between 31a6c3f and 212cd89.

📒 Files selected for processing (21)
  • .agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md
  • .agents/skills/nemoclaw-maintainer-cut-release-tag/scripts/release-e2e-evidence.mts
  • .agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md
  • .agents/skills/nemoclaw-maintainer-e2e/SKILL.md
  • .agents/skills/nemoclaw-maintainer-evening/SKILL.md
  • .agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md
  • .agents/skills/nemoclaw-maintainer-policies/references/release-train.md
  • .agents/skills/nemoclaw-skills-guide/SKILL.md
  • .github/workflows/release-daily-tag.yaml
  • .github/workflows/release-latest-tag.yaml
  • ci/source-shape-test-budget.json
  • package.json
  • scripts/release-cut-tag.sh
  • test/helpers/vitest-watch-triggers.ts
  • test/maintainer-e2e-skill.test.ts
  • test/maintainer-skills-policy.test.ts
  • test/release-daily-tag-workflow.test.ts
  • test/release-e2e-evidence.test.ts
  • test/release-latest-tag-workflow.test.ts
  • test/release-latest-tag.test.ts
  • test/vitest-watch-triggers.test.ts
💤 Files with no reviewable changes (3)
  • package.json
  • test/release-e2e-evidence.test.ts
  • .agents/skills/nemoclaw-maintainer-cut-release-tag/scripts/release-e2e-evidence.mts

3. **Evening** (`/nemoclaw-maintainer-evening`) — Check shipped work and the pre-tag changelog PR.
Confirm that the pre-tag changelog PR contains `docs/changelog/YYYY-MM-DD.mdx` for the release.
Identify open items and prepare the QA summary. Record the release SHA and required E2E evidence.
Identify open items and prepare the QA summary. Record the release SHA and asynchronous E2E runs that still need overnight triage; E2E does not block the tag or merging.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Replace the manual tag instruction.

Line 60 defines E2E as asynchronous, but the next instruction still tells the maintainer to cut the tag after confirmation. The normal tag path has no maintainer confirmation step.

Direct the maintainer to monitor Release / Daily Tag. Reserve manual tag creation for the documented recovery cases. Otherwise, a maintainer can bypass the protected scheduled release path.

Based on release policy context, normal tagging is scheduled and manual tagging is recovery-only.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md at line 60,
Update the release instructions near the QA summary to direct maintainers to
monitor the “Release / Daily Tag” workflow rather than wait for confirmation or
manually create the tag. Restrict manual tag creation to the documented recovery
cases, preserving asynchronous E2E triage without blocking the protected
scheduled release path.

Comment on lines +556 to +559
expect(cutResult.status).not.toBe(0);
expect(cutResult.stderr).toContain("--scheduled requires a schedule event");
expect(localTagObject(fixture, "v0.0.2")).toBe("");
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Assert that rejected cuts do not create a remote tag.

localTagObject checks only the fixture worktree. It does not validate the release automation boundary.

Also assert that refs/tags/v0.0.2 is absent from origin after each rejected scheduled cut.

Proposed test update
     expect(cutResult.stderr).toContain("--scheduled requires a schedule event");
     expect(localTagObject(fixture, "v0.0.2")).toBe("");
+    expect(remoteCommit(fixture, "refs/tags/v0.0.2")).toBe("");
     expect(cutResult.stderr).toContain("Latest remote semver tag changed from v0.0.1 to v0.0.3");
     expect(localTagObject(fixture, "v0.0.2")).toBe("");
+    expect(remoteCommit(fixture, "refs/tags/v0.0.2")).toBe("");

As per path instructions, prefer observable outcomes through the public boundary over private-shape assertions.

Also applies to: 571-573

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/release-latest-tag.test.ts` around lines 556 - 559, Extend the rejected
scheduled-cut assertions in the test cases around cutResult and localTagObject
to verify that origin does not contain refs/tags/v0.0.2 after each rejection.
Use the existing public remote-observation mechanism rather than relying only on
localTagObject, and preserve the current status, error-message, and local-tag
assertions.

Source: Path instructions

Comment on lines +71 to +72
".github/workflows/release-daily-tag.yaml",
".github/workflows/release-latest-tag.yaml",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Remove the release workflow paths from OPAQUE_INPUTS.

OPAQUE_INPUTS classifies these paths as inputs that must not select focused tests. test/helpers/vitest-watch-triggers.ts Lines 127-134 maps the same paths to focused release-workflow tests.

Keep the focused mappings. Remove these two fixture entries. Otherwise, this test either fails or asserts behavior that conflicts with the watch-trigger contract.

Based on the provided watch-trigger context, opaque inputs must not act as focused test triggers.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/vitest-watch-triggers.test.ts` around lines 71 - 72, Remove the two
release workflow fixture entries from the OPAQUE_INPUTS collection in the
watch-trigger test, while preserving the focused mappings in the related
watch-trigger helper and all other opaque inputs.

@cjagwani

cjagwani commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #8485. The release remains semi-automatic: an agent verifies docs and housekeeping, an authorized maintainer confirms and signs locally, and E2E remains asynchronous. The scheduled workflow, release bot, and Actions signing-secret design were removed. A replacement PR was required because this repository does not permit rewriting the published branch after rebasing onto Carlos's merged #8477.

@cjagwani cjagwani closed this Aug 6, 2026
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / low confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: Partial review preserved 0 canonical finding(s) and 3 terminology decision(s) before the advisor stopped.

Model lanes

  • GPT-5.6 Terra (primary): Failed after a partial review · low confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Failed

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

3 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • define — asynchronous E2E stabilization at .agents/skills/nemoclaw-maintainer-policies/references/release-train.md:46: Define the term at first use as post-merge E2E triage that does not delay tags or merges.
  • established — empty selectors at .agents/skills/nemoclaw-maintainer-e2e/SKILL.md:327: Keep the established term and state the selected test set where selection behavior matters.
  • justified — manual recovery at .agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md:54: Keep the modifier and state the conditions that select this recovery path.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite against this exact revision.

Recommended E2E: cloud-inference, cloud-onboard, llama-cpp-dgx-spark-qualification, managed-image-multiarch-startup, security-posture, inference-routing, network-policy

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants