Skip to content

test(e2e): keep calibration ancestry durable - #8023

Merged
jyaunches merged 2 commits into
mainfrom
codex/fix-calibration-pr-ref-ancestry
Jul 31, 2026
Merged

test(e2e): keep calibration ancestry durable#8023
jyaunches merged 2 commits into
mainfrom
codex/fix-calibration-pr-ref-ancestry

Conversation

@prekshivyas

@prekshivyas prekshivyas commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

Calibration validation now uses only durable workflow revisions for local Git ancestry. Exact tested revisions remain bound to their run and job receipts, but closing an evidence-source PR no longer breaks unrelated CI when its ref disappears.

Related Issue

Fixes #8022.

Changes

  • Preserve exact testedSha, run ID, job ID, and run URL receipt validation and require five distinct tested revisions.
  • Keep the restart-safe change ancestor, workflow-head ancestor, unchanged runtime-input boundary, and derived-budget checks.
  • Stop resolving PR-only tested commits through the local checkout because GitHub can stop advertising those objects after a PR closes.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: This changes only deterministic internal calibration validation and no user-facing command, configuration, workflow, default, or error contract.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Exact-head nine-category security and provenance review passed with no findings: test(e2e): keep calibration ancestry durable #8023 (comment)
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: The changed test comment distinguishes immutable run receipt identity from the durable workflow revision used for local ancestry; no supported product or documentation surface changes.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — fresh worktree reproduced the missing-PR-ref failure before the change; npx vitest run test/onboard-performance-config-schema.test.ts passes 14/14 afterward. Repository, project-membership, source-shape, test-size, Biome, and diff checks pass.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Required PR CI is pending.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Prekshi Vyas prekshiv@nvidia.com

Summary by CodeRabbit

  • Tests
    • Expanded startup calibration coverage to verify five distinct tested revisions and their exact run mappings.
    • Improved ancestry validation by checking the workflow’s head revision while preserving verification of the revision actually tested.
    • Added coverage for cases where the tested revision is no longer advertised by the source control provider, improving confidence in calibration results and recorded verification details.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The startup calibration schema test now requires five distinct tested SHAs. It validates ancestry through workflowHeadSha while retaining exact testedSha values in run and job receipts.

Changes

Calibration test validation

Layer / File(s) Summary
Calibration revision and ancestry assertions
test/onboard-performance-config-schema.test.ts
The test requires five distinct tested revisions, verifies each run-to-testedSha mapping, and validates ancestry through workflowHeadSha.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested labels: area: e2e

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The test updates address issue #8022 by validating durable ancestry, exact receipt mappings, and distinct tested revisions without local PR-ref reachability.
Out of Scope Changes check ✅ Passed The changes are limited to calibration schema tests and directly support the linked issue objectives.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: making calibration ancestry validation durable in end-to-end tests.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-calibration-pr-ref-ancestry

Comment @coderabbitai help to get the list of available commands.

@prekshivyas

Copy link
Copy Markdown
Collaborator Author

Exact-head security and provenance review

PASS at a984a48ac1409a7818e17161fac1cace979cf528 against base aa187e19b65acd0f2f7309e65956fd98a2d7d501; no findings.

The change removes only a local ancestry lookup for testedSha values that can be PR-only commits and therefore disappear from advertised Git refs. It retains the restart-safe change-to-workflow-head ancestry check, the workflow-head-to-verified-runtime-input boundary, unchanged runtime-input diff validation, derived-budget arithmetic, exact run/job URLs, distinct run IDs and job IDs, distinct tested revisions, and an explicit immutable run ID-to-tested SHA mapping.

  1. Secrets and Credentials — PASS. No credentials, secrets, tokens, or credential handling changed.
  2. Input Validation and Data Sanitization — PASS. Exact 40-character SHA validation and exact receipt mappings remain enforced.
  3. Authentication and Authorization — PASS. No authentication, authorization, scope, or permission surface changed.
  4. Dependencies and Third-Party Libraries — PASS. No dependency or supply-chain input changed.
  5. Error Handling and Logging — PASS. A clean checkout no longer crashes on an unadvertised PR object; no internal or sensitive output is added.
  6. Cryptography and Data Protection — PASS. No cryptographic or protected-data behavior changed.
  7. Configuration and Security Headers — PASS. No runtime, container, endpoint, network, workflow permission, or web-security configuration changed.
  8. Security Testing — PASS. A fresh worktree reproduced the missing-ref failure before the change; the calibration suite passes 14/14 afterward. Exact receipt identity is pinned in executable assertions.
  9. System Security — PASS. Durable ancestry remains fail-closed, while ephemeral PR revision identity is validated without making repository tests depend on deleted refs or network access. The change does not alter runtime privileges or evidence collection.

File reviewed: test/onboard-performance-config-schema.test.ts.

@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Second-opinion E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: None

1 optional E2E recommendation
  • full-e2e

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@github-code-quality

github-code-quality Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit a984a48 in the codex/fix-calibratio... branch remains at 96%, unchanged from commit 1e57000 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit a984a48 in the codex/fix-calibratio... branch remains at 81%, unchanged from commit 60b33ab in the main branch.

Show a code coverage summary of the most impacted files.
File main 60b33ab codex/fix-calibratio... a984a48 +/-
src/lib/domain/.../connect-env.ts 97% 89% -8%
src/lib/actions...ntime-health.ts 91% 85% -6%
src/lib/state/o...oard-session.ts 91% 91% 0%
src/lib/actions...dbox/connect.ts 77% 77% 0%
src/lib/onboard...ime-boundary.ts 84% 84% 0%
src/lib/actions...ess-recovery.ts 81% 82% +1%
src/lib/onboard...eway-service.ts 82% 83% +1%
src/lib/actions...troy-gateway.ts 84% 86% +2%
src/lib/actions...sor-relaunch.ts 85% 92% +7%
src/lib/core/ga...eway-address.ts 92% 100% +8%

Updated July 31, 2026 18:52 UTC

@jyaunches
jyaunches merged commit 299050f into main Jul 31, 2026
76 of 86 checks passed
@jyaunches
jyaunches deleted the codex/fix-calibration-pr-ref-ancestry branch July 31, 2026 19:00
senthilr-nv added a commit that referenced this pull request Aug 1, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical dated changelog entry for `v0.0.100` so the
maintainer release plan can verify the pre-tag documentation
prerequisite. The entry summarizes the user-facing changes merged since
`v0.0.99` and links to the relevant guides.

## Changes

- Add `docs/changelog/2026-07-31.mdx` with the exact `## v0.0.100`
heading.
- Cover restored OpenClaw pairing, transactional replacement, Deep
Agents Code, onboarding recovery, lifecycle cleanup, Hermes builds, host
provenance, documentation, and trusted E2E evidence.
- Distinguish active Docker and Kubernetes runtime-bundle enforcement
from the still-inactive managed shared-state transaction foundation.

## Source Coverage

The release entry maps the doc-impacting merged PRs in the
`v0.0.99..main` release range to `docs/changelog/2026-07-31.mdx`: #8021,
#8024, #7973, #8028, #7947, #7788, #7884, #8023, #7969, #8020, #7989,
#8000, #7907, #7942, #7567, #8013, #7955, #8017, #8014, #8015, #7629,
#7644, #7821, #7971, and #7991.

PR #7974 was reviewed after the final rebase and excluded because it
changes internal maintainer-skill attribution policy and tests only; it
does not change a user-facing product or documentation surface.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification: the
changelog contract test validates the dated entry, version heading, SPDX
form, and route constraints.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/changelog/2026-07-31.mdx`; exact-head review passed
for `6093f44f`; writing rules and documentation style reviewed; `npx
vitest run test/changelog-docs.test.ts` passed 6/6; `npm run docs`
passed with zero Fern errors and two generic Fern upgrade notices.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 6093f44 -->
<!-- docs-review-agents-blob-sha: 3dd7c24 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable; no DGX Station host script changed.
- Station profile/scenario: Not applicable.
- Result: Not applicable.
- Supporting evidence: Not applicable.

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npx
vitest run test/changelog-docs.test.ts` passed 6/6 at `6093f44f`.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to a dated
prose-only release entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
validation passed with zero errors; Fern emitted two generic upgrade
notices.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the changelog entry has the required parser-safe MDX SPDX header;
dated changelog entries intentionally do not use page frontmatter.

---

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Added release notes for v0.0.100.
* Documented improvements to restore pairing, sandbox replacement,
onboarding recovery, lifecycle cleanup, runtime handling, build support,
host readiness, and end-to-end validation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
prekshivyas pushed a commit that referenced this pull request Aug 1, 2026
<!-- markdownlint-disable MD041 -->
## Summary

`hermes status` reported a running foreground gateway as stopped because
NemoClaw renames the managed entrypoint to `hermes.real`, while Hermes
recognized only `hermes` and `hermes.exe`. This change applies a
SHA-pinned, exact-shape image patch that recognizes NemoClaw's managed
basename without changing Hermes's gateway subcommand grammar.

## Related Issue

Closes #7804.

## Changes

- Add an idempotent image-time patcher that adds `hermes.real` to
Hermes's gateway entry-token allowlist and fails on upstream
source-shape drift.
- Pin and apply the patcher in the Hermes image, then probe the real
installed matcher during the image build.
- Register the installed matcher assertion in the checked-in image probe
runner so BuildKit and legacy OpenShell gateway builders execute the
same proof.
- Test the renamed and upstream entrypoints, both detection paths,
negative subcommands, look-alike basenames, idempotence, source drift,
payload placement, and digest synchronization.
- Classify the build-only patcher SHA as an integrity pin in the
managed-startup Docker input inventory required by current `main`.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: The change restores existing
direct Hermes process detection. It does not change documented host
commands, configuration, workflows, defaults, or lifecycle behavior.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Codex Desktop security
review passed all nine repository categories for exact head
`504b365f2da5fcb8581edfba724e02686f30faa0`; refer to the [exact-head
agent review
evidence](#7885 (comment)).
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Exact-State Evidence

- PR SHA: `504b365f2da5fcb8581edfba724e02686f30faa0`.
- Base SHA: `299050fc0563db0cd3a803298a6fb986dcfbb745` from
`upstream/main`.
- Refresh: signed merge commit
`504b365f2da5fcb8581edfba724e02686f30faa0` preserves both signed
canonical-base refresh `9a42cffe6af5e4944207b3c5bb3badba15099171` and
concurrent GitHub Verified branch update
`f1cfeb15a63ab6f529fe1883c1cc1bb919b0ea6c`. Both parents have the same
tree and integrate tested base
`299050fc0563db0cd3a803298a6fb986dcfbb745`, so the push remains
fast-forward and non-force.
- Current-main integration: signed commit
`8c673831c0a48f3aed0f40efad1b4af64a0b5475` routes the unchanged
installed matcher assertions through current main's builder-independent
Hermes probe runner. The complete seven-file diff has stable patch ID
`da20cca51b92f870fbe7fb713ed07bc4066bf01d`.
- Current-base regression: `test(e2e): keep calibration ancestry durable
(#8023)` at `299050fc0` fixes the unrelated orphaned closed-PR
calibration reference that failed the preceding required CLI shard. The
refreshed calibration contract passes 14/14; no evidence from that
failed head/base pair is reused.
- Product-scope gate: **PASS**, independently of GitHub merge state. The
PR fixes issue #7804 in the existing Tested Hermes agent surface
documented in `docs/reference/platform-support.mdx`. It adds no
integration, configuration, lifecycle, ownership, or compatibility
surface.
- Contributor compliance: the PR includes the contributor's DCO
declaration. All eighteen displayed commits must remain GitHub Verified
at the final gate.

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: No documentation or Fern paths changed. The review covered
the seven-file exact-head diff, comments, docstrings, probe naming, test
titles, terminology, structure, voice, and code-sample presentation.
Focused and calibration regression tests passed 160/160, all diff-aware
hook stages passed, and `git diff --check` passed.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 504b365 -->
<!-- docs-review-agents-blob-sha: c052d60 -->

## DGX Station Hardware Evidence

<!-- Required only when scripts/prepare-dgx-station-host.sh changes.
Maintainers must review the linked evidence before approving or merging.
This is human-reviewed evidence, not authenticated hardware provenance.
Exceptional bypasses use existing repository governance and must be
documented on the PR. -->
- [ ] Tested on DGX Station
- Tested commit: Not applicable; this PR does not change
`scripts/prepare-dgx-station-host.sh`.
- Station profile/scenario: Not applicable.
- Result: Not applicable.
- Supporting evidence: Not applicable.

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: the
seven-file Hermes patch, probe-runner, image-layout, managed-startup,
profile-policy, cron-runtime, Discord-recovery, and refreshed
calibration regression suite passed 160/160 on `504b365f2`; the security
specialist also passed 132/132 exact-head Hermes/image and managed
profile contracts.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable; the exact
image patcher, checked-in probe runner, image-layout, and Docker input
inventory contracts are covered by the focused suite and exact-head CI.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved Hermes gateway detection for renamed entrypoints and
supported runtime commands.
  * Prevented false positives from lookalike or unrelated commands.

* **Reliability**
* Added validation to ensure the gateway patch is applied safely and
consistently.
* Added integrity checks and image-build probes to verify the behavior
automatically.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Co-authored-by: Senthil Ravichandran <senthilr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI calibration ancestry check depends on deleted PR refs

2 participants