This repository was archived by the owner on May 13, 2026. It is now read-only.
ci(e2e-cloud): per-assertion summary + workflow annotations for nightly runs - #833
Merged
Merged
Conversation
…ly runs The previous Summary step used `find -name "*.failed"` against each run-dir, but run-scenario.sh never writes those marker files — so every nightly was silently labelled ✅ PASS in the step summary regardless of outcome. That masked the bob.shared_record_count[Photography] regression in face-discovery-3node, which has been failing every completed nightly since at least 2026-04-24 without anyone noticing. This change: - run-scenario.sh tees stdout+stderr into $SESSION_DIR/run.log so the summary step (and humans, post-mortem on the existing artifact) can grep [PASS]/[FAIL] for one specific run. - New test-framework/scripts/render-ci-summary.sh parses each run-*/run.log and emits, per scenario: overall status (PASS / FAIL /⚠️ INCOMPLETE for SIGTERM/timeout), failing-step name if any, per-assertion PASS/FAIL counts, and a collapsible details block. - Stdout (markdown) → $GITHUB_STEP_SUMMARY; stderr (::error file=… annotations) → job log, where GHA promotes them to top-of-page annotations so a recurring assertion fail is named on every run instead of buried in scrollback. Verified locally against four synthetic fixtures (PASS, assertion-FAIL, step-FAIL with cascading assertion fails, SIGTERM-style INCOMPLETE). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
e2e-cloud.ymlusedfind -name "*.failed"against eachrun-*dir, butrun-scenario.shnever writes those marker files — so every nightly was silently labelled ✅ PASS in the step summary regardless of outcome. That masked thebob.shared_record_count[Photography]regression inface-discovery-3node, which has been failing every completed nightly since at least 2026-04-24 without anyone noticing (independently confirmed against the 2026-04-24 and 2026-05-04 run logs).run-scenario.shnow tees stdout+stderr into$SESSION_DIR/run.log, so the summary step (and humans, post-mortem on the existinge2e-session-logsartifact) can grep[PASS]/[FAIL]for one specific run instead of re-parsing the full workflow job log.test-framework/scripts/render-ci-summary.shparses eachrun-*/run.logand emits, per scenario: overall status (PASS/FAIL/⚠️ INCOMPLETEfor SIGTERM/timeout), the failing-step name if any, per-assertionPASS/FAILcounts, and a collapsible details block. Stdout (markdown) →$GITHUB_STEP_SUMMARY; stderr (::error file=…annotations) → job log, where GHA promotes them to top-of-page annotations so a recurring assertion fail is named on every run instead of buried in scrollback.Net effect on the next nightly
The recurring
bob.shared_record_count[Photography]: expected>=1 actual=0will appear as a top-of-page red annotation ontest-framework/scenarios/face-discovery-3node.yamland as a named row in the summary table, instead of as a generic "scenario failed" line nobody clicks through to investigate.Test plan
bash -n)render-ci-summary.shagainst four syntheticrun-*/run.logfixtures: PASS scenario (1/0), assertion-FAIL scenario (2/1 with named failing assertion), step-FAIL scenario (4 cascading assertion fails + named failing step), SIGTERM-style INCOMPLETE (no PASSED/FAILED marker →::error::annotations on stderr.workflow_dispatch): verify the step summary shows the per-scenario table and thatbob.shared_record_count[Photography]lands as a top-of-page annotation onface-discovery-3node.yaml.Follow-up (separate PR)
Track 2: root-cause why
bob.shared_record_count[Photography]returns 0 inface-discovery-3node. All steps complete cleanly (alice publishes, bob face-searches + connects, alice accepts + shares, bob receives notification) — only the author-attribution check fails. PR #396 plumbedauthor_pub_keythrough query response; investigation should start by capturing bob's record aftershare_recordin a--keep-sessionrun and comparing theauthor_pub_keyfield againstalice.public_keyfromnodes.json.🤖 Generated with Claude Code