Skip to content
This repository was archived by the owner on May 13, 2026. It is now read-only.

ci(e2e-cloud): per-assertion summary + workflow annotations for nightly runs - #833

Merged
shiba4life merged 1 commit into
mainfrom
ci/e2e-summary-per-assertion-annotations
May 4, 2026
Merged

ci(e2e-cloud): per-assertion summary + workflow annotations for nightly runs#833
shiba4life merged 1 commit into
mainfrom
ci/e2e-summary-per-assertion-annotations

Conversation

@shiba4life

Copy link
Copy Markdown
Collaborator

Summary

  • The previous Summary step in e2e-cloud.yml used find -name "*.failed" against each run-* dir, but run-scenario.sh never writes those marker files — so every nightly was silently labelled ✅ PASS in the step summary regardless of outcome. That masked the bob.shared_record_count[Photography] regression in face-discovery-3node, which has been failing every completed nightly since at least 2026-04-24 without anyone noticing (independently confirmed against the 2026-04-24 and 2026-05-04 run logs).
  • run-scenario.sh now tees stdout+stderr into $SESSION_DIR/run.log, so the summary step (and humans, post-mortem on the existing e2e-session-logs artifact) can grep [PASS]/[FAIL] for one specific run instead of re-parsing the full workflow job log.
  • New test-framework/scripts/render-ci-summary.sh parses each run-*/run.log and emits, per scenario: overall status (PASS / FAIL / ⚠️ INCOMPLETE for SIGTERM/timeout), the failing-step name if any, per-assertion PASS/FAIL counts, and a collapsible details block. Stdout (markdown) → $GITHUB_STEP_SUMMARY; stderr (::error file=… annotations) → job log, where GHA promotes them to top-of-page annotations so a recurring assertion fail is named on every run instead of buried in scrollback.

Net effect on the next nightly

The recurring bob.shared_record_count[Photography]: expected>=1 actual=0 will appear as a top-of-page red annotation on test-framework/scenarios/face-discovery-3node.yaml and as a named row in the summary table, instead of as a generic "scenario failed" line nobody clicks through to investigate.

Test plan

  • Bash syntax check on both modified scripts (bash -n)
  • Local smoke test of render-ci-summary.sh against four synthetic run-*/run.log fixtures: PASS scenario (1/0), assertion-FAIL scenario (2/1 with named failing assertion), step-FAIL scenario (4 cascading assertion fails + named failing step), SIGTERM-style INCOMPLETE (no PASSED/FAILED marker → ⚠️ INCOMPLETE annotation). All four cases render correct markdown and emit the right ::error:: annotations on stderr.
  • Next scheduled nightly run (or manual workflow_dispatch): verify the step summary shows the per-scenario table and that bob.shared_record_count[Photography] lands as a top-of-page annotation on face-discovery-3node.yaml.

Follow-up (separate PR)

Track 2: root-cause why bob.shared_record_count[Photography] returns 0 in face-discovery-3node. All steps complete cleanly (alice publishes, bob face-searches + connects, alice accepts + shares, bob receives notification) — only the author-attribution check fails. PR #396 plumbed author_pub_key through query response; investigation should start by capturing bob's record after share_record in a --keep-session run and comparing the author_pub_key field against alice.public_key from nodes.json.

🤖 Generated with Claude Code

…ly runs

The previous Summary step used `find -name "*.failed"` against each
run-dir, but run-scenario.sh never writes those marker files — so every
nightly was silently labelled ✅ PASS in the step summary regardless of
outcome. That masked the bob.shared_record_count[Photography] regression
in face-discovery-3node, which has been failing every completed nightly
since at least 2026-04-24 without anyone noticing.

This change:
- run-scenario.sh tees stdout+stderr into $SESSION_DIR/run.log so the
  summary step (and humans, post-mortem on the existing artifact) can
  grep [PASS]/[FAIL] for one specific run.
- New test-framework/scripts/render-ci-summary.sh parses each
  run-*/run.log and emits, per scenario: overall status (PASS / FAIL /
  ⚠️ INCOMPLETE for SIGTERM/timeout), failing-step name if any,
  per-assertion PASS/FAIL counts, and a collapsible details block.
- Stdout (markdown) → $GITHUB_STEP_SUMMARY; stderr (::error file=…
  annotations) → job log, where GHA promotes them to top-of-page
  annotations so a recurring assertion fail is named on every run
  instead of buried in scrollback.

Verified locally against four synthetic fixtures (PASS, assertion-FAIL,
step-FAIL with cascading assertion fails, SIGTERM-style INCOMPLETE).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@shiba4life
shiba4life enabled auto-merge May 4, 2026 18:54
@shiba4life
shiba4life added this pull request to the merge queue May 4, 2026
Merged via the queue into main with commit c404b26 May 4, 2026
12 checks passed
@shiba4life
shiba4life deleted the ci/e2e-summary-per-assertion-annotations branch May 4, 2026 19:04
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant