Fail project/product agent runs when the agent posts no summary comment#295
Merged
Conversation
Run 30241204433 (2026-07-27, project.yml) reported all-green but the gpt-5-mini agent never made its required `gh issue comment` summary call on orchestration issue #294 -- silence indistinguishable from a genuine clean day. Add an "Assert agent posted a summary" step (after the agent step, before "Close orchestration issue") that queries the orchestration issue's comments and checks for at least one comment NOT matching the workflow's own fixed boilerplate text (cleanup auto-close / close-step auto-close / this step's own recorded-silence comment) -- author login can't distinguish agent from workflow since both post through the same GH_TOKEN identity. On failure it posts a recorded-silence comment naming the run, then a final `if: always()` step (after the close step, so the issue is still closed either way) fails the job with exit 1. Applied to both project.yml (one orchestration issue) and product.yml (one per bot, inside the per-bot matrix leg). Presence only -- content is never judged, so a legitimate clean-day summary still passes. monitor.yml is deliberately untouched: it has no orchestration issue and legitimately files nothing on a healthy run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
Szer
added a commit
that referenced
this pull request
Jul 27, 2026
… agent effort (#297) monitor.yml's change_context step already issued the ArgoCD-history probe BEFORE "Disconnect VPN (post gather)" (confirmed via source + actual step timestamps on runs 30233672995/30240150429) — the step-ordering hypothesis does not hold on current main. What the code did do wrong: it fetched the ArgoCD application JSON TWICE per bot — once via probe_optional() (body discarded) and again, unguarded, to build the displayed history. Under this step's `set -e`/`pipefail`, a transient failure on that second, unguarded curl (even after the probe itself succeeded) aborts the whole gather loop, silently starving every bot processed afterwards of its change_context_manifest.json — exactly the guard's "emitted no valid manifest" fallback text. Add probe_optional_body() to lib.sh: same status/log semantics as probe_optional(), but captures the response body to a file in the same request, so callers needing the body don't issue a second, unguarded fetch. monitor.yml's change_context step now fetches the ArgoCD application once and reads history from the captured file. PR #293's required-vs-optional split is unchanged: probe failure still yields "degraded(optional): ..." and the bot still lands in clean_bots (see fixture test in the PR description). Also raises project.yml's codex-action effort from low to medium per owner approval (PR #295 made a silent-summary miss visible; this addresses the next lever). product.yml and monitor.yml agent effort are untouched. Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Run 30241204433 of
project.yml(2026-07-27 ~05:57 UTC) reportedcleanup: success,project: success-- every step green. It created orchestration issue #294 and auto-closed it, but the agent never posted its required summary comment (.github/prompts/project.md: "Your final tool call should always be thegh issue commentthat posts the summary"). #294's only comment is the workflow's own auto-close message. A manually-dispatched run 40 minutes earlier, same workflow/prompt/model, DID post a summary and filed #283 -- so this is non-deterministic LLM behaviour (gpt-5-mini,effort: low), not a config difference, and will recur. A silent no-op is indistinguishable from a genuine clean day, which is exactly the failure class.github/AGENT-FLOWS-REDESIGN.md§1 exists to eliminate.project.ymlandproduct.yml(product: inside the per-botagentmatrix leg, one orchestration issue per bot), placed after the agent step and before "Close orchestration issue". It runsgh issue view <n> --json commentsand checks for at least one comment whose body does not match the workflow's own fixed boilerplate text (cleanup's stale-issue auto-close, the "Close orchestration issue" step's comment, or this step's own recorded-silence comment). Author login can't be used to distinguish "agent" from "workflow" here -- both post through the sameGH_TOKEN(defaultGITHUB_TOKEN) identity, so only the fixed text of the workflow's own comments separates them.if: always(), reads the assert step's output andexit 1s if no summary was posted. This ordering keeps the existingif: always()close step working (the orchestration issue is always closed, whether the agent spoke or not) while still making the overall run fail.monitor.ymlis deliberately not touched: it has no orchestration issue and legitimately files nothing on a healthy run, so asserting output there would be wrong.Not changed (explicit deviation-avoidance)
effort: lowonproject.yml's agent invocation is unchanged, per the brief. See PR discussion / session report for a recommendation on whether to raise it, given this is the second known low-effort project-agent behavioural gap (the first was #222/#271, silent-no-op is a new one) -- not changed here since the brief scoped this PR to the assertion mechanism only.Validation
actionlint(v1.7.12, shellcheck on PATH) over.github/workflows/project.ymland.github/workflows/product.yml(and the full.github/workflows/**directory): 0 findings.bash -non everyrun:block touched: all pass.gh, no network) against three cannedgh issue view --json commentspayloads:agent_posted=true, fail step exits 0.agent_posted=false, the recorded-silence comment text is printed, fail step exits 1.Test plan
actionlint .github/workflows/project.yml .github/workflows/product.yml-- 0 findingsbash -non all touchedrun:blocksghproject.yml/product.ymlto confirm the step behaves against livegh issue viewoutput (not exercised here -- no real issues were created or posted to)🤖 Generated with Claude Code
https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU