💡 Test: maintainer Idea trigger check (3)

[don-petry]

Third trigger check for the feature-ideation workflow, opened by a maintainer. Expected: one enhancement comment. This Discussion will be closed after the check.

[github-actions[bot]]

🔍 Analyst Enhancement: turn the manual trigger check into an automated canary for the Idea path

This comment confirms the maintainer → redispatch → single-idea path works end to end. It is the expected result of this check. It also suggests a way to make this check repeatable, so nobody has to open throwaway Discussions to run it.

Sharper problem framing

The Idea-enhancement path has several layers, and each one has failed in production in a different way. So far the only way to find these failures has been to open real Discussions and look at them:

When (2026-11-03 UTC)	Manual check	What it found	Fix
03:24	—	Enhancements ran on ideas from people who are not maintainers	#1447
12:06	#1450 / #1451	Both runs were skipped, including the maintainer's (wrong author check)	#1452
12:14	#1453 / #1454	redispatch passed, but the dispatched run failed: claude-code-action rejected the github-actions[bot] actor	#1455
12:41	#1458 (this)	Expected to pass	—

The scheduled path has the same blind spot. The Friday scheduled runs on 10-09, 10-16, 10-23 and 10-30 all failed in Gather Project Signals → Install Python jsonschema. On 10-30 the error was Could not open requirements file: .feature-ideation-tooling/scripts/feature-ideation-requirements.txt. That is four weeks without ideation and no alert. daily-org-status reports PRs with failing CI, but it does not report failed scheduled workflow runs.

The real need is a liveness and correctness signal for an agentic workflow that rarely runs and depends on events. The unit tests are good: test/workflows/feature-ideation/*.bats covers redispatch, the context loader, mutations and lint. But the failures above happened at the seams the unit tests can't see: the actor identity seen by the third-party action, the checkout ref for tooling, and the GitHub event semantics.

Prior art and platform constraints

• GitHub event semantics shape the design. Events created with GITHUB_TOKEN do not start new workflow runs. workflow_dispatch and repository_dispatch are the only exceptions, and they are exactly why redispatch works (GitHub Docs: Triggering a workflow). So a canary that creates a Discussion with the job token will never fire discussion: created. A GitHub App token would fire the event, but redispatch filters out user.type == 'Bot' on purpose, so an App-authored canary is also skipped by design. A true end-to-end test from the event onwards needs a human or machine-user credential. Most of the value is available without one (see the options below).
• claude-code-action's actor gate. Bots are rejected unless they are listed in allowed_bots, and the action's own docs warn against '*' (security.md mirror; PR #826, human-actor check). Action bumps come in often (Dependabot PRs moved it 1.0.183 → 1.0.239 in about 3 weeks). Any bump that changes this gate breaks the Idea path silently, which is the strongest argument for a canary that runs after each bump.
• Local emulation won't close the gap. nektos/act runs jobs in Docker but needs hand-made event payloads and local tokens, and it behaves differently from hosted runners (act README; centlinux guide). It cannot exercise the cross-run dispatch or the real action's actor check.
• The synthetic-monitoring pattern. Run the critical path on a schedule with hard assertions against a dedicated test fixture, plus an if: failure() alert step. This is standard practice for web apps (qaskills synthetic monitoring guide; AWS CloudWatch Synthetics overview). It carries over well to agentic CI: the "user journey" is Idea → enhancement comment.

Options

Option	What it covers	Cost / risk	Verdict
A. Run-failure alert — a small job, or a section in daily-org-status, that flags any failed schedule/workflow_dispatch run of feature-ideation.yml	Liveness of every path. It would have caught all 4 Friday failures	Near zero. Read-only actions: read	Do first
B. Pinned canary Discussion + dry_run — one permanent, maintainer-authored ideas Discussion ("🧪 Canary — do not close"). A weekly job (and one after each claude-code-action bump) runs gh workflow run feature-ideation.yml -f target_discussion=<N> -f dry_run=true -f research_depth=quick and then checks the dry-run.jsonl artifact	Context loader → action actor gate → model → mutation helper. That covers everything after redispatch, including the #1455 failure	One quick model run per week. It never consumes the canary: in dry-run the marker is never posted, so load-discussion-context.sh's "already enhanced" check never trips	Do second
C. Fixture-driven redispatch test — extract the redispatch script into scripts/feature-ideation/redispatch.sh (like the other tested helpers) and add bats cases with stubbed gh for write, admin, read, none and Bot authors	The author and category gate. It would have caught #1452	Low. It reuses the existing stubs/gh harness	Do alongside B
D. Full end-to-end with a machine user that opens and closes a real Discussion weekly	Literally everything, including event delivery	Needs a machine-user PAT (a credential to rotate and protect), and pollutes Discussions	Only if B + C miss a real failure

For B, the assertions are concrete. The run must conclude success. dry-run.jsonl must contain exactly one comment_on_discussion entry targeting the canary's node ID, whose body starts with <!-- feature-ideation:enhanced -->, and zero create_discussion entries. A negative canary should also run: dispatch with the number of a non-maintainer test Discussion and assert proceed=false with no model step run, which mirrors #1454.

Risks and unknowns

• Model nondeterminism. Assert on the structure (one comment, marker, right target, no creates), never on the text, or the canary will fail at random.
• Cost drift. Pin research_depth=quick. The run can be cut further by short-circuiting web research when the Discussion title starts with 🧪 Canary, but that adds a special case to the prompt, so watch it.
• Concurrency. feature-ideation.yml uses concurrency: feature-ideation with cancel-in-progress: false, so a canary dispatched near the Friday cron will queue rather than collide. Schedule it for a different day (for example Wednesday).
• Fleet pause. A canary must honour AGENTS_PAUSED and report "paused", not "failed", or it will raise false alarms during fleet pauses.
• Unknown: whether the downstream callers (Broodly, TalkTerm, markets) have the same Friday failures. Option A should be fleet-wide from the start.

Phased validation path

1. Week 1: Add Option A: list feature-ideation.yml runs with conclusion=failure in the last 8 days, in the daily org status for every repo. Check it by confirming it would have flagged 10-09 through 10-30.
2. Week 2: Extract and bats-test redispatch (Option C). Replay 💡 Test: maintainer Idea trigger check #1450–💡 Test: non-maintainer Idea trigger check (2) #1454 as fixtures. The pre-fix(feature-ideation): read the author's repository permission in redispatch #1452 code should fail the maintainer case.
3. Weeks 3–4: Create the pinned canary Discussion and the weekly dry-run dispatch (Option B), plus a trigger on merged Dependabot PRs that touch anthropics/claude-code-action. Run it in parallel with one more manual check to confirm both agree.
4. Retire manual checks: after 4 green canary weeks, stop opening "Test: … trigger check (N)" Discussions.

Success metrics

• Time to detect a broken ideation run: from about 4 weeks today (the Oct incident) to ≤ 24 h.
• Zero throwaway test Discussions per fix, down from 5 created today.
• Every claude-code-action bump PR gets a canary result before the next scheduled Friday run.
• Canary false-failure rate < 1 in 10 runs (that is, the assertions are structural, not about content).
• Canary cost ≤ 1 quick-depth run per week per repo.

Sources: run and PR history of combsmith/.github (#1447, #1452, #1455; feature-ideation runs 2026-10-09 → 2026-11-03), GitHub Docs: Triggering a workflow, claude-code-action security.md, claude-code-action PR #826, nektos/act README, Synthetic monitoring guide, CloudWatch Synthetics.