Skip to content

fix(workflows): restore Actions Expression interpolation in Claude prompts#5520

Merged
MarkusNeusinger merged 1 commit intomainfrom
fix/workflow-prompt-interpolation
Apr 29, 2026
Merged

fix(workflows): restore Actions Expression interpolation in Claude prompts#5520
MarkusNeusinger merged 1 commit intomainfrom
fix/workflow-prompt-interpolation

Conversation

@MarkusNeusinger
Copy link
Copy Markdown
Owner

Summary

Three workflows (impl-review.yml, spec-create.yml, report-validate.yml) used shell-style $VAR inside with: prompt: | blocks of claude-code-action. That block is a YAML string handed to a Node/Bun action — no shell ever runs, so $VAR was sent to Claude as a literal placeholder instead of the actual value. Result: Claude couldn't reliably identify the PR / spec / library to review and silently produced no quality_score.txt, which the validate step turns into ai-review-failed.

Symptoms observed today (2026-04-29)

5 stuck implementation PRs from 2026-04-27, all with ai-review-failed despite the prior fixes branch (#5410) and the audit branch (#5515) landing in between:

PR Branch Pre-fix labels
#5476 seaborn/marimekko-basic ai-review-failed, quality:78
#5480 altair/marimekko-basic ai-review-failed, quality:82
#5481 letsplot/marimekko-basic ai-rejected, quality:76
#5483 plotnine/marimekko-basic ai-review-failed
#5486 plotly/line-basic ai-review-failed

Re-dispatching review on each confirmed the bug: the run log of Run AI Quality Review shows the prompt being passed verbatim:

PROMPT: Read prompts/workflow-prompts/ai-quality-review.md and follow those instructions.

Variables for this run:
- LIBRARY: $LIBRARY    # ← literal, never expanded
- SPEC_ID: $SPEC_ID
- PR_NUMBER: $PR_NUMBER
- ATTEMPT: $ATTEMPT

Claude's review then either ran for ~20s and exited with no quality_score.txt (4 PRs failed), or recovered by inferring values from cwd (1 PR succeeded with quality:82). The intermittent pattern is exactly what you'd expect from "the prompt is ambiguous and Claude has to guess from context."

Root cause

Commit 252977cf3 ("chore: fix critical audit findings", 2026-04-28 22:46) routed several ${{ github.event.* }} and step-output values through step-level env: and rewrote the in-prompt references as $VAR. That is the correct mitigation for run: shell steps and Python heredocs in the same workflows (and those changes stay in place). Inside with: prompt: | it is the wrong tool: the value is consumed by a JS action, not a shell, so there is no injection surface to mitigate and $VAR does not interpolate.

spec-create.yml and report-validate.yml carry the identical anti-pattern in their prompt: blocks. They haven't surfaced as failures yet only because no triggering issue has come in since 2026-04-28.

The fix

Revert only the descriptive header lines of each prompt: block back to GitHub Actions Expression syntax (${{ ... }}), which the runner substitutes into the YAML string before the action receives it. Keep:

  • All env: blocks (harmless; lets future prompt content reference env vars if useful)
  • All $VAR references inside embedded bash code samples in the prompt (e.g. gh issue edit $ISSUE_NUMBER). Those are executed by Claude's Bash tool which inherits the step env: and expands them correctly — and rewriting them would re-enable the injection vector the audit was right to close.
             Variables for this run:
-            - LIBRARY: $LIBRARY
-            - SPEC_ID: $SPEC_ID
-            - PR_NUMBER: $PR_NUMBER
-            - ATTEMPT: $ATTEMPT
+            - LIBRARY: ${{ steps.pr.outputs.library }}
+            - SPEC_ID: ${{ steps.pr.outputs.specification_id }}
+            - PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
+            - ATTEMPT: ${{ steps.attempts.outputs.display }}

(analogous 8-line revert in spec-create.yml × 2 prompt blocks and 4-line revert in report-validate.yml).

Diff total: 3 files, 16 ±.

Test plan

  • After merge, redispatch impl-review.yml for the 4 stuck PRs (gh workflow run impl-review.yml -f pr_number=<N> for 5476, 5483, 5486; 5480 already got a 82 in the redispatch and should now stabilize)
  • Verify each run's Run AI Quality Review step log shows real values (e.g. - LIBRARY: plotly) in the PROMPT echo, not $LIBRARY
  • Verify quality_score.txt is produced and ai-review-failed label is removed
  • On next spec-request-labeled issue, verify the spec-create prompt sees the issue title/body
  • On next report-pending-labeled issue, verify the report-validate prompt sees the issue title/body

🤖 Generated with Claude Code

…ompts

Three workflows passed runtime values to claude-code-action via shell-style
`$VAR` placeholders inside the `with: prompt: |` block. That block is a YAML
input to a JavaScript Action — there is no shell, so `$VAR` is never
expanded. Claude received literal `$LIBRARY` / `$ISSUE_TITLE` etc. instead
of the actual values, and silently failed (no `quality_score.txt` produced;
review step exited "success" but validate step then exit 1 with
`ai-review-failed`).

Symptom seen on `main`:
- 4 of 5 stuck PRs (#5476, #5483, #5486, ~#5480 sometimes) hit
  `ai-review-failed` repeatedly. Run logs show review step running ~20s
  with literal `$LIBRARY` in the PROMPT echo. 5480 occasionally
  recovered because Claude inferred values from cwd, hence the
  intermittent pattern.
- spec-create / report-validate not yet observed failing only because no
  triggering issue has come in since 2026-04-28; both would fail the same
  way.

Root cause: 252977c ("chore: fix critical audit findings") attempted to
mitigate workflow injection by routing `${{ github.event.* }}` through
`env:` and dereferencing as `$VAR`. That is the correct mitigation for
shell `run:` steps and Python heredocs (and those changes are kept).
Inside `with: prompt: |` it is wrong: the value is a YAML string handed to
a Node/Bun action, not executed by a shell, so there is no injection
surface and `$VAR` does not interpolate.

Fix: revert *only* the descriptive header lines of each `prompt:` block to
use `${{ ... }}` Actions Expressions, which the runner substitutes into the
prompt string before the action receives it. The shell-style `$VAR`
references inside embedded bash code samples (e.g. `gh issue edit
$ISSUE_NUMBER`) are left as-is — they are run by Claude's Bash tool which
inherits the step `env:` and expands them correctly. The `env:` blocks
themselves are also left in place; they are harmless and keep both call
paths working.

Files:
- impl-review.yml: 4 lines (LIBRARY / SPEC_ID / PR_NUMBER / ATTEMPT header)
- spec-create.yml: 8 lines (issue header in initial + retry prompts)
- report-validate.yml: 4 lines (issue header)
Copilot AI review requested due to automatic review settings April 29, 2026 10:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Restores correct GitHub Actions expression interpolation inside claude-code-action with: prompt: | blocks so runtime context (issue/PR/spec/library) is actually passed to Claude, avoiding ambiguous prompts that can lead to missing quality_score.txt outputs and downstream ai-review-failed results.

Changes:

  • Update spec-create.yml Claude prompts to use ${{ github.event.issue.* }} instead of literal $ISSUE_* placeholders.
  • Update report-validate.yml Claude prompt to use ${{ github.event.issue.* }} instead of literal $ISSUE_* placeholders.
  • Update impl-review.yml Claude prompt “Variables for this run” header to use step outputs (${{ steps.*.outputs.* }}) instead of literal $LIBRARY/$SPEC_ID/... placeholders.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
.github/workflows/spec-create.yml Fix prompt variable interpolation for issue details in both primary + retry Claude steps.
.github/workflows/report-validate.yml Fix prompt variable interpolation for issue details in the Claude validation step.
.github/workflows/impl-review.yml Fix prompt variable interpolation for library/spec/pr/attempt in the AI quality review step.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 29, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@MarkusNeusinger MarkusNeusinger merged commit 338fc61 into main Apr 29, 2026
13 checks passed
@MarkusNeusinger MarkusNeusinger deleted the fix/workflow-prompt-interpolation branch April 29, 2026 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants