threat-detection: stage prompt-template, prompt-import-tree, aw_info.json and comment-memory into the detection dir

[davidslater]

Created by GitHub Ace · View Session

Summary

buildPrepareDetectionFilesStep (pkg/workflow/threat_detection_steps.go:174-199) stages only four things into /tmp/gh-aw/threat-detection/:

aw-prompts/prompt.txt
agent_output.json
aw-*.patch
aw-*.bundle

The detector documents and consumes several more. Because they are never staged, multiple documented detection features are silently inert in production — they render their degraded-fallback text and produce no warning. This affects detection quality, i.e. it is a security-coverage gap, not a cosmetic one.

Missing inputs

1. aw-prompts/prompt-template.txt + aw-prompts/prompt-import-tree.json — REQUIRED

threat-detect performs a trusted-template vs untrusted-runtime-input breakdown of the workflow prompt and renders a ## Prompt Analysis (Trusted vs Untrusted Content) section that tells the model to concentrate injection analysis on the interpolated regions. This is the headline detection-quality improvement the external detector has over the inline path.

gh-aw already produces and already uploads both files — interpolate_prompt.cjs writes them, and pkg/workflow/compiler_activation_outputs.go:153-154 uploads them in the agent artifact. They are simply not copied into the staged detection directory.

Effect today: every external-detector run renders No prompt template or import tree available. Prompt analysis was not performed. The feature has been a no-op since it shipped.

2. aw_info.json — REQUIRED

Upstream #747 makes threat-detect consume aw_info.json as bounded, untrusted activation context (trigger, event, actor) and inventory all staged artifacts. Without staging, the {ACTIVATION_CONTEXT} prompt section is empty. The prompt's security guidelines instruct the model to use activation context to distinguish legitimate behavior from threats, so its absence should measurably increase false positives.

3. comment-memory/ — investigate, then stage

actions/setup/md/threat_detection.md has a ## Comment Memory Files section, and setup_threat_detection.cjs lists *.md under /tmp/gh-aw/threat-detection/comment-memory/. Neither upstream nor a read of buildPrepareDetectionFilesStep found a step that populates that directory.

If that reading is correct, comment-memory content is going unanalyzed on both the inline and the external path — attacker-influenced persisted content sitting outside the detection surface. Upstream #743 landed the detector-side discovery and {COMMENT_MEMORY_FILES} prompt section; the staging half is ours.

First task is verification against a real run, since this one is a genuine "we may have misread it" item, not a confirmed gap.

4. experiments/ — optional

Documented but never loaded; inventory-only in the detector today. Stage only if it should be inside the detection surface — decide explicitly rather than by omission.

Proposed change

Extend buildPrepareDetectionFilesStep, mirroring the existence-guarded style already used for patches and bundles:

cp /tmp/gh-aw/aw-prompts/prompt-template.txt     /tmp/gh-aw/threat-detection/aw-prompts/prompt-template.txt     2>/dev/null || true
cp /tmp/gh-aw/aw-prompts/prompt-import-tree.json /tmp/gh-aw/threat-detection/aw-prompts/prompt-import-tree.json 2>/dev/null || true
cp /tmp/gh-aw/aw_info.json                       /tmp/gh-aw/threat-detection/aw_info.json                       2>/dev/null || true
# comment-memory/ (pending verification)

All are optional inputs on the detector side — artifacts.Load probes with fileExists and degrades gracefully — so this is safe against both v0.3.0 and v0.4.0 and can land independently of the pin bump. v0.4.0 additionally emits ::warning::ERR_VALIDATION and a prompt_analysis_degraded runlog event when they are absent, so after #50644 the degradation stops being silent either way.

Related caution: GITHUB_STEP_SUMMARY writability under AWF

Upstream #747 writes an artifact-inventory table to GITHUB_STEP_SUMMARY and treats a failed write as a config error (exit 2, spec TD-20c). In the AWF sandbox, if GITHUB_STEP_SUMMARY is set but its path is not on a writable mount, detection would fail closed. Verify the detection job runs with GITHUB_STEP_SUMMARY on a writable mount; if not, either add the mount or ask upstream to soften that to a warning.

Acceptance criteria

• prompt-template.txt, prompt-import-tree.json, and aw_info.json staged into the detection dir.
• Verified against a real run whether comment-memory/ is present; staged if absent.
• Explicit decision recorded on experiments/.
• GITHUB_STEP_SUMMARY writability under AWF confirmed for the detection job.
• Compiler test asserting all prompt artifacts are copied into the external detector input directory.
• End-to-end check that the detector renders a populated ## Prompt Analysis (Trusted vs Untrusted Content) section rather than the degraded fallback.
• make recompile.

References

• Parity: prompt-template.txt / prompt-import-tree.json are never staged, so trusted-vs-untrusted prompt analysis is silently disabled gh-aw-threat-detection#686
• Parity: comment-memory files are not analyzed (missing {COMMENT_MEMORY_FILES} prompt section and artifact discovery) gh-aw-threat-detection#689
• Parity: documented artifacts aw_info.json and experiments/ are never loaded or surfaced gh-aw-threat-detection#704
• Upstream draft (for github/gh-aw): compiler-side changes needed for external threat-detect parity gh-aw-threat-detection#726 (item A)

[github-actions]

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Assigning to @pelikhan — this touches pkg/workflow/threat_detection_steps.go compiler internals, an area they have recently worked on (secret masking, compiler logic).

Generated by 🔧 Daily Assign Issue To User · auto · 56.7 AIC · ⊞ 5.6K · ◷