threat-detection: enable and upload the structured JSONL run log on the external path

[davidslater]

Created by GitHub Ace · View Session

Summary

threat-detect emits a structured JSONL run log via --log-file (or THREAT_DETECTION_LOG_FILE): run_start, artifacts_loaded, prompt_built, attempt_start, attempt_no_verdict, attempt_recorded, verdict, detection_failed, status, plus engine invocation metadata. In v0.4.0 conclude additionally mirrors conclude_start, conclude_verdict, conclude_directory_listing, conclude_detection_log, and conclude_outcome.

gh-aw never enables it, so the best triage artifact for detection failures is never produced.

Current code

pkg/workflow/threat_detection_external.go:354-360:

threatDetectCmd := fmt.Sprintf(
  "%s && threat-detect --engine %s --output %s %s",
  npmPathSetup, engineID,
  shellEscapeArg(constants.ThreatDetectionResultPath),
  shellEscapeArg(constants.ThreatDetectionDir),
)

buildUploadDetectionArtifactStep (lines 447-460) uploads only detection_result.json and detection.log.

Impact

attempt_no_verdict — the event that explains why a run ended without a verdict, which is the single most common confusing failure mode — exists only in the JSONL. Without it, post-hoc analysis is limited to scraping the free-form detection.log tee, which interleaves engine transcript, AWF lifecycle noise, and detector stderr.

Proposed change

threatDetectCmd := fmt.Sprintf(
  "%s && threat-detect --engine %s --log-file %s --output %s %s",
  npmPathSetup, engineID,
  shellEscapeArg(constants.ThreatDetectionRunlogPath),
  shellEscapeArg(constants.ThreatDetectionResultPath),
  shellEscapeArg(constants.ThreatDetectionDir),
)

Add a ThreatDetectionRunlogPath constant alongside the existing ThreatDetection* constants in pkg/constants/constants.go, pointing at /tmp/gh-aw/threat-detection/detection-runlog.jsonl, and add it to buildUploadDetectionArtifactStep.

Two hard constraints:

1. The path must be inside constants.ThreatDetectionDir so it lands in the AWF :rw mount and survives to the host. That mount already exists (appendThreatDetectionRWMount).
2. --log-file and --output must not resolve to the same file — the binary rejects that explicitly, so the distinct filename matters.

Note: upstream #746 defaults the runlog to that same path whenever --output is set, so this may already work implicitly on v0.4.0. Passing it explicitly is still preferable so the path is visible in the compiled lock file rather than depending on an implicit binary default.

Optionally, once conclude delegation lands (#50646), also pass --log-file "${RESULT_DIR}/conclude-runlog.jsonl" from conclude_threat_detection.sh and add that path to the upload list, giving JSONL parity with the inline path for the conclusion itself.

gh aw audit support

Verify gh aw audit recognizes and surfaces the JSONL after artifact download. If it currently only inventories the file, add ingestion/reporting so the runlog is queryable from the audit output rather than requiring a manual artifact download. This is the part that makes the artifact actually useful during triage.

Acceptance criteria

• --log-file passed explicitly to a path inside ThreatDetectionDir, distinct from --output.
• Shared path constant defined in pkg/constants.
• detection-runlog.jsonl (and conclude-runlog.jsonl if added) included in the upload step's path: list.
• Compiler coverage, notably pkg/workflow/threat_detection_isolation_test.go.
• gh aw audit ingests/surfaces the runlog.
• make recompile.

References

• Parity: the structured JSONL run log (--log-file) is never enabled or uploaded on the external detector path gh-aw-threat-detection#697
• Parity: threat-detect conclude emits far less diagnostic output than parse_threat_detection_results.cjs gh-aw-threat-detection#692
• Upstream draft (for github/gh-aw): compiler-side changes needed for external threat-detect parity gh-aw-threat-detection#726 (item C)

[github-actions]

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Assigning to @lpcox — pelikhan already has 2 issues this run; lpcox is the next most recent active contributor.

Generated by 🔧 Daily Assign Issue To User · auto · 56.7 AIC · ⊞ 5.6K · ◷