[WIP] Enable and upload structured JSONL run log on external path

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>threat-detection: enable and upload the structured JSONL run log on the external path</issue_title>
<issue_description>> Created by GitHub Ace · View Session

Summary

threat-detect emits a structured JSONL run log via --log-file (or THREAT_DETECTION_LOG_FILE): run_start, artifacts_loaded, prompt_built, attempt_start, attempt_no_verdict, attempt_recorded, verdict, detection_failed, status, plus engine invocation metadata. In v0.4.0 conclude additionally mirrors conclude_start, conclude_verdict, conclude_directory_listing, conclude_detection_log, and conclude_outcome.

gh-aw never enables it, so the best triage artifact for detection failures is never produced.

Current code

pkg/workflow/threat_detection_external.go:354-360:

threatDetectCmd := fmt.Sprintf(
  "%s && threat-detect --engine %s --output %s %s",
  npmPathSetup, engineID,
  shellEscapeArg(constants.ThreatDetectionResultPath),
  shellEscapeArg(constants.ThreatDetectionDir),
)

buildUploadDetectionArtifactStep (lines 447-460) uploads only detection_result.json and detection.log.

Impact

attempt_no_verdict — the event that explains why a run ended without a verdict, which is the single most common confusing failure mode — exists only in the JSONL. Without it, post-hoc analysis is limited to scraping the free-form detection.log tee, which interleaves engine transcript, AWF lifecycle noise, and detector stderr.

Proposed change

threatDetectCmd := fmt.Sprintf(
  "%s && threat-detect --engine %s --log-file %s --output %s %s",
  npmPathSetup, engineID,
  shellEscapeArg(constants.ThreatDetectionRunlogPath),
  shellEscapeArg(constants.ThreatDetectionResultPath),
  shellEscapeArg(constants.ThreatDetectionDir),
)

Add a ThreatDetectionRunlogPath constant alongside the existing ThreatDetection* constants in pkg/constants/constants.go, pointing at /tmp/gh-aw/threat-detection/detection-runlog.jsonl, and add it to buildUploadDetectionArtifactStep.

Two hard constraints:

1. The path must be inside constants.ThreatDetectionDir so it lands in the AWF :rw mount and survives to the host. That mount already exists (appendThreatDetectionRWMount).
2. --log-file and --output must not resolve to the same file — the binary rejects that explicitly, so the distinct filename matters.

Note: upstream #746 defaults the runlog to that same path whenever --output is set, so this may already work implicitly on v0.4.0. Passing it explicitly is still preferable so the path is visible in the compiled lock file rather than depending on an implicit binary default.

Optionally, once conclude delegation lands (#50646), also pass --log-file "${RESULT_DIR}/conclude-runlog.jsonl" from conclude_threat_detection.sh and add that path to the upload list, giving JSONL parity with the inline path for the conclusion itself.

gh aw audit support

Verify gh aw audit recognizes and surfaces the JSONL after artifact download. If it currently only inventories the file, add ingestion/reporting so the runlog is queryable from the audit output rather than requiring a manual artifact download. This is the part that makes the artifact actually useful during triage.

Acceptance criteria

• --log-file passed explicitly to a path inside ThreatDetectionDir, distinct from --output.
• Shared path constant defined in pkg/constants.
• detection-runlog.jsonl (and conclude-runlog.jsonl if added) included in the upload step's path: list.
• Compiler coverage, notably pkg/workflow/threat_detection_isolation_test.go.
• gh aw audit ingests/surfaces the runlog.
• make recompile.

References

• Parity: the structured JSONL run log (--log-file) is never enabled or uploaded on the external detector path gh-aw-threat-detection#697
• Parity: threat-detect conclude emits far less diagnostic output than parse_threat_detection_results.cjs gh-aw-threat-detection#692
• Upstream draft (for github/gh-aw): compiler-side changes needed for external threat-detect parity gh-aw-threat-detection#726 (item C)</issue_description>
• Closes threat-detection: enable and upload the structured JSONL run log on the external path #50651

Comments on the Issue (you are @copilot in this section)

• Fixes threat-detection: enable and upload the structured JSONL run log on the external path #50651