Repository navigation
[detection-analysis] Detection Analysis Report — last 24h (2026-11-02) #74254
Closed
Replies: 1 comment
|
This discussion has been marked as outdated by Detection Analysis Report. A newer discussion is available at Discussion #74665. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
gh-aw-detection: false): 74 (11.5%)Warning
All 18 misconfigured workflows are CI "smoke test" fixtures that explicitly set
gh-aw-detection: falsein frontmatter (not accidental). No detection-job-specific failures or inconsistent per-workflow settings were found in this window.Coverage note: The pre-downloaded log collection hit its storage/count budget partway through. The compact
summary.jsonsnapshot only covered 2026-11-02T07:16–23:12 (405 runs); I used theagenticworkflows logsMCP tool with the suppliedcontinuationcursor to fetch the remaining 2026-11-01T23:16–07:07 window and confirmed it contains only 8 additional runs, all non-executing (cancelled/obsolete PR runs), so no detection-relevant data was missed there. Separately, the live log cache grew to 837 run directories during this analysis, but only 646 had bothaw_info.jsonandrun_summary.jsonfully written at analysis time; the remaining 191 (~23%) were still downloading and are not included in the metrics below.Comparison Chart
Detection-enabled runs show a notably lower success rate (70.63% vs 89.19%) and higher average token usage. However, per-run job attribution shows zero runs where the
detectionjob itself failed (555 succeeded, 65 skipped, 24 absent/not-run) — the lower success rate among detection-enabled runs is driven entirely byagentjob failures (e.g. thesafe_outputs/agent-logic failure clusters seen across ~44+ runs in this window), not by the detection feature itself.Misconfigured Workflows
gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.gh-aw-detection: falsefrom frontmatter unless there is a documented reason to opt out of detection; re-enable detection for prompt-injection / safety coverage.All flagged entries are CI smoke-test workflows (
.github/workflows/smoke-*.md) that setgh-aw-detection: falsedirectly in frontmatter — this appears to be a deliberate, repo-wide convention for engine/runtime smoke tests rather than an oversight, since the same pattern is consistent across ~18 distinct smoke workflows. Flagged per the audit's literal threshold (>3 runs/window with detection explicitly disabled); worth a one-time confirmation with the workflow owners that this opt-out is still intentional, rather than treating each as an individual bug.View All Run Metrics
Full per-workflow breakdown for the 646-run analyzed window (198 workflows):
View Historical Trend
34 historical daily data points exist in cache-memory (2026-08-25 → 2026-11-02; 22 of the last 30 days have data). Detection-run volume and regular-run volume both show high day-to-day variance; detection success rate has ranged roughly 20–95% over the period, consistent with it tracking overall agent-job reliability rather than a detection-specific regression.
Recommendations
gh-aw-detection: falsebaked into frontmatter. If intentional, consider documenting this convention (e.g. a comment in each smoke workflow or a shared template) so future audits don't need to re-flag it; if not intentional, re-enable detection viagh aw update/frontmatter edit per workflow.detectionjob in this window — the feature itself is operating correctly where enabled.gh-aw-detectionand is already covered by the existing failure-cluster investigation (safe_outputsstep failures across 44 runs, plusagent_logicfailures).References:
gh-aw-detection: falserungh-aw-detection: falserunAll reactions