[detection-analysis] Detection Analysis Report — 2026-08-03 #50116
Closed
Replies: 1 comment
|
This discussion has been marked as outdated by Detection Analysis Report. A newer discussion is available at Discussion #50421. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
2026-08-03T11:50:47Z→2026-08-03T23:21:28Z(~11.5h); no earlier runs were returned by the log tool for the requested 24h range, so the gap likely reflects a quiet period rather than missing data.Warning
6 workflows were flagged as potentially misconfigured with respect to
gh-aw-detection. See details below.Comparison Chart
Misconfigured Workflows
Daily Regulatory Report Generatorgh-aw-detection: trueis not setgh-aw-detection: trueif this workflow performs security/quality detection, or rename to avoid implying a detection roleMCP Inspector Agentgh-aw-detection: trueis not setgh-aw-detection: trueor renameDaily Security Observability Reportgh-aw-detection: trueis not setgh-aw-detection: trueor renameDeep Reportgh-aw-detection: trueis not setgh-aw-detection: trueor renameSuper Linter Reportgh-aw-detection: trueis not setgh-aw-detection: trueor renameSmoke Copilot Sub Agentsgh-aw-detectionexplicitly set tofalseon a recurring scheduled workflow (run#34, daily cadence implied by run-number progression)gh-aw-detection: trueif it generates/executes untrusted content, or document the opt-outNote on methodology: Rule 1 ("
gh-aw-detection: falseon a workflow with >3 runs in the last 7 days") was evaluated using the 24h window's run cadence (schedule trigger + run-number progression) as a proxy, since a full 7-day artifact download was impractical within this run's time budget — onlySmoke Copilot Sub Agentshad an explicitfalsevalue in this window. Rule 3 ("detection-related steps failed") could not be evaluated precisely: only theusageartifact was downloaded per run, which doesn't expose step-level job logs, so it's not possible to distinguish a failed detection step from a general job failure. 16 detection-enabled runs failed overall in this window (see full table below) but none could be confirmed as detection-step-specific failures.Daily Agentic Workflow AIC Usage Auditwas excluded from the name-based rule per its documented repository-level opt-out (mirrored fromgithubnext/agentic-ops).View All Run Metrics
View Historical Trend
Success rates for both groups have held in the 80–95% band over the past month, with an intermittent shared dip around Jul 09–10 (likely a broader infra/API incident affecting all engines rather than a detection-specific issue). Misconfigured-workflow counts have trended down from a Jul 24 peak of 14 to single digits recently, though today's count (6) ticked back up from 1 the prior two days — driven mostly by the report/inspector-named workflows above.
Recommendations
gh-aw-detection: true(or rename) to the five report/inspector-named workflows flagged above — their names imply a detection/monitoring role that isn't reflected in configuration.Smoke Copilot Sub Agents' explicitgh-aw-detection: false— if it's a deliberate opt-out, consider documenting it the same wayDaily Agentic Workflow AIC Usage Auditis documented, so future audits don't re-flag it.PR Code Quality Reviewerhad the lowest success rate in this window (57.9% over 19 runs) — worth a separate reliability investigation independent of detection status.usageartifacts) for detection-enabled runs, so misconfiguration rule 3 (failed detection steps) can be verified directly rather than approximated.All reactions