[evals] Daily Evals Feature Report - 2026-08-06

[github-actions]

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Executive Summary

Across the last 7 full UTC days, 29 workflows produced evals results in 48 runs; question-level analysis was capped to the most recent 40 runs. The evals pipeline itself is reliable with a 100.0% artifact success rate and no missing-result or push_evals_state failures.

The feature is still degraded as a quality signal: the overall YES rate was 54.4%, only 12/29 workflows achieved a 100% run pass rate in the analyzed set, and 28 answers were emitted as UNKNOWN instead of the documented binary YES/NO. For binary-rate calculations below, UNKNOWN was treated as a non-passing outcome.

Note

Status: DEGRADED — evals jobs are producing artifacts, but the analyzed question signal is degraded because the overall YES rate is 54.4% and 28 answers were UNKNOWN.

Note

Analysis was capped to the most recent 40 of 48 evals-result runs to stay within the 40-run limit; failure counting still covered the full fetched set.

Key Metrics

Metric	Value
Workflows with evals	29
Runs analyzed	40
Runs with evals results	40
Evals job success rate	100.0%
Overall YES rate	54.4%

Per-Workflow Pass Rates

Workflow	Runs	Evals Job Success	Run Pass Rate	Lowest-Scoring Question
AI Moderator	2	100.0%	0.0%	action-taken: Did the agent apply at least one label (spam, ai-generated, link-spam, or ai-inspected) or call noop? (0.0% YES)
Auto-Triage Issues	1	100.0%	0.0%	human-bug-labeled: Did the agent label a non-report issue describing unexpected behavior, expected versus actual results, or a reproducible failure with bug and the relevant component label (or needs-triage if no component is clear)? (0.0% YES)
Code Scanning Fixer	1	100.0%	0.0%	pr_created_or_noop: Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed? (0.0% YES)
Daily AgentRx Trace Optimizer	1	100.0%	0.0%	sub_agent_strategy_goal_met: Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed? (0.0% YES)
Daily Assign Issue To User	1	100.0%	0.0%	comment-posted: Did the agent post a comment explaining the assignment decision? (0.0% YES)
Daily Elixir Credo Snippet Audit	1	100.0%	0.0%	elixir_snippets_linted: Did the agent use Credo to lint Elixir snippets found in repository markdown files? (0.0% YES)
Daily VulnHunter Scan	1	100.0%	0.0%	issue_created_or_noop: Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable? (0.0% YES)
Daily Workflow Updater	1	100.0%	0.0%	update_pr_created_or_noop: Did the agent create a pull request for required updates, or report that no changes were needed? (0.0% YES)
Daily action/setup/* Security Audit	2	100.0%	0.0%	issue_or_noop: Was at least one security issue created with actionable recommendations, or was noop called when no new findings were found? (0.0% YES)
Design Decision Gate 🏗️	3	100.0%	0.0%	adr-check-performed: Does the agent output confirm that it checked for existing ADRs before deciding on an action? (0.0% YES)
Discussion Task Miner - Code Quality Improvement Agent	1	100.0%	0.0%	labels-applied: Does the agent output confirm that the created issues include the expected labels (code-quality, automation, task-mining)? (0.0% YES)
Issue Arborist	1	100.0%	0.0%	issues_analyzed: Did the agent analyze recent issues and identify related issue relationships? (0.0% YES)
Issue Monster	2	100.0%	0.0%	single_issue_scoped: Does the agent output show that at most one issue was assigned to Copilot per run? (0.0% YES)
PR Code Quality Reviewer	4	100.0%	0.0%	findings_scoped: Does the agent output show that the review findings are limited to changes in the pull request diff rather than unrelated code? (0.0% YES)
PR Triage Agent	1	100.0%	0.0%	report-produced: Does the agent output include a triage report summarizing the PRs processed? (0.0% YES)
Release	1	100.0%	0.0%	community-attribution: Does the agent output include attribution to community contributors in the release highlights? (0.0% YES)
Test Quality Sentinel	3	100.0%	0.0%	model_size_goal_met: Does the agent output show that the objective for experiment model_size was successfully completed? (0.0% YES)
CLI Version Checker	1	100.0%	100.0%	cli_versions_checked: Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)? (100.0% YES)
Daily Cli Tools Tester	1	100.0%	100.0%	cli_tools_tested: Did the agent run exploratory tests on the audit, logs, and compile CLI tools? (100.0% YES)
Daily Container Image Security Scan	1	100.0%	100.0%	container_images_scanned: Did the agent analyze container images for vulnerabilities, updates, and rejected licenses? (100.0% YES)
Daily Go Test Parallelizer	1	100.0%	100.0%	pr_created_or_noop: Did the agent create a pull request for a safe test change, or use noop when no safe change was available? (100.0% YES)
Daily Safe Outputs Conformance Checker	1	100.0%	100.0%	conformance_checked: Did the agent run a conformance check against the Safe Outputs specification implementation? (100.0% YES)
ESLint Refiner	1	100.0%	100.0%	eslint_trends_analyzed: Did the agent analyze ESLint diagnostics trends to identify rule refinement opportunities? (100.0% YES)
Multi-Device Docs Tester	1	100.0%	100.0%	device_tests_completed: Did the agent test the documentation site across the requested device form factors? (100.0% YES)
PR Sous Chef	2	100.0%	100.0%	comment-added: Did the agent add a comment to at least one pull request? (100.0% YES)
Smoke Copilot	1	100.0%	100.0%	build-succeeded: Did the gh-aw binary build succeed? Look for a successful make build step or explicit mention that compilation succeeded. (100.0% YES)
Sub-Issue Closer	1	100.0%	100.0%	issues_checked: Did the agent check parent issues for the completion status of all their sub-issues? (100.0% YES)
Tidy	1	100.0%	100.0%	pr_created_or_noop: Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed? (100.0% YES)
[aw] Failure Investigator (6h)	1	100.0%	100.0%	failures_investigated: Did the agent investigate agentic workflow failures from the last 6 hours and produce findings? (100.0% YES)

Per-Question Breakdown per Workflow

NO counts below include UNKNOWN answers as non-passing outcomes to preserve binary YES/NO rates; raw UNKNOWN emissions totaled 28.

AI Moderator

Question ID	Question	YES	NO	YES Rate
action-taken	Did the agent apply at least one label (spam, ai-generated, link-spam, or ai-inspected) or call noop?	0	2	0.0%
no-unsupported-action	Does the agent output show that only allowed safe-output actions (add-labels, hide-comment, noop) were used?	0	2	0.0%
rationale-provided	Does the agent output include a rationale explaining why the label(s) were applied or why noop was called?	0	2	0.0%

Auto-Triage Issues

Question ID	Question	YES	NO	YES Rate
human-bug-labeled	Did the agent label a non-report issue describing unexpected behavior, expected versus actual results, or a reproducible failure with bug and the relevant component label (or needs-triage if no component is clear)?	0	1	0.0%
labels-applied	Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?	1	0	100.0%
report-created	Was a summary discussion created listing the issues processed and the labels applied?	0	1	0.0%

Code Scanning Fixer

Question ID	Question	YES	NO	YES Rate
alerts_analyzed	Did the agent analyze code scanning alerts and identify at least one fixable alert, or correctly skip when no fixable alerts were found?	1	0	100.0%
pr_created_or_noop	Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?	0	1	0.0%

Daily AgentRx Trace Optimizer

Question ID	Question	YES	NO	YES Rate
sub_agent_strategy_goal_met	Does the agent output show that the objective for experiment sub_agent_strategy was successfully completed?	0	1	0.0%

Daily Assign Issue To User

Question ID	Question	YES	NO	YES Rate
assignment-made	Did the agent assign one or more unassigned issues to a user, or correctly call noop when no unassigned issues were found?	1	0	100.0%
comment-posted	Did the agent post a comment explaining the assignment decision?	0	1	0.0%
contributor-selected	Does the agent output show that a relevant contributor was selected from recent merged PRs?	0	1	0.0%

Daily Elixir Credo Snippet Audit

Question ID	Question	YES	NO	YES Rate
elixir_snippets_linted	Did the agent use Credo to lint Elixir snippets found in repository markdown files?	0	1	0.0%
fixes_proposed_or_noop	Were fixes proposed in a PR or issue, or was noop used when no linting violations were found?	1	0	100.0%

Daily VulnHunter Scan

Question ID	Question	YES	NO	YES Rate
issue_created_or_noop	Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?	0	1	0.0%
scan_completed	Did the agent download the prepared VulnHunter bundle artifact, load its vulnhunt skill instructions, and complete a repository scan?	0	1	0.0%

Daily Workflow Updater

Question ID	Question	YES	NO	YES Rate
action_versions_checked	Did the agent check GitHub Actions versions for available updates?	1	0	100.0%
update_pr_created_or_noop	Did the agent create a pull request for required updates, or report that no changes were needed?	0	1	0.0%

Daily action/setup/* Security Audit

Question ID	Question	YES	NO	YES Rate
issue_or_noop	Was at least one security issue created with actionable recommendations, or was noop called when no new findings were found?	0	2	0.0%
security_aspect_rotated	Did the agent load and update the cache-memory state to rotate the security focus aspect using the 60/40 rule?	0	2	0.0%

Design Decision Gate 🏗️

Question ID	Question	YES	NO	YES Rate
action-taken	Did the agent add a PR comment, push a draft ADR, or call noop?	3	0	100.0%
adr-check-performed	Does the agent output confirm that it checked for existing ADRs before deciding on an action?	0	3	0.0%
decision-justified	Does the agent output explain why an ADR is required or why no ADR gate was triggered for this PR?	3	0	100.0%

Discussion Task Miner - Code Quality Improvement Agent

Question ID	Question	YES	NO	YES Rate
labels-applied	Does the agent output confirm that the created issues include the expected labels (code-quality, automation, task-mining)?	0	1	0.0%
output-produced	Did the agent create at least one code quality issue or add a comment?	1	0	100.0%
tasks-extracted	Does the agent output show that actionable tasks were identified from the analyzed discussions?	0	1	0.0%

Issue Arborist

Question ID	Question	YES	NO	YES Rate
issues_analyzed	Did the agent analyze recent issues and identify related issue relationships?	0	1	0.0%
sub_issues_linked_or_noop	Were related issues linked as sub-issues, or was noop used when no linkable relationships were found?	0	1	0.0%

Issue Monster

Question ID	Question	YES	NO	YES Rate
issue_assigned	Did the agent assign at least one issue to the Copilot coding agent, or correctly skip when no suitable issues were found?	2	0	100.0%
single_issue_scoped	Does the agent output show that at most one issue was assigned to Copilot per run?	0	2	0.0%

PR Code Quality Reviewer

Question ID	Question	YES	NO	YES Rate
findings_scoped	Does the agent output show that the review findings are limited to changes in the pull request diff rather than unrelated code?	0	4	0.0%
review_posted	Did the agent post a code review comment on the pull request?	2	2	50.0%

PR Triage Agent

Question ID	Question	YES	NO	YES Rate
labels-applied	Did the agent apply triage labels to at least one pull request?	1	0	100.0%
report-produced	Does the agent output include a triage report summarizing the PRs processed?	0	1	0.0%
triage-data-set	Does the agent output confirm that category, risk, and action data were determined for each processed PR?	0	1	0.0%

Release

Question ID	Question	YES	NO	YES Rate
community-attribution	Does the agent output include attribution to community contributors in the release highlights?	0	1	0.0%
highlights-included	Does the agent output confirm that release highlights were generated and prepended to the release notes?	1	0	100.0%
release-updated	Did the agent update the release with generated highlights?	1	0	100.0%

Test Quality Sentinel

Question ID	Question	YES	NO	YES Rate
model_size_goal_met	Does the agent output show that the objective for experiment model_size was successfully completed?	0	3	0.0%

CLI Version Checker

Question ID	Question	YES	NO	YES Rate
cli_versions_checked	Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?	1	0	100.0%
docker_images_checked	Did the agent check for new versions and digest changes of Docker images in pkg/cli/docker_images.go (actionlint, syft, grype, grant, zizmor, poutine, runner-guard, yamllint)?	1	0	100.0%
updates_applied_or_noop	Were version or digest updates applied and a PR created, or was noop used when all tools were already up to date?	1	0	100.0%

Daily Cli Tools Tester

Question ID	Question	YES	NO	YES Rate
cli_tools_tested	Did the agent run exploratory tests on the audit, logs, and compile CLI tools?	1	0	100.0%
test_results_reported	Were the test results reported with any issues or anomalies identified?	1	0	100.0%

Daily Container Image Security Scan

Question ID	Question	YES	NO	YES Rate
container_images_scanned	Did the agent analyze container images for vulnerabilities, updates, and rejected licenses?	1	0	100.0%
findings_reported_or_noop	Did the agent report actionable image findings, or use noop when no findings required action?	1	0	100.0%

Daily Go Test Parallelizer

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Did the agent create a pull request for a safe test change, or use noop when no safe change was available?	1	0	100.0%
tests_analyzed	Did the agent analyze Go tests to identify a safe candidate for t.Parallel?	1	0	100.0%

Daily Safe Outputs Conformance Checker

Question ID	Question	YES	NO	YES Rate
conformance_checked	Did the agent run a conformance check against the Safe Outputs specification implementation?	1	0	100.0%
tasks_created_or_noop	Were agentic tasks created for Copilot to address critical/high/medium/low issues, or was noop used when implementation was compliant?	1	0	100.0%

ESLint Refiner

Question ID	Question	YES	NO	YES Rate
eslint_trends_analyzed	Did the agent analyze ESLint diagnostics trends to identify rule refinement opportunities?	1	0	100.0%
refinements_reported	Did the agent report actionable ESLint rule refinements or explain why no refinement was needed?	1	0	100.0%

Multi-Device Docs Tester

Question ID	Question	YES	NO	YES Rate
device_tests_completed	Did the agent test the documentation site across the requested device form factors?	1	0	100.0%
results_reported	Did the agent report the multi-device test results and any responsive design or functionality findings?	1	0	100.0%

PR Sous Chef

Question ID	Question	YES	NO	YES Rate
comment-added	Did the agent add a comment to at least one pull request?	2	0	100.0%
nudge-targeted	Does the agent output show a specific reason why the selected PR needs a nudge toward maintainer investigation?	2	0	100.0%
pr-evaluated	Does the agent output confirm that it evaluated at least one open PR for nudge eligibility?	2	0	100.0%

Smoke Copilot

Question ID	Question	YES	NO	YES Rate
build-succeeded	Did the gh-aw binary build succeed? Look for a successful make build step or explicit mention that compilation succeeded.	1	0	100.0%
issue-created	Was a smoke test issue created with test results? Look for a create_issue output containing 'Smoke Test' in the title.	1	0	100.0%
smoke-passed	Did all or most smoke tests pass? Look for an overall PASS status or the majority of tests showing ✅ in the agent output.	1	0	100.0%

Sub-Issue Closer

Question ID	Question	YES	NO	YES Rate
issues_checked	Did the agent check parent issues for the completion status of all their sub-issues?	1	0	100.0%
issues_closed_or_noop	Were completed parent issues closed with a comment, or does the agent output confirm no issues were ready to close?	1	0	100.0%

Tidy

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?	1	0	100.0%
tidy_completed	Did the agent run code formatting and tidying tools on the codebase?	1	0	100.0%

[aw] Failure Investigator (6h)

Question ID	Question	YES	NO	YES Rate
failures_investigated	Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?	1	0	100.0%
issues_created_or_closed	Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed?	1	0	100.0%

Quality Signals

The most failing question in the analyzed window was findings_scoped in PR Code Quality Reviewer, with 0.0% YES across 4 scored answers; all 4 non-YES outcomes were UNKNOWN.

adr-check-performed in Design Decision Gate 🏗️ also scored 0.0% YES, but here the failures were explicit NO answers rather than schema drift. model_size_goal_met showed the same 0% YES pattern in Test Quality Sentinel, again driven by UNKNOWN outputs instead of binary results.

In total, 17 workflows had sub-100% run pass rates in the analyzed sample. The degradation clusters into two modes: explicit evaluation misses in prompt-specific checks such as ADR verification, issue scoping, and release attribution; and evaluator contract drift where workflows emitted UNKNOWN for questions that the feature specification says should be binary.

Recommendations

• Investigate why the evals engine emitted UNKNOWN in 28 records across 12 workflows, starting with AI Moderator, PR Code Quality Reviewer, and Test Quality Sentinel; these responses are the main reason the overall YES rate fell into the degraded band.
• Tighten workflow prompts or evaluator rubrics for the fully explicit failing checks: adr-check-performed in Design Decision Gate 🏗️, single_issue_scoped in Issue Monster, and community-attribution in Release.
• Add a validation or lint step for eval outputs that rejects non-binary answers before publishing evals.jsonl, so schema drift is surfaced as an evals-job failure instead of silently degrading downstream reporting.

References

§31079102370 AI Moderator
§31076026748 PR Code Quality Reviewer
§31076026740 Design Decision Gate 🏗️

Generated by 🧪 Daily Evals Feature Report · gpt54 · 82.4 AIC · ⊞ 11.9K · ◷

• expires on Aug 13, 2026, 12:10 AM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #51036

View newer issue

---

This action was performed automatically by the Daily Evals Feature Report workflow.