[observability] Observability Coverage Report - 2026-08-07

[github-actions[bot]]

Executive Summary

Analyzed 20 representative workflow runs from the last 7 days using the repository-wide logs fetch output saved under /tmp/gh-aw/aw-mcp/logs/, then capped the audit to 20 runs per workflow-report instructions. All 20 sampled runs were firewall-enabled and all 20 showed MCP Gateway startup in run_summary.json, but recursive artifact discovery found zero retained access.log files and zero retained gateway.jsonl or rpc-messages.jsonl files.

This is a systemic observability retention gap rather than a single-workflow defect. The saved bundles still contain summary.json in 20/20 runs and agent-stdio.log in 20/20 runs, and those derived artifacts show 1426 firewall requests across 33 unique domains with 106 blocked requests (7.4%). That means execution-time telemetry existed, but the canonical raw logs needed for debugging were not preserved in the run artifacts.

Key Alerts and Anomalies

🔴 Critical Issues:

• AWF Firewall raw logs are missing in all 20/20 sampled firewall-enabled runs. Recursive discovery under each run-<id>/ found no access.log, including the canonical sandbox/firewall/logs/squid-logs/access.log and legacy sandbox/firewall/logs/access.log locations.
• MCP raw telemetry is missing in all 20/20 sampled MCP-enabled runs. Recursive discovery under each run-<id>/ found neither gateway.jsonl nor rpc-messages.jsonl, so there is no canonical per-call trace left for postmortem debugging.

⚠️ Warnings:

• aw_info.json was absent from all sampled bundles; only 2/20 sampled runs retained an awf-config.json file, so configuration provenance is inconsistent even when derived firewall summaries are present.
• safe_output.jsonl was absent from all sampled bundles. Safe-output job steps completed, but the structured safe-output log was not retained as an artifact.
• agent_output.json was present in 19/20 sampled runs, so supplementary telemetry coverage is better than raw firewall/MCP coverage but still not complete.

Coverage Summary

Component	Runs Analyzed	Logs Present	Coverage	Status
AWF Firewall (access.log)	20	0	0.0%	🔴
MCP Gateway (gateway.jsonl or rpc-messages.jsonl)	20	0	0.0%	🔴

📋 Detailed Run Analysis

Firewall-Enabled Runs

Entries/Allowed/Blocked are derived from summary.json because raw access.log was missing in every sampled run.

Workflow	Run ID	access.log	Entries	Allowed	Blocked	Status
Daily BYOK Ollama Test	§31223940604	❌	59	59	0	🔴
Smoke Copilot - AOAI (Entra)	§31222505053	❌	173	138	35	🔴
Smoke Copilot	§31222502383	❌	168	134	34	🔴
Impeccable Skills Reviewer	§31221978069	❌	26	26	0	🔴
Cache directory setup	§31227824424	❌	94	94	0	🔴
PR Sous Chef	§31227649983	❌	47	47	0	🔴
PR Sous Chef	§31226747645	❌	32	32	0	🔴
Detection Analysis Report	§31226672216	❌	138	138	0	🔴
Daily Reliability Review	§31225875315	❌	51	51	0	🔴
Impeccable Skills Reviewer	§31223955510	❌	26	26	0	🔴
Matt Pocock Skills Reviewer	§31223955505	❌	42	42	0	🔴
Design Decision Gate 🏗️	§31223955476	❌	63	63	0	🔴
Test Quality Sentinel	§31223955470	❌	130	130	0	🔴
PR Code Quality Reviewer	§31223955453	❌	6	2	4	🔴
Daily AWF Spec Compiler Surfacing Review	§31223464787	❌	52	52	0	🔴
Issue Monster	§31223293352	❌	22	22	0	🔴
ESLint Monster	§31222548390	❌	10	10	0	🔴
Smoke OTEL	§31222509303	❌	74	74	0	🔴
Smoke Codex	§31222433145	❌	154	121	33	🔴
Smoke Gemini	§31222432945	❌	59	59	0	🔴

Missing Firewall Logs (access.log)

Workflow	Run ID	Date	Link
Daily BYOK Ollama Test	31223940604	2026-08-07	§31223940604
Smoke Copilot - AOAI (Entra)	31222505053	2026-08-07	§31222505053
Smoke Copilot	31222502383	2026-08-07	§31222502383
Impeccable Skills Reviewer	31221978069	2026-08-07	§31221978069
Cache directory setup	31227824424	2026-08-07	§31227824424
PR Sous Chef	31227649983	2026-08-07	§31227649983
PR Sous Chef	31226747645	2026-08-07	§31226747645
Detection Analysis Report	31226672216	2026-08-07	§31226672216
Daily Reliability Review	31225875315	2026-08-07	§31225875315
Impeccable Skills Reviewer	31223955510	2026-08-07	§31223955510
Matt Pocock Skills Reviewer	31223955505	2026-08-07	§31223955505
Design Decision Gate 🏗️	31223955476	2026-08-07	§31223955476
Test Quality Sentinel	31223955470	2026-08-07	§31223955470
PR Code Quality Reviewer	31223955453	2026-08-07	§31223955453
Daily AWF Spec Compiler Surfacing Review	31223464787	2026-08-07	§31223464787
Issue Monster	31223293352	2026-08-07	§31223293352
ESLint Monster	31222548390	2026-08-07	§31222548390
Smoke OTEL	31222509303	2026-08-07	§31222509303
Smoke Codex	31222433145	2026-08-07	§31222433145
Smoke Gemini	31222432945	2026-08-07	§31222432945

MCP-Enabled Runs

Workflow	Run ID	Telemetry Source	Entries	Servers	Tool Calls	Errors	Status
Daily BYOK Ollama Test	§31223940604	❌ None	N/A	N/A	N/A	N/A	🔴
Smoke Copilot - AOAI (Entra)	§31222505053	❌ None	N/A	N/A	N/A	N/A	🔴
Smoke Copilot	§31222502383	❌ None	N/A	N/A	N/A	N/A	🔴
Impeccable Skills Reviewer	§31221978069	❌ None	N/A	N/A	N/A	N/A	🔴
Cache directory setup	§31227824424	❌ None	N/A	N/A	N/A	N/A	🔴
PR Sous Chef	§31227649983	❌ None	N/A	N/A	N/A	N/A	🔴
PR Sous Chef	§31226747645	❌ None	N/A	N/A	N/A	N/A	🔴
Detection Analysis Report	§31226672216	❌ None	N/A	N/A	N/A	N/A	🔴
Daily Reliability Review	§31225875315	❌ None	N/A	N/A	N/A	N/A	🔴
Impeccable Skills Reviewer	§31223955510	❌ None	N/A	N/A	N/A	N/A	🔴
Matt Pocock Skills Reviewer	§31223955505	❌ None	N/A	N/A	N/A	N/A	🔴
Design Decision Gate 🏗️	§31223955476	❌ None	N/A	N/A	N/A	N/A	🔴
Test Quality Sentinel	§31223955470	❌ None	N/A	N/A	N/A	N/A	🔴
PR Code Quality Reviewer	§31223955453	❌ None	N/A	N/A	N/A	N/A	🔴
Daily AWF Spec Compiler Surfacing Review	§31223464787	❌ None	N/A	N/A	N/A	N/A	🔴
Issue Monster	§31223293352	❌ None	N/A	N/A	N/A	N/A	🔴
ESLint Monster	§31222548390	❌ None	N/A	N/A	N/A	N/A	🔴
Smoke OTEL	§31222509303	❌ None	N/A	N/A	N/A	N/A	🔴
Smoke Codex	§31222433145	❌ None	N/A	N/A	N/A	N/A	🔴
Smoke Gemini	§31222432945	❌ None	N/A	N/A	N/A	N/A	🔴

Missing MCP Telemetry (no gateway.jsonl or rpc-messages.jsonl)

Workflow	Run ID	Date	Link
Daily BYOK Ollama Test	31223940604	2026-08-07	§31223940604
Smoke Copilot - AOAI (Entra)	31222505053	2026-08-07	§31222505053
Smoke Copilot	31222502383	2026-08-07	§31222502383
Impeccable Skills Reviewer	31221978069	2026-08-07	§31221978069
Cache directory setup	31227824424	2026-08-07	§31227824424
PR Sous Chef	31227649983	2026-08-07	§31227649983
PR Sous Chef	31226747645	2026-08-07	§31226747645
Detection Analysis Report	31226672216	2026-08-07	§31226672216
Daily Reliability Review	31225875315	2026-08-07	§31225875315
Impeccable Skills Reviewer	31223955510	2026-08-07	§31223955510
Matt Pocock Skills Reviewer	31223955505	2026-08-07	§31223955505
Design Decision Gate 🏗️	31223955476	2026-08-07	§31223955476
Test Quality Sentinel	31223955470	2026-08-07	§31223955470
PR Code Quality Reviewer	31223955453	2026-08-07	§31223955453
Daily AWF Spec Compiler Surfacing Review	31223464787	2026-08-07	§31223464787
Issue Monster	31223293352	2026-08-07	§31223293352
ESLint Monster	31222548390	2026-08-07	§31222548390
Smoke OTEL	31222509303	2026-08-07	§31222509303
Smoke Codex	31222433145	2026-08-07	§31222433145
Smoke Gemini	31222432945	2026-08-07	§31222432945

🔍 Telemetry Quality Analysis

Firewall Log Quality

• Recursive discovery result for raw Squid logs: no access.log files found anywhere under the 20 sampled run-<id>/ directories.
• Total firewall requests visible in derived summaries: 1426
• Domains accessed: 33 unique
• Blocked requests: 106 (7.4%)
• Most accessed domains: api.githubcopilot.com:443, o205451.ingest.us.sentry.io:443, otlp-gateway-prod-eu-west-2.grafana.net:443, api.anthropic.com:443, api.openai.com:443
• Most blocked domains: content-autofill.googleapis.com:443, www.google.com:443, accounts.google.com:443, android.clients.google.com:443, safebrowsingohttpgateway.googleapis.com:443

Gateway Log Quality

• Telemetry source: none retained; recursive discovery found no gateway.jsonl and no rpc-messages.jsonl in any sampled run directory.
• Total raw MCP entries analyzed: 0
• MCP servers used: N/A from raw telemetry; run_summary.json confirms MCP Gateway startup in all sampled runs.
• Total tool calls: N/A
• Error rate: N/A
• Average response time: N/A

Additional Artifact Coverage

• agent-stdio.log: 20/20
• agent_output.json: 19/20
• summary.json: 20/20
• run_summary.json: 20/20
• awf-config.json: 2/20 (runs §31223940604 and §31223955453)

Healthy Runs Summary

• No sampled run met the raw-log health criteria for firewall or MCP telemetry. Derived summaries exist, but canonical debugging artifacts do not.

Recommended Actions

1. Preserve raw Squid access.log in the agent artifact bundle for every firewall-enabled run, including workflow_call-prefixed paths, and fail the observability upload step when the file is absent.
2. Preserve either mcp-logs/gateway.jsonl or mcp-logs/rpc-messages.jsonl for every MCP-enabled run and add a post-agent assertion that marks the run degraded when both are missing.
3. Standardize metadata retention by always uploading aw_info.json or awf-config.json, plus safe_output.jsonl, so raw telemetry can be correlated with configuration and safe-output activity.

📊 Historical Trends

• This run did not compute a multi-day baseline beyond the current 7-day sampling window. The immediate signal is already severe: 0.0% raw firewall coverage and 0.0% raw MCP telemetry coverage in the capped sample.

References:

• §31223940604
• §31222505053
• §31222433145

Generated by 📊 Daily Observability Report for AWF Firewall and MCP Gateway · gpt54 · 107.4 AIC · ⌖ 7.88 AIC · ⊞ 13.9K · ◷

• expires on Aug 8, 2026, 4:01 PM UTC-08:00