[observability] Observability Coverage Report - 2026-08-07

[github-actions[bot]]

Executive Summary

Analyzed 14 recent workflow runs from the last 7 days, capped for breadth across workflows and weighted toward failed runs. The sampled set contained 4 failed runs, 9 successful runs, and 1 in-progress run. AWF firewall and MCP runtime evidence appeared in all 14 runs, but only one run, §31135930383, retained the critical observability artifacts needed for debugging.

Firewall coverage was 7.1% (1 of 14 firewall-enabled runs with access.log), and MCP telemetry coverage was 7.1% (1 of 14 MCP-enabled runs with either gateway.jsonl or rpc-messages.jsonl). That leaves 13 sampled runs missing both network egress audit logs and MCP JSONL telemetry. This is a critical observability gap because runtime logs show the AWF topology and MCP gateway path were active even when the artifacts were not retained.

Key Alerts and Anomalies

Critical Issues:

• 13 of 14 firewall-enabled runs were missing access.log, so egress behavior could not be audited after the run.
• 13 of 14 MCP-enabled runs were missing both gateway.jsonl and rpc-messages.jsonl, so tool-call traces could not be reconstructed.
• §31134143397 confirms this is not just a post-download blind spot: the workflow job log explicitly says WARNING: Squid access.log not found under /tmp/gh-aw/sandbox/firewall/logs and No gateway.jsonl or rpc-messages.jsonl found for steering or DIFC_FILTERED scanning.

Warnings:

• The only retained firewall log came from §31135930383 and showed allowed traffic only; no deny-path entries were captured in the sampled period.
• The only retained MCP telemetry file in the sample was fallback rpc-messages.jsonl; no gateway.jsonl file was discovered in any analyzed run.
• Supporting metadata was also thin outside the single fully retained run: all_aw_info.json and safeoutputs.jsonl were present in only 1 of 14 runs.

Coverage Summary

Component	Runs Analyzed	Logs Present	Coverage	Status
AWF Firewall (access.log)	14	1	7.1%	Critical
MCP Gateway (gateway.jsonl or rpc-messages.jsonl)	14	1	7.1%	Critical
Overall complete observability coverage	14	1	7.1%	Critical

Detailed Run Analysis

Firewall-Enabled Runs

Workflow	Run ID	access.log	Entries	Allowed	Blocked	Status
Daily BYOK Ollama Test	§31134143397	Missing	0	0	0	Critical
Daily Reliability Review	§31135630380	Missing	0	0	0	Critical
PR Code Quality Reviewer	§31134098453	Missing	0	0	0	Critical
PR Code Quality Reviewer	§31132973257	Missing	0	0	0	Critical
Detection Analysis Report	§31135930383	Present	531	189	0	Warning
PR Description Updater	§31135528489	Missing	0	0	0	Critical
Impeccable Skills Reviewer	§31134098450	Missing	0	0	0	Critical
Issue Monster	§31136496374	Missing	0	0	0	Critical
Design Decision Gate	§31132973244	Missing	0	0	0	Critical
Matt Pocock Skills Reviewer	§31132973281	Missing	0	0	0	Critical
Test Quality Sentinel	§31132973191	Missing	0	0	0	Critical
ESLint Monster	§31132565983	Missing	0	0	0	Critical
Daily AWF Spec Compiler Surfacing Review	§31133776316	Missing	0	0	0	Critical
Duplicate Code Detector	§31132226792	Missing	0	0	0	Critical

Missing Firewall Logs (access.log)

Workflow	Run ID	Date	Link
Daily BYOK Ollama Test	31134143397	2026-08-07	§31134143397
Daily Reliability Review	31135630380	2026-08-07	§31135630380
PR Code Quality Reviewer	31134098453	2026-08-07	§31134098453
PR Code Quality Reviewer	31132973257	2026-08-06	§31132973257
PR Description Updater	31135528489	2026-08-07	§31135528489
Impeccable Skills Reviewer	31134098450	2026-08-07	§31134098450
Issue Monster	31136496374	2026-08-07	§31136496374
Design Decision Gate	31132973244	2026-08-06	§31132973244
Matt Pocock Skills Reviewer	31132973281	2026-08-06	§31132973281
Test Quality Sentinel	31132973191	2026-08-06	§31132973191
ESLint Monster	31132565983	2026-08-06	§31132565983
Daily AWF Spec Compiler Surfacing Review	31133776316	2026-08-07	§31133776316
Duplicate Code Detector	31132226792	2026-08-06	§31132226792

MCP-Enabled Runs

Workflow	Run ID	Telemetry Source	Entries	Servers	Tool Calls	Errors	Status
Daily BYOK Ollama Test	§31134143397	None	0	0	0	0	Critical
Daily Reliability Review	§31135630380	None	0	0	0	0	Critical
PR Code Quality Reviewer	§31134098453	None	0	0	0	0	Critical
PR Code Quality Reviewer	§31132973257	None	0	0	0	0	Critical
Detection Analysis Report	§31135930383	rpc-messages.jsonl	22	3	7	0	Healthy
PR Description Updater	§31135528489	None	0	0	0	0	Critical
Impeccable Skills Reviewer	§31134098450	None	0	0	0	0	Critical
Issue Monster	§31136496374	None	0	0	0	0	Critical
Design Decision Gate	§31132973244	None	0	0	0	0	Critical
Matt Pocock Skills Reviewer	§31132973281	None	0	0	0	0	Critical
Test Quality Sentinel	§31132973191	None	0	0	0	0	Critical
ESLint Monster	§31132565983	None	0	0	0	0	Critical
Daily AWF Spec Compiler Surfacing Review	§31133776316	None	0	0	0	0	Critical
Duplicate Code Detector	§31132226792	None	0	0	0	0	Critical

Missing MCP Telemetry (no gateway.jsonl or rpc-messages.jsonl)

Workflow	Run ID	Date	Link
Daily BYOK Ollama Test	31134143397	2026-08-07	§31134143397
Daily Reliability Review	31135630380	2026-08-07	§31135630380
PR Code Quality Reviewer	31134098453	2026-08-07	§31134098453
PR Code Quality Reviewer	31132973257	2026-08-06	§31132973257
PR Description Updater	31135528489	2026-08-07	§31135528489
Impeccable Skills Reviewer	31134098450	2026-08-07	§31134098450
Issue Monster	31136496374	2026-08-07	§31136496374
Design Decision Gate	31132973244	2026-08-06	§31132973244
Matt Pocock Skills Reviewer	31132973281	2026-08-06	§31132973281
Test Quality Sentinel	31132973191	2026-08-06	§31132973191
ESLint Monster	31132565983	2026-08-06	§31132565983
Daily AWF Spec Compiler Surfacing Review	31133776316	2026-08-07	§31133776316
Duplicate Code Detector	31132226792	2026-08-06	§31132226792

Telemetry Quality Analysis

Firewall Log Quality

• Only discovered access.log: run-31135930383/sandbox/firewall/logs/access.log
• Raw access.log lines analyzed: 531
• Parsed egress requests from run summary: 189
• Domains accessed: 3 unique
• Blocked requests: 0 (0%)
• Most accessed domains: api.anthropic.com:443 (119), o205451.ingest.us.sentry.io:443 (35), otlp-gateway-prod-eu-west-2.grafana.net:443 (35)

Gateway Log Quality

• Only discovered MCP telemetry file: run-31135930383/mcp-logs/rpc-messages.jsonl
• Telemetry source: rpc-messages.jsonl (canonical fallback)
• Total entries analyzed: 22
• MCP servers used: agenticworkflows, github, safeoutputs
• Total tool calls: 7 outgoing tools/call requests
• Error rate: 0%
• Average response time: N/A from fallback RPC log without pairing metadata

Additional Telemetry Coverage

• agent-stdio.log present: 13 of 14 runs
• run_summary.json present: 14 of 14 runs
• all_aw_info.json present: 1 of 14 runs
• safeoutputs.jsonl or safe_output.jsonl present: 1 of 14 runs

Healthy Runs Summary

• §31135930383 was the only sampled run with retained MCP JSONL telemetry and firewall access logs.
• Even that run used fallback rpc-messages.jsonl rather than preferred gateway.jsonl, and the firewall sample did not exercise any deny-path logging.

Recommended Actions

1. Fail artifact collection or post-run validation when a firewall-enabled run finishes without sandbox/firewall/logs/access.log.
2. Make MCP telemetry retention fail-closed when neither mcp-logs/gateway.jsonl nor mcp-logs/rpc-messages.jsonl is present for an MCP-enabled run, including early-exit failures.
3. Move all_aw_info.json and structured safe-output JSONL into the always-uploaded baseline artifact set so failed runs keep enough metadata to explain what was enabled.

Historical Trends

This run analyzed a capped 14-run sample from the last 7 days. No prior daily observability report baseline was compared in this execution, so trend conclusions are intentionally deferred.

References:

• §31134143397
• §31135930383
• §31135630380

Generated by 📊 Daily Observability Report for AWF Firewall and MCP Gateway · gpt54 · 101 AIC · ⊞ 13.8K · ◷

• expires on Aug 7, 2026, 5:11 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Daily Observability Report for AWF Firewall and MCP Gateway.

A newer discussion is available at Discussion #51237.