[observability] Observability Coverage Report - 2026-08-16

[github-actions[bot]]

Executive Summary

I analyzed 20 representative workflow runs from the last 7 days, capped at one run per workflow and prioritizing failed runs where available. Both core observability surfaces met basic coverage expectations: all 20 firewall-enabled runs contained a discovered access.log, and all 20 MCP-enabled runs contained discovered MCP telemetry under mcp-logs/rpc-messages.jsonl.

No critical missing-log condition was found. The main issues are quality-related rather than absence-related. Firewall coverage is weakened because only 2 of 20 analyzed runs recorded any blocked egress, which means 18 runs do not demonstrate deny-path logging. MCP coverage relies entirely on the raw JSON-RPC fallback log; all 202 selected telemetry entries were parseable JSONL, but none included the explicit top-level type field required by this audit rubric, so every MCP-enabled run is a schema-quality warning rather than a telemetry absence failure.

Key Alerts and Anomalies

No critical issues detected.

🔴 Critical Issues:

• None. No firewall-enabled run was missing access.log, and no MCP-enabled run was missing both gateway.jsonl and rpc-messages.jsonl.

⚠️ Warnings:

• Firewall deny-path visibility is weak: 18 of 20 analyzed runs logged allowed traffic only, with no blocked requests visible in access.log.
• MCP telemetry is present only via rpc-messages.jsonl fallback: 20 of 20 analyzed runs had parseable JSONL, but 0 of 202 entries exposed the rubric's required top-level type field.
• safeoutputs.jsonl was absent in 4 runs (AI Moderator, Avenger, Daily AWF Spec Compiler Surfacing Review, Smoke Gemini), reducing secondary output-tracing coverage.

Coverage Summary

Component	Runs Analyzed	Logs Present	Coverage	Status
AWF Firewall (access.log)	20	20	100%	⚠️
MCP Gateway (gateway.jsonl or rpc-messages.jsonl)	20	20	100%	⚠️

📋 Detailed Run Analysis

Discovery Notes

• Firewall log path discovered for all analyzed runs: sandbox/firewall/logs/access.log.
• MCP telemetry path discovered for all analyzed runs: mcp-logs/rpc-messages.jsonl.
• No selected run contained gateway.jsonl; all MCP coverage came from the canonical fallback file.

Firewall-Enabled Runs

Workflow	Run ID	access.log	Entries	Allowed	Blocked	Status
AI Moderator	§31977459595	✅	40	24	0	⚠️
Agent Job Health Monitor	§31978490829	✅	474	240	0	⚠️
Avenger	§31976674745	✅	170	93	0	⚠️
Cache directory setup	§31979490474	✅	254	122	0	⚠️
Daily AWF Spec Compiler Surfacing Review	§31975886283	✅	41	2	0	⚠️
Design Decision Gate 🏗️	§31976124522	✅	145	72	0	⚠️
Detection Analysis Report	§31978446846	✅	410	168	0	⚠️
Duplicate Code Detector	§31974713438	✅	168	88	0	⚠️
ESLint Monster	§31974954632	✅	23	14	0	⚠️
Impeccable Skills Reviewer	§31976124568	✅	245	84	0	⚠️
Matt Pocock Skills Reviewer	§31976124616	✅	279	109	0	⚠️
PR Code Quality Reviewer	§31976124874	✅	37	22	0	⚠️
PR Sous Chef	§31979349605	✅	39	20	0	⚠️
Ponytail Reviewer	§31976124670	✅	76	34	1	✅
Smoke Copilot - AOAI (Entra)	§31974723462	✅	171	66	0	⚠️
Smoke Copilot MAI	§31974724676	✅	32	10	0	⚠️
Smoke Copilot Small	§31974725563	✅	20	10	0	⚠️
Smoke Gemini	§31974687297	✅	5	2	1	✅
Smoke OTEL	§31974726993	✅	154	85	0	⚠️
Test Quality Sentinel	§31976124661	✅	30	16	0	⚠️

Missing Firewall Logs (access.log)

None.

MCP-Enabled Runs

Workflow	Run ID	Telemetry Source	Entries	Servers	Tool Calls	Errors	Status
AI Moderator	§31977459595	rpc-messages.jsonl	2	1	0	0	⚠️
Agent Job Health Monitor	§31978490829	rpc-messages.jsonl	8	2	2	0	⚠️
Avenger	§31976674745	rpc-messages.jsonl	2	1	0	0	⚠️
Cache directory setup	§31979490474	rpc-messages.jsonl	8	2	1	0	⚠️
Daily AWF Spec Compiler Surfacing Review	§31975886283	rpc-messages.jsonl	2	1	0	0	⚠️
Design Decision Gate 🏗️	§31976124522	rpc-messages.jsonl	8	1	3	0	⚠️
Detection Analysis Report	§31978446846	rpc-messages.jsonl	18	3	5	0	⚠️
Duplicate Code Detector	§31974713438	rpc-messages.jsonl	34	3	12	0	⚠️
ESLint Monster	§31974954632	rpc-messages.jsonl	4	1	1	0	⚠️
Impeccable Skills Reviewer	§31976124568	rpc-messages.jsonl	8	1	3	0	⚠️
Matt Pocock Skills Reviewer	§31976124616	rpc-messages.jsonl	12	1	5	0	⚠️
PR Code Quality Reviewer	§31976124874	rpc-messages.jsonl	4	1	1	0	⚠️
PR Sous Chef	§31979349605	rpc-messages.jsonl	4	1	1	0	⚠️
Ponytail Reviewer	§31976124670	rpc-messages.jsonl	8	2	1	0	⚠️
Smoke Copilot - AOAI (Entra)	§31974723462	rpc-messages.jsonl	26	4	8	0	⚠️
Smoke Copilot MAI	§31974724676	rpc-messages.jsonl	6	1	2	0	⚠️
Smoke Copilot Small	§31974725563	rpc-messages.jsonl	8	2	1	0	⚠️
Smoke Gemini	§31974687297	rpc-messages.jsonl	6	2	0	0	⚠️
Smoke OTEL	§31974726993	rpc-messages.jsonl	30	3	12	0	⚠️
Test Quality Sentinel	§31976124661	rpc-messages.jsonl	4	1	1	0	⚠️

Missing MCP Telemetry (no gateway.jsonl or rpc-messages.jsonl)

None.

🔍 Telemetry Quality Analysis

Firewall Log Quality

• Total access.log entries analyzed: 2,813
• Domains accessed: 17 unique
• Blocked requests: 2 (0.16%)
• Most accessed domains: api.anthropic.com:443 (375), api.githubcopilot.com:443 (296), o205451.ingest.us.sentry.io:443 (235)

Gateway Log Quality

• Telemetry source: rpc-messages.jsonl fallback in all 20 runs; gateway.jsonl was not present in the selected set
• Total entries analyzed: 202
• JSONL parse success: 202 of 202 entries
• Required top-level type field present: 0 of 202 entries
• MCP servers used: safeoutputs, serena, github, sentry, agenticworkflows, grafana, mcpscripts
• Total tool calls derived from outgoing tools/call requests: 59
• Error rate: 0%
• Average response time: N/A when derived from rpc-messages.jsonl without request/response duration pairing

Healthy Runs Summary

• Firewall-healthy runs with both allowed and blocked traffic visible: Ponytail Reviewer, Smoke Gemini
• Failed runs still retained core observability artifacts: Avenger and Smoke Gemini both had access.log, MCP telemetry, agent-stdio.log, and AWF config metadata
• Supporting artifact coverage across the 20-run set: agent-stdio.log 20/20, AWF config metadata 20/20, safeoutputs.jsonl 16/20

Recommended Actions

1. Add a lightweight synthetic denied-egress probe to at least one representative job per workflow family so access.log consistently captures both allow and deny paths.
2. Emit gateway.jsonl in more runs, or extend rpc-messages.jsonl so every entry includes an explicit top-level type field compatible with the audit rubric.
3. Standardize safeoutputs.jsonl emission even for noop-only or no-write runs so downstream debugging can correlate the decision path reliably.

📊 Historical Trends

No retained baseline was available inside this run for day-over-day comparison. This report is a current 7-day-window snapshot using the most recent representative 20-run sample.

References: §31976674745, §31974687297, §31978446846

Generated by 📊 Daily Observability Report for AWF Firewall and MCP Gateway · gpt54 · 115.1 AIC · ⌖ 12.9 AIC · ⊞ 14K · ◷

• expires on Aug 17, 2026, 3:55 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Daily Observability Report for AWF Firewall and MCP Gateway.

A newer discussion is available at Discussion #53529.