[observability] Observability Coverage Report - 2026-09-04

[github-actions[bot]]

Executive Summary

I analyzed 20 recent workflow runs downloaded from the last 7-day window, capped per the workflow guidance and limited to at most two runs per workflow. The fetched sample was recent-heavy: the initial repository-wide logs call returned 30 runs spanning roughly one hour on 2026-09-04, so this report is a representative capped slice rather than full-week census coverage.

AWF Firewall coverage was 65.0% across 20 firewall-enabled runs, and MCP telemetry coverage was 65.0% across 20 MCP-enabled runs. 7 runs had critical observability gaps, all caused by missing runtime artifacts on workflows whose compiled definitions include both firewall and MCP Gateway steps.

Where logs were present, the firewall and MCP story was consistent: access.log existed under sandbox/firewall/logs/, and MCP telemetry was recorded via the canonical fallback mcp-logs/rpc-messages.jsonl. No analyzed run exposed gateway.jsonl; current healthy coverage relies entirely on the RPC fallback path.

Key Alerts and Anomalies

🔴 Critical Issues:

• Impeccable Skills Reviewer run 33927409720 is firewall-enabled but missing access.log.
• Impeccable Skills Reviewer run 33927409720 is MCP-enabled but missing both gateway.jsonl and rpc-messages.jsonl.
• PR Code Quality Reviewer run 33927409806 is firewall-enabled but missing access.log.
• PR Code Quality Reviewer run 33927409806 is MCP-enabled but missing both gateway.jsonl and rpc-messages.jsonl.
• Test Quality Sentinel run 33927409809 is firewall-enabled but missing access.log.
• Test Quality Sentinel run 33927409809 is MCP-enabled but missing both gateway.jsonl and rpc-messages.jsonl.
• Ponytail Reviewer run 33927409826 is firewall-enabled but missing access.log.
• Ponytail Reviewer run 33927409826 is MCP-enabled but missing both gateway.jsonl and rpc-messages.jsonl.
• Design Decision Gate 🏗️ run 33927409837 is firewall-enabled but missing access.log.
• Design Decision Gate 🏗️ run 33927409837 is MCP-enabled but missing both gateway.jsonl and rpc-messages.jsonl.
• Matt Pocock Skills Reviewer run 33927409908 is firewall-enabled but missing access.log.
• Matt Pocock Skills Reviewer run 33927409908 is MCP-enabled but missing both gateway.jsonl and rpc-messages.jsonl.
• AI Moderator run 33927325443 is firewall-enabled but missing access.log.
• AI Moderator run 33927325443 is MCP-enabled but missing both gateway.jsonl and rpc-messages.jsonl.

⚠️ Warnings:

• Daily Documentation Healer run 33930351076: firewall log present with 22 entries, allowed=4, blocked=0
• Matt Pocock Skills Reviewer run 33927521651: firewall log present with 704 entries, allowed=259, blocked=0
• Impeccable Skills Reviewer run 33927521662: firewall log present with 593 entries, allowed=148, blocked=0
• Design Decision Gate 🏗️ run 33927521473: firewall log present with 25 entries, allowed=14, blocked=0
• Test Quality Sentinel run 33927521595: firewall log present with 64 entries, allowed=18, blocked=0
• Daily Reliability Review run 33927293402: firewall log present with 893 entries, allowed=5, blocked=0
• Daily BYOK Ollama Test run 33925327513: firewall log present with 6 entries, allowed=2, blocked=0
• Daily AWF Spec Compiler Surfacing Review run 33925033555: firewall log present with 22 entries, allowed=4, blocked=0
• Auto-Triage Issues run 33923711273: firewall log present with 10 entries, allowed=2, blocked=0
• PR Sous Chef run 33923268323: firewall log present with 97 entries, allowed=58, blocked=0

Coverage Summary

Component	Runs Analyzed	Logs Present	Coverage	Status
AWF Firewall (access.log)	20	13	65.0%	🔴
MCP Gateway (gateway.jsonl or rpc-messages.jsonl)	20	13	65.0%	🔴

📋 Detailed Run Analysis

Firewall-Enabled Runs

Workflow	Run ID	access.log	Entries	Allowed	Blocked	Status
PR Sous Chef	§33930375719	✅	162	68	8	✅
Daily Documentation Healer	§33930351076	✅	22	4	0	⚠️
Matt Pocock Skills Reviewer	§33927521651	✅	704	259	0	⚠️
Impeccable Skills Reviewer	§33927521662	✅	593	148	0	⚠️
Design Decision Gate 🏗️	§33927521473	✅	25	14	0	⚠️
Test Quality Sentinel	§33927521595	✅	64	18	0	⚠️
Ponytail Reviewer	§33927521624	✅	61	28	1	✅
Impeccable Skills Reviewer	§33927409720	❌	0	0	0	🔴
PR Code Quality Reviewer	§33927409806	❌	0	0	0	🔴
Test Quality Sentinel	§33927409809	❌	0	0	0	🔴
Ponytail Reviewer	§33927409826	❌	0	0	0	🔴
Design Decision Gate 🏗️	§33927409837	❌	0	0	0	🔴
Matt Pocock Skills Reviewer	§33927409908	❌	0	0	0	🔴
AI Moderator	§33927325443	❌	0	0	0	🔴
Daily Reliability Review	§33927293402	✅	893	5	0	⚠️
Daily BYOK Ollama Test	§33925327513	✅	6	2	0	⚠️
Daily AWF Spec Compiler Surfacing Review	§33925033555	✅	22	4	0	⚠️
Issue Monster	§33924924030	✅	65	26	2	✅
Auto-Triage Issues	§33923711273	✅	10	2	0	⚠️
PR Sous Chef	§33923268323	✅	97	58	0	⚠️

Missing Firewall Logs (access.log)

Workflow	Run ID	Date	Link
Impeccable Skills Reviewer	33927409720	2026-09-04	§33927409720
PR Code Quality Reviewer	33927409806	2026-09-04	§33927409806
Test Quality Sentinel	33927409809	2026-09-04	§33927409809
Ponytail Reviewer	33927409826	2026-09-04	§33927409826
Design Decision Gate 🏗️	33927409837	2026-09-04	§33927409837
Matt Pocock Skills Reviewer	33927409908	2026-09-04	§33927409908
AI Moderator	33927325443	2026-09-04	§33927325443

MCP-Enabled Runs

Workflow	Run ID	Telemetry Source	Entries	Servers	Tool Calls	Errors	Status
PR Sous Chef	§33930375719	rpc-messages.jsonl	12	1	5	0	✅
Daily Documentation Healer	§33930351076	rpc-messages.jsonl	2	1	0	0	✅
Matt Pocock Skills Reviewer	§33927521651	rpc-messages.jsonl	2	1	0	0	✅
Impeccable Skills Reviewer	§33927521662	rpc-messages.jsonl	2	1	0	0	✅
Design Decision Gate 🏗️	§33927521473	rpc-messages.jsonl	4	1	1	0	✅
Test Quality Sentinel	§33927521595	rpc-messages.jsonl	4	1	1	0	✅
Ponytail Reviewer	§33927521624	rpc-messages.jsonl	4	1	1	0	✅
Impeccable Skills Reviewer	§33927409720	❌ None	0	0	0	0	🔴
PR Code Quality Reviewer	§33927409806	❌ None	0	0	0	0	🔴
Test Quality Sentinel	§33927409809	❌ None	0	0	0	0	🔴
Ponytail Reviewer	§33927409826	❌ None	0	0	0	0	🔴
Design Decision Gate 🏗️	§33927409837	❌ None	0	0	0	0	🔴
Matt Pocock Skills Reviewer	§33927409908	❌ None	0	0	0	0	🔴
AI Moderator	§33927325443	❌ None	0	0	0	0	🔴
Daily Reliability Review	§33927293402	rpc-messages.jsonl	6	2	0	0	✅
Daily BYOK Ollama Test	§33925327513	rpc-messages.jsonl	6	2	0	0	✅
Daily AWF Spec Compiler Surfacing Review	§33925033555	rpc-messages.jsonl	2	1	0	0	✅
Issue Monster	§33924924030	rpc-messages.jsonl	14	1	6	0	✅
Auto-Triage Issues	§33923711273	rpc-messages.jsonl	2	1	0	0	✅
PR Sous Chef	§33923268323	rpc-messages.jsonl	12	1	5	0	✅

Missing MCP Telemetry (no gateway.jsonl or rpc-messages.jsonl)

Workflow	Run ID	Date	Link
Impeccable Skills Reviewer	33927409720	2026-09-04	§33927409720
PR Code Quality Reviewer	33927409806	2026-09-04	§33927409806
Test Quality Sentinel	33927409809	2026-09-04	§33927409809
Ponytail Reviewer	33927409826	2026-09-04	§33927409826
Design Decision Gate 🏗️	33927409837	2026-09-04	§33927409837
Matt Pocock Skills Reviewer	33927409908	2026-09-04	§33927409908
AI Moderator	33927325443	2026-09-04	§33927325443

🔍 Telemetry Quality Analysis

Firewall Log Quality

• Total access.log entries analyzed: 2724
• Domains accessed: 9 unique
• Blocked requests: 11 (0.4%)
• Most accessed domains: api.githubcopilot.com, o205451.ingest.us.sentry.io, otlp-gateway-prod-eu-west-2.grafana.net

Gateway Log Quality

• Telemetry source: rpc-messages.jsonl (canonical fallback) on all healthy MCP runs; gateway.jsonl absent in this sample
• Total entries analyzed: 72
• MCP servers used: safeoutputs, github
• Total tool calls: 19
• Error rate: 0.0%
• Average response time: N/A when derived from rpc-messages.jsonl without request/response pairing

Healthy Runs Summary

3 analyzed runs had both firewall and MCP telemetry present. Representative healthy examples: PR Sous Chef §33930375719, Ponytail Reviewer §33927521624, Issue Monster §33924924030.

Recommended Actions

1. Investigate why several failed runs produced no uploaded runtime artifacts even though their compiled workflows include both Start MCP Gateway and Print firewall logs; missing artifacts are the direct cause of all critical findings.
2. Add or harden post-failure artifact upload for /tmp/gh-aw/sandbox/firewall/logs/ and /tmp/gh-aw/mcp-logs/ so failed agent jobs still emit debugging telemetry.
3. Consider promoting structured gateway.jsonl generation alongside the existing RPC fallback, since the current healthy runs depend entirely on rpc-messages.jsonl for MCP observability.

📊 Historical Trends

Historical trend analysis is not included in this single-sample daily report. The current run provides a point-in-time coverage slice from the last 7 days.

---

Report generated automatically by the Daily Observability Report workflow
Analysis window: Last 7 days | Runs analyzed: 20

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• ab.chatgpt.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"

See Network Configuration for more information.

Generated by 📊 Daily Observability Report for AWF Firewall and MCP Gateway · codex · gpt54 · 125.9 AIC · ⌖ 8.98 AIC · ⊞ 14.2K · ◷

• expires on Sep 5, 2026, 3:54 PM UTC-08:00

[github-actions[bot]]

Ook ook! Caveman bot say hello. Me run smoke test, all fire and rock solid. Good discussion, me like.

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• host.docker.internal
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "host.docker.internal"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · copilot · auto · 47.5 AIC · ⌖ 2.83 AIC · ⊞ 10.1K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}