Incident Diagnosis

Module: Incident Diagnosis

Module flag: incident

Overview

The Incident Diagnosis module turns a broad report such as “production is failing” into one bounded, read-only correlation run. It inspects recent Cloud Run deployments and revisions, current versus baseline request/error/latency metrics, error logs, Admin Activity and IAM changes, discovered dependencies, and optional Personalized Service Health events.

The result contains possible root causes in ranked order. Every hypothesis includes an independent likelihood score, evidence, contradicting evidence, and suggested read-only investigation. A score is an evidence-strength heuristic, not a probability; scores do not need to sum to 100.

Enable This Module

GCP_PROJECT_ID=my-project ./aura-tracker-gcp --modules incident

The module loads its own GCP clients. You do not need to enable the raw cloudrun, logging, or monitoring tool modules unless you also want those individual tools available to the LLM.

Optional Platform-Health Correlation

Important

This integration is optional and disabled by default. Aura Tracker queries Google's documented servicehealth.googleapis.com/activity log stream only when include_platform_health=true. Before opting in, verify that billing is enabled, enable servicehealth.googleapis.com, and give the runtime identity both roles/logging.viewer and roles/servicehealth.viewer. Missing optional access does not stop the other incident checks; it is surfaced as a skipped check or coverage gap.

Automatic Setup (Recommended)

Run the idempotent admin setup command. It creates aura-tracker-mcp when missing; when the account already exists, it keeps the account and reconciles the requested APIs and IAM roles.

export PROJECT_ID=my-project
SERVICE_HEALTH_ENABLED=true bash scripts/setup-iam.sh

This single command enables servicehealth.googleapis.com, grants roles/servicehealth.viewer, and reconciles the core roles/logging.viewer grant. Run it as a team admin with permission to enable services and change project IAM. Setup flags are additive; omitting the flag later does not disable the API or revoke the role.

If Aura Tracker runs with user Application Default Credentials, grant the optional role to that user instead of the service account.

Manual Equivalent

SA_EMAIL="aura-tracker-mcp@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud services enable servicehealth.googleapis.com \
  --project="${PROJECT_ID}"

gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
  --member="serviceAccount:${SA_EMAIL}" \
  --role="roles/servicehealth.viewer" \
  --condition=None

Opt In and Verify

Set the tool parameter on each diagnosis that should include platform events:

{
  "project_id": "my-project",
  "include_platform_health": true
}

Confirm that the API is enabled and test read access to the log stream:

gcloud services list \
  --enabled \
  --project="${PROJECT_ID}" \
  --filter='config.name:servicehealth.googleapis.com'

gcloud logging read \
  "logName=\"projects/${PROJECT_ID}/logs/servicehealth.googleapis.com%2Factivity\"" \
  --project="${PROJECT_ID}" \
  --freshness=30d \
  --limit=5

An empty query can be valid when the project had no matching events. Permission or API errors indicate setup is incomplete. Google notes that new events can take a few hours to appear after API enablement. For details, see Google's Service Health logs and Service Health access control documentation.

Required APIs

gcloud services enable \
  run.googleapis.com \
  logging.googleapis.com \
  monitoring.googleapis.com \
  pubsub.googleapis.com \
  sqladmin.googleapis.com \
  vpcaccess.googleapis.com

servicehealth.googleapis.com is optional; enable it only for platform-health correlation.

Required IAM

Role	Used for
roles/run.viewer	Service state, traffic, and revision history
roles/logging.viewer	Error logs, Admin Activity/IAM changes, and Service Health logs
roles/monitoring.viewer	Request count, 5xx rate, latency, and downstream Cloud Run health
roles/pubsub.viewer	Pub/Sub dependency health
roles/cloudsql.viewer	Cloud SQL dependency state
roles/vpcaccess.viewer	Serverless VPC Access connector state
roles/servicehealth.viewer	Optional Personalized Service Health access; required only with include_platform_health=true

No mutation role is required. scripts/setup-iam.sh reconciles the core read-only roles. Pass SERVICE_HEALTH_ENABLED=true at any time to enable the Service Health API and reconcile the optional viewer role, including when the service account already exists.

Tool

gcp_incident_diagnose

Parameter	Required	Default	Description
project_id	no	default environment	Environment alias or configured project ID
environment	no	production	Value matched against env, environment, or stage labels
service_name	no	inferred	Cloud Run service; omit for label-based production discovery
region	no	discovered	Region; omit to find a named service across regions
lookback_minutes	no	60	Active incident window, 5–720 minutes
baseline_minutes	no	240	Comparison window immediately preceding the active window
max_services	no	10	Maximum inferred services, 1–25
max_dependencies	no	10	Maximum dependencies checked per service, 1–25
detail_level	no	standard	summary, standard, or detailed; controls returned cause/evidence/pattern/timeline limits
include_platform_health	no	false	Query active, relevant Service Health log events

The active and baseline windows together are capped at 1,440 minutes. The complete diagnosis has a 45-second budget and checks at most four services concurrently.

Scope Safety

• An explicit service and region is accepted directly.
• A service without a region is discovered across regions. Every match within max_services is analyzed and ambiguity is reported.
• With no service, only services whose env, environment, or stage label matches the requested environment are analyzed.
• If no matching production label exists, the tool returns status: needs_scope with candidates. It does not silently scan the entire project.

Correlation and Scoring

The engine compares the active window with the preceding baseline and evaluates these hypotheses:

• recent deployment or revision regression;
• IAM or runtime service-account regression;
• application exception or request-handler failure;
• downstream dependency failure;
• p99 latency regression;
• active, relevant Google Cloud platform event;
• unclassified incident when symptoms are real but causal evidence is incomplete.

Evidence adds explicit weights and contradicting evidence subtracts explicit penalties. Scores are clamped to 0–100 and mapped to high (75+), medium (45–74), or low (20–44). Error messages are normalized into stable fingerprints so request IDs and numeric values do not split one recurring failure into many patterns.

Dependency Checks

Topology discovery supplies the bounded dependency list. The engine checks:

• Cloud SQL instance state;
• Pub/Sub topic existence plus a bounded subscription check. The newest gauge sample inside the incident window is used—rather than the historical maximum—so a recovered backlog spike is not reported as an active failure. At least 10,000 undelivered messages or 300-second oldest-unacked age marks it unhealthy;
• Serverless VPC Access connector state;
• downstream Cloud Run 5xx rate;
• topology-only evidence for dependencies without a safe health endpoint.

A failed collector does not fail the whole diagnosis. The coverage object records complete, partial, and skipped checks, while warnings explain missing permissions, API failures, truncation, or timeout.

If no failure signal is visible but one or more collectors failed, the top-level status is inconclusive, not healthy.

Example

“Production is failing. Run gcp_incident_diagnose for my-project, include platform health, and show likely causes, evidence, and the next read-only investigation.”

If your services are not environment-labelled:

“Diagnose service payments-api in us-central1 for the last 90 minutes and compare it with the previous six hours.”

Guardrails

• The tool is read-only and is annotated as non-destructive and idempotent.
• It never rolls back a revision or changes traffic.
• Suggested investigations name existing read-only MCP tools and include arguments, but remain suggestions.
• Revision fingerprints can detect configuration changes without returning environment or secret values.
• Optional platform health is only ranked when an active event overlaps the current window and references Cloud Run, the affected region, or the service.