Skip to content

v0.4.0 — SRE troubleshooting with GraphRAG

Latest

Choose a tag to compare

@Chuck0816 Chuck0816 released this 18 Nov 04:13
55b01db

Highlights

PolicyDesk now has a second domain for SRE incident troubleshooting, next to the existing policy assistant. The policy domain is unchanged and remains the default.

With KB_DOMAIN=oncall, PolicyDesk combines a service catalog in Neo4j, runbooks and postmortems in Markdown, and GraphRAG retrieval to answer questions like:

  • If product-catalog is down, which services are affected?
  • checkout latency is up and payment errors are rising. What is the likely root cause?
  • Who owns valkey-cart and where do we escalate?
  • Has this Kafka consumer lag happened before? How was it fixed?

What's new

Knowledge graph

  • Service catalog import: service_catalog.yaml and alerts.yaml are validated and loaded into Neo4j without an LLM, covering services, datastores, teams, dependencies, owners, escalation paths, and alerts linked to runbook sections.
  • Postmortem extraction: Qwen extracts incidents, affected services, root causes, symptoms, and mitigations with a strict schema. Extracted names are aligned to the catalog, including Chinese aliases, and symptoms to a controlled vocabulary. Results are cached in Redis.
  • Safe provenance: every graph fact records the documents and chunks that support it. Deleting a postmortem removes only facts that no other document supports. A failed extraction changes nothing, and a failed graph write rolls back.

Retrieval

  • Three modes: hybrid (the existing chain, used as the baseline), graph, and hybrid+graph.
  • Deterministic entity linking on the original question and fixed Cypher templates; the LLM never writes Cypher.
  • Root-cause ranking based on shared downstream dependencies, distance, and incident history.
  • Graph facts keep reserved slots after reranking, so the reranker cannot drop them.
  • The catalog is also rendered as text, so the hybrid baseline sees the same facts and the comparison is fair.

Interface

  • Web demo: mode switch, side-by-side comparison (hybrid vs hybrid + graph), separate document evidence and graph reasoning paths, catalog import, Markdown upload, and alert updates without re-uploading the catalog.
  • CLI: import-catalog, Markdown add, and --mode.
  • Metrics: retrieval, first-token, generation, and total time, plus LLM calls and estimated tokens, for every answer.
  • Structured output: each answer ends with a JSON block (root_cause_services, affected_services, owner_team, uncertain) for evaluation.

Demo corpus

examples/oncall_demo/ contains a 16-component shop catalog, 5 alerts, 5 runbooks, 5 postmortems, and 16 test questions with expected answers.

Upgrade

pip install -r requirements.txt
docker compose --profile oncall up -d --wait