A local prototype of a multimodal AI incident commander for simulated SRE incidents. OpsVoice combines LLM-based tool selection, structured tool schemas, simulated infrastructure telemetry, incident-state management, voice interaction, Slack integration, human-in-the-loop guardrails, persistent conversation history, observability, and a dataset-driven evaluation harness.
Project status: Independent engineering project / prototype. Infrastructure actions and most telemetry are simulated locally; this is not a production SRE system.
LLM agents can reduce the amount of repetitive work involved in incident investigation, but giving a model direct access to infrastructure creates a significant safety problem: incorrect tool selection, malformed arguments, ambiguous targets, or prompt injection could result in destructive actions.
This project explores a safer architecture for AI-assisted incident response:
- Let the LLM interpret natural-language requests and select from predefined operational tools.
- Keep deterministic validation and execution outside the model.
- Separate read-only investigation from destructive operations.
- Require explicit approval for high-risk actions.
- Evaluate the agent's actual tool-selection trajectory rather than only its final response.
- Instrument the agent so its execution can be observed and debugged.
The goal is not autonomous production infrastructure management. The goal is to experiment with the engineering boundaries required to make LLM-powered operational workflows more reliable and controllable.
User / Operator
|
+--------------+--------------+
| | |
v v v
Web Frontend Slack CLI / Voice
| | |
+--------------+--------------+
|
| HTTP / WebSocket
v
+--------------------+
| FastAPI API |
| |
| Agent orchestration|
| Voice processing |
| WebSocket events |
+---------+----------+
|
v
+--------------------+
| LLM Agent |
| |
| Tool selection |
| Argument generation|
+---------+----------+
|
v
+--------------------+
| Tool / Schema Layer|
| |
| Pydantic validation|
| Allowed values |
| Ambiguity checks |
| Tool classification|
+---------+----------+
|
+------+------+
| |
v v
Read / Diagnostic Destructive
Tools Tools
| |
| +----v-----+
| | RBAC / |
| | Guardrail|
| +----+-----+
| |
| Human approval
| |
+-------+-------+
|
v
+-------------------+
| Execution Layer |
| |
| Simulation Engine |
| PostgreSQL |
| GitHub |
| Slack |
+---------+---------+
|
+----------+----------+
| |
v v
Incident / Telemetry Agent Events
| |
+----------+----------+
|
v
WebSocket / UI
Agent execution
|
v
OpenTelemetry / OpenInference
|
v
Arize Phoenix
- A user provides an incident request through the web interface, Slack, CLI, or voice.
- The request is sent to the FastAPI backend.
- The LLM receives the conversation context and a predefined set of operational tools.
- The model selects a tool and generates structured arguments.
- Pydantic schemas validate the arguments before execution.
- The system checks for invalid or ambiguous targets and prevents the agent from guessing operational resources.
- Read-only diagnostic tools can execute against the simulated environment.
- Destructive tools are classified separately and pass through authorization and confirmation guardrails.
- Approved actions are executed against the simulation or supported integrations.
- Results are returned to the agent and client.
- Agent execution and LLM activity can be observed through OpenTelemetry/Phoenix.
- The same API path can be exercised by the evaluation harness to test tool selection, arguments, and safety behavior.
OpsVoice includes a local simulation engine that creates a controlled SRE environment instead of connecting an autonomous agent directly to production infrastructure.
The simulated environment contains services such as:
frontend-nextjs
backend-fastapi
database-postgres
The simulation can introduce incident scenarios including:
- PostgreSQL connection-pool exhaustion
- Backend memory pressure / OOM conditions
- Frontend runtime or hydration failures
The simulator exposes changing telemetry and incident state so the agent has an environment to investigate and reason about.
A simplified incident flow looks like:
Normal system
|
v
Fault injection
|
v
Abnormal metrics / logs
|
v
Agent investigation
|
+---- metrics
+---- logs
+---- traces
+---- alerts
|
v
Hypothesis / remediation
|
v
Guarded action
|
v
Simulated recovery
The repository currently contains a 500-scenario evaluation dataset in eval_dataset.json.
Scenarios cover:
- metric retrieval
- log retrieval
- incident-state updates
- destructive-action confirmation
- structured argument constraints
- unknown or unauthorized services
- ambiguous requests
- prompt-injection attempts against operational tools
- rollback behavior
test_harness.py executes the scenarios against the running FastAPI API and evaluates the resulting agent behavior.
The current checks focus on:
-
Tool selection Whether the agent selected the expected operational tool.
-
Argument extraction Whether the generated arguments match the expected structured inputs.
-
Guardrail behavior Whether destructive actions correctly require confirmation.
The evaluation is intentionally trajectory-oriented: the objective is to test the agent's operational decision path rather than simply judging whether its final natural-language response sounds reasonable.
OpsVoice treats LLM output as untrusted intent rather than trusted executable commands.
Natural language
|
v
LLM decision
|
v
Structured tool call
|
v
Pydantic validation
|
v
Target / argument checks
|
+---- read-only action ------> simulated execution
|
+---- destructive action ----> authorization
|
v
human approval
|
v
guarded execution
The agent does not generate arbitrary shell commands or receive unrestricted infrastructure access.
Destructive operations are explicitly classified and require additional controls.
| Component | Purpose |
|---|---|
agent.py |
FastAPI application, WebSocket handling, LLM agent loop, memory, and API layer |
tools.py |
Operational tool definitions, Pydantic schemas, validation, guardrails, and tool execution |
simulation_engine.py |
Simulated services, telemetry, incidents, and recovery behavior |
voice_client.py |
CLI voice interaction using speech recognition and text-to-speech |
voice_utils.py |
Voice transcription and synthesis utilities |
slack_bot.py |
Slack-based incident interaction and approval workflow |
test_harness.py |
Dataset-driven evaluation runner and deterministic graders |
eval_dataset.json |
500 evaluation scenarios |
generate_evals.py |
Utility for generating evaluation scenarios |
chaos.py |
Fault-injection / chaos-testing utility |
patch.py |
Development utility for patching or modifying simulation behavior |
seed_db.py |
Database initialization / seed utility |
cli.py |
Command-line interface |
client.py |
API/client interaction utility |
frontend/ |
Next.js incident-response interface |
docker-compose.yml |
Local service orchestration |
requirements.txt |
Python dependencies |
The agent interacts with a predefined operational toolset rather than generating arbitrary shell commands or infrastructure commands.
This reduces the model's action space and makes tool calls:
- schema-validatable
- testable
- observable
- easier to authorize
- easier to audit
The LLM decides which supported operation it wants to perform, while deterministic application code controls how that operation can actually execute.
Tool arguments are represented using structured Pydantic models.
This provides a deterministic boundary between probabilistic model output and application logic.
The system can reject:
- invalid service names
- unsupported metric types
- invalid action parameters
- malformed arguments
- ambiguous targets
- out-of-range values
The model is therefore not trusted simply because it produced syntactically valid JSON.
The project deliberately uses a controlled local simulation for most infrastructure operations.
This makes it possible to experiment with:
- destructive actions
- rollback
- incident recovery
- fault injection
- evaluation
- failure scenarios
without giving an experimental LLM access to real production systems.
Read-only investigation and destructive remediation are treated differently.
For example:
query_metrics
fetch_logs
query_traces
can execute as diagnostic operations, while actions such as:
execute_rollback
restart_service
scale_replicas
kill_long_query
trigger_failover
grant_emergency_access
revoke_access
block_ip
are handled as higher-risk operations.
The architecture therefore follows:
Investigate freely
|
v
Propose remediation
|
v
Authorization / guardrail
|
v
Human approval
|
v
Execute
The model does not determine whether an operator is authorized to perform a destructive action.
Authorization is enforced by application logic and role checks.
This separates:
LLM decision
from:
actual authorization
The evaluation harness sends complete scenarios through the running API instead of only unit-testing individual helper functions.
This allows the project to evaluate behavior across the actual path:
User prompt
|
v
FastAPI
|
v
LLM
|
v
Tool selection
|
v
Argument validation
|
v
Guardrails
This is more representative of agent behavior than testing only isolated functions.
The project uses OpenTelemetry/OpenInference instrumentation with Arize Phoenix so that LLM and agent execution can be inspected.
This makes it possible to investigate questions such as:
- Which tool did the model select?
- What arguments did it generate?
- How long did the LLM call take?
- What was the execution path?
- Where did an agent trajectory fail?
Observability is treated as part of the agent architecture rather than an afterthought.
The same underlying agent can be accessed through different interfaces:
Web UI
Slack
CLI
Voice
The operational logic remains in the backend instead of being duplicated across each interface.
OpsVoice includes local voice interaction using speech-to-text and text-to-speech components.
The browser/CLI voice path follows the general pattern:
Microphone
|
v
Audio
|
v
Speech-to-text
|
v
Agent
|
v
Text response
|
v
Text-to-speech
|
v
Audio response
The project uses Faster-Whisper for local transcription and Piper/other configured TTS components for voice synthesis.
Voice is treated as another interface to the same operational agent rather than a separate conversational system.
OpsVoice can also operate through Slack.
The Slack workflow supports:
- natural-language incident requests
- agent responses
- voice-note transcription
- destructive-action approval
- role-aware operational actions
A high-risk action can result in an approval flow rather than immediate execution:
Slack request
|
v
Agent
|
v
Destructive action detected
|
v
Guardrail message
|
+---- Approve ----> execute
|
+---- Reject -----> cancel
chaos.py provides a small fault-injection mechanism for creating a long-running PostgreSQL operation.
The workflow is intended to demonstrate:
Fault
|
v
Database degradation
|
v
Detection
|
v
Alert
|
v
Agent investigation
This provides a bridge between the simulated incident environment and real database behavior without requiring a production cluster.
- Most infrastructure operations are simulated rather than connected to real Kubernetes or cloud infrastructure.
- The project is designed for local development and experimentation.
- The evaluation dataset currently contains 500 scenarios and but it still does not represent comprehensive production coverage.
- The current agent orchestration is intentionally lightweight and does not implement a fully autonomous long-running planning loop.
- Conversation memory is relatively simple and currently relies on persisted recent messages rather than a sophisticated incident-memory architecture.
- WebSocket connection and simulation state are primarily process-local, limiting horizontal scalability.
- Destructive-action execution would require stronger server-side action identity, approval state, and authorization checks before being considered production-ready.
- Voice interaction is not yet a fully streaming, low-latency conversational voice system.
- Infrastructure operations lack production-grade idempotency and durable execution state.
- The project does not claim production SRE reliability, autonomous remediation, or real-world incident-resolution outcomes.
A production version would need significantly stronger infrastructure and security boundaries, including:
- authenticated operator identity
- durable incident and action state
- immutable audit logs
- server-side approval records
- authorization checks at execution time
- action IDs and idempotency keys
- real Kubernetes/cloud integrations
- Prometheus/Grafana or equivalent telemetry integrations
- durable event streaming
- distributed WebSocket/session management
- secrets management
- provider fallback and model health monitoring
- rate limiting and abuse protection
- prompt-injection and tool-abuse defenses
- structured execution error handling
- incident-level state management
- comprehensive evaluation datasets
- regression testing across model/provider versions
- latency and cost monitoring
- continuous evaluation and observability
The key production principle would remain the same:
The LLM should propose operational intent, while deterministic systems enforce authorization, validation, execution policy, and safety.
Building an agent for operational workflows made it clear that the LLM is only one component of the system.
The harder engineering problems are around the model:
Tool design
|
v
Structured validation
|
v
State management
|
v
Authorization
|
v
Human approval
|
v
Execution
|
v
Observability
|
v
Evaluation
A capable model can select the correct action, but that alone does not make an operational agent reliable.
The most important lesson was to keep deterministic work outside the model wherever possible and treat model output as untrusted input that must pass through explicit interfaces and safety boundaries.
Independent prototype built for engineering experimentation and portfolio demonstration.
The project intentionally prioritizes architectural experimentation over production deployment. It demonstrates how an LLM-powered operational workflow can be structured around constrained tools, simulated infrastructure, evaluation, observability, and human-controlled remediation.