LLM Integration en

中文 | English

LLM Integration & Verification

Maintained version: v0.51.2 ｜ Last updated: 2026-07-06

EnerOS provides the LlmClient trait abstraction and LlmAgent decorator in crates/eneros-ai, supporting OpenAI / Anthropic / Ollama backends, and completed the real-scenario verification framework in v0.50.1 / v0.50.2. This page summarizes the integration interface and verification methodology; for full decisions see ADR-0017 and ADR-0018.

LlmClient trait

#[async_trait]
pub trait LlmClient: Send + Sync {
    async fn complete(&self, prompt: &str) -> Result<String, LlmError>;
    fn model_name(&self) -> &str;
}

Implementation	crate module	Backend	Default model
OpenAiClient	eneros-ai::openai	OpenAI API	gpt-4o-mini
AnthropicClient	eneros-ai::anthropic	Anthropic API	claude-3-5-sonnet
OllamaClient	eneros-ai::ollama	Local Ollama	qwen2.5:7b-instruct
MockLlmClient	eneros-test-utils	Test stub	—

LlmAgent Decorator

pub struct LlmAgent<T: LlmClient> {
    client: T,
    prompt_template: PromptTemplate,
    parser: LlmDecisionParser,
}

impl<T: LlmClient> LlmAgent<T> {
    pub fn new(client: T, template: PromptTemplate) -> Self { /* ... */ }

    pub async fn decide(&self, ctx: &DecisionContext) -> Result<Vec<AgentAction>> {
        let prompt = self.prompt_template.render(ctx);
        let response = self.client.complete(&prompt).await?;
        self.parser.parse(&response)
    }
}

Prompt Templates

crates/eneros-ai/src/prompts/ provides Prompt templates for 7 domain Agents:

Template	Purpose	Input context	Expected output
dispatch	Dispatch decisions	Load / generation / reserves	Unit commitment + output allocation
forecast	Load forecasting	Historical time-series / weather	Future 24h load curve
operation	Operation execution	Current topology / alarms	Operation sequence
planning	Operations planning	Maintenance schedule / load forecast	Day-ahead plan
trading	Market trading	Prices / load forecast	Bidding curve
self_healing	Fault self-healing	Fault location / topology	Isolation + restoration strategy
maintenance	Equipment maintenance	Equipment status / historical alarms	Maintenance recommendations

v0.50.1 100-Scenario Verification

tests/llm_verify/ provides 100 real power scenario preflight + mock execution framework:

tests/llm_verify/
├── fixtures/           # 100 scenario JSONs (IEEE-14 topology + faults + loads)
├── src/
│   ├── preflight.rs    # Scenario preflight (parameter validation)
│   ├── mock_executor.rs# Mock LLM executor
│   ├── harness.rs      # Test harness
│   ├── scenarios.rs    # Scenario loader
│   ├── metrics.rs      # C1-C4 metric calculation
│   └── report.rs       # Verification report generation
└── tests/
    └── llm_verify_100.rs

4 Hard Metrics

Metric	Meaning	Pass condition
C1	Action consistency (Agent action type matches scenario expectation)	≥ 80%
C2	Decision auditability (decision traceable in audit log)	100%
C3	Violation detection rate (decisions violating safety constraints are detected)	100%
C4	Latency compliance (decision latency within budget)	≥ 95%

Note: C1 excludes Self-Healing scenarios because SelfHealingAgent::handle_emergency legitimately produces EmergencyOverride from a deterministic pipeline.

v0.50.2 20-Round Semantic Verification

tests/llm_verify_semantic_20/ samples 20 scenarios from the 100, adding a semantic correctness framework:

tests/llm_verify_semantic_20/
├── src/
│   ├── semantic.rs     # 11 expectation rules + 3-dim matching
│   ├── recording_client.rs # RecordingLlmClient (records LLM responses)
│   └── report.rs       # Semantic report
└── tests/
    └── llm_verify_semantic_20.rs

11 Expectation Rules

Each scenario defines 11 expectations (e.g., "must disconnect faulted branch", "must isolate fault zone", "must restore non-faulted zone supply"), evaluated via 3-dim matching (keyword / numeric / topology).

RecordingLlmClient

Records real LLM responses to JSON files for offline replay and regression testing:

pub struct RecordingLlmClient<T: LlmClient> {
    inner: T,
    records: Vec<LlmRecord>,
}

Real LLM Calls (User Side)

The test framework defaults to MockLlmClient to skip real calls. Users with local Ollama + qwen2.5:7b-instruct can enable real verification:

# Start Ollama
ollama pull qwen2.5:7b-instruct
ollama serve

# Run 100-scenario real verification
cargo test --test llm_verify_100 --features llm_verify_real

# Run 20-round semantic verification
cargo test --test llm_verify_semantic_20 --features llm_verify_real

Without Ollama, tests gracefully skip via preflight probe (preflight_probe() → Ok(())), returning "1 passed; 0 failed".

Known Limitations (v0.51.2)

• Only local Ollama supported; cloud LLM real-call testing not integrated
• Only IEEE-14 topology; IEEE 30/118 not covered
• Only hard invariants (N-1 / thermal / voltage); soft constraints not covered
• Single run; no multi-run averaging
• InMemoryAuditSink does not have HMAC chain
• Token usage not captured
• Semantic matching is keyword-only; no semantic embedding
• 7b model is the lower bound; not validated on larger models
• Permissive fallback may mask some errors

Related Documentation

• ADR-0017 LLM Real-Scenario Verification
• ADR-0018 LLM Semantic Verification
• Agent Runtime — LlmAgent in the 7 Agents
• SafetyGateway — Compliance audit of LLM decisions
• Main repo crates/eneros-ai/ — source
• Main repo tests/llm_verify/ — verification framework