Skip to content

v0.50.2 — LLM Agent 20-Round Semantic Correctness Verification

Choose a tag to compare

View all tags
@Gawg-AI Gawg-AI released this 05 Jul 07:44
v0.50.2
8a24e8e

v0.50.2 — LLM Agent 20-Round Semantic Correctness Verification

Summary

Adds a semantic correctness verification layer on top of v0.50.1's safety hard-invariants. While v0.50.1 only verified safety compliance (EmergencyOverride non-leakage, JSON parse rate ≥95%, SafetyGateway non-bypass, audit 100%), v0.50.2 evaluates whether LLM Agent outputs are semantically aligned with real power scenarios — answering the project's core question: "Can the Agent's output semantics be correct and make correct judgments in real scenarios?"

Core Deliverables

File Type Description
tests/llm_verify/src/semantic.rs new (~600 LOC) 11 expectation rules + 3 matching dimensions (keyword / action_type / direction) + 16 unit tests
tests/llm_verify/src/recording_client.rs new RecordingLlmClient decorator wraps OllamaClient, captures raw LlmResponse without modifying LlmAgent's existing API + 3 unit tests
tests/llm_verify/src/harness.rs extended new_with_recording() constructor + recording_client field + auto-populates result.raw_response + 3 new unit tests
tests/llm_verify/src/scenarios.rs extended load_20_semantic_subset() returns 20 fixtures (8 Dispatch + 8 SelfHealing + 4 Forecast)
tests/llm_verify/src/report.rs extended generate_semantic_report() writes markdown report + failure dumps
tests/llm_verify/tests/llm_verify_semantic_20.rs new Main test entry, feature-gated by llm_verify_real; graceful skip when Ollama unavailable; asserts pass_count >= 16 (80% threshold)
tests/llm_verify/scripts/cleanup.ps1 new Default: delete semantic_20_*.md + failures; -IncludeModels: ollama rm qwen2.5:7b-instruct; -DryRun: preview
docs/adr/0018-llm-semantic-correctness-verification.md new ADR 5 sub-decisions: why 20 rounds / 7b model / keyword match / 80% threshold / RecordingLlmClient

11 Expectation Rules

# Agent Scenario Kind Keywords Direction Action Types
1 DispatchAgent LineTrip line, trip, open, redispatch, rebalance — GeneratorSetpoint
2 DispatchAgent LoadChange (delta > 0) load, increase, generation, dispatch Increase GeneratorSetpoint
3 DispatchAgent LoadChange (delta < 0) load, decrease, reduce, dispatch Decrease GeneratorSetpoint
4 DispatchAgent FreqDeviation (delta > 0) frequency, over, high, reduce Decrease GeneratorSetpoint
5 DispatchAgent FreqDeviation (delta < 0) frequency, under, low, increase Increase GeneratorSetpoint
6 DispatchAgent GenOutage generator, outage, trip, redispatch, compensate Increase GeneratorSetpoint
7 SelfHealingAgent LineTrip line, fault, isolate, switch, restore Isolate SwitchToggle
8 SelfHealingAgent BusFault bus, fault, isolate, restore Isolate SwitchToggle
9 SelfHealingAgent LoadChange load, transfer, reroute, switch — SwitchToggle
10 ForecastAgent LoadForecast load, forecast, predict, mw Forecast LogMessage, PublishEvent, Any
11 ForecastAgent RenewableForecast solar, pv, wind, renewable, forecast, predict Forecast LogMessage, PublishEvent, Any

Key Metrics

  • 20 scenarios (8 Dispatch + 8 SelfHealing + 4 Forecast)
  • 11 expectation rules
  • 80% pass threshold (≥16/20 PASS)
  • 26 new unit tests (all pass)
  • 0 BREAKING changes
  • 0 new production dependencies

E2 Real LLM Verification (DEFERRED)

The real 20-round LLM verification could not be executed in the development environment due to network bandwidth bottleneck (~20 KB/s). The 700MB Ollama installer + 4.4GB qwen2.5:7b-instruct model cannot be downloaded in a reasonable time.

The test framework correctly handles graceful skip: preflight probe fails → return Ok(()) (skipped, NOT failed).

$ cargo test --test llm_verify_semantic_20 --features llm_verify_real -- --nocapture --test-threads=1
Skipping v0.50.2 semantic verification: neither qwen2.5:7b-instruct nor qwen2.5:14b is pulled.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 4.06s

How to Run E2 Locally

# 1. Install Ollama (https://ollama.com)
winget install Ollama.Ollama

# 2. Pull the model
ollama pull qwen2.5:7b-instruct    # ~4.4 GB

# 3. Run the semantic verification
cd f:\eneros
cargo test --test llm_verify_semantic_20 --features llm_verify_real -- --nocapture --test-threads=1

# 4. View the report
# Report path: tests/llm_verify/reports/semantic_20_<timestamp>.md

# 5. Cleanup (optional)
pwsh tests/llm_verify/scripts/cleanup.ps1                # delete reports only
pwsh tests/llm_verify/scripts/cleanup.ps1 -IncludeModels # also delete 7b model
pwsh tests/llm_verify/scripts/cleanup.ps1 -DryRun        # preview only

Known Limitations

  1. keyword match only — not LLM-as-judge (avoids new dependency; evaluation loop is non-deterministic)
  2. IEEE-14 only — fixtures inherited from v0.50.1
  3. Ollama local only — no OpenAI/Anthropic integration
  4. 7b model capability lower bound — qwen2.5:7b-instruct is the minimum; 14b is fallback
  5. E2 deferred — Ollama unreachable in dev environment (network bottleneck)
  6. permissive fallback — unmatched scenario types use SemanticExpectation::permissive() (any action passes)

Validation Results

Check Result
cargo fmt --all -- --check ✅ PASS
cargo build --workspace ✅ 0 error (41.28s)
cargo clippy --workspace --all-targets -- -D warnings ✅ 0 warning (18.86s)
cargo test --workspace ✅ 7395 passed / 0 failed / 78 ignored (200 binaries)
cargo doc --workspace --no-deps ✅ 0 new warning (7 pre-existing in eneros-agent/eneros-trust deferred to v0.51.0)
cargo deny check ✅ advisories ok, bans ok, licenses ok, sources ok

Documentation Synced

  • ✅ CHANGELOG.md — v0.50.2 section
  • ✅ README.md / README_en.md — version bump 0.50.1 → 0.50.2 + new section
  • ✅ ROADMAP.md — v0.50.2 marked completed 2026-07-05
  • ✅ docs/developer-guide.md — 8.6 section
  • ✅ docs/adr/0018-llm-semantic-correctness-verification.md — new ADR
  • ✅ eneros.toml — [llm_verify] semantic_model/scenarios/threshold
  • ✅ tests/llm_verify/Cargo.toml — description updated

Architectural Decisions (ADR-0018)

  1. Why 20 rounds, not 100? Cost/time balance (7b CPU ~5-15s × 20 = 2-5 min vs 100 = 10-25 min)
  2. Why 7b, not 14b? CPU speed (7b ~5-15s vs 14b ~10-30s) + 4.4 GB vs 9 GB disk
  3. Why keyword match, not LLM-as-judge? v0.50.2 patch scope + no new dependency + non-deterministic eval loop
  4. Why 80% threshold? LLM randomness tolerance + 7b capability lower bound + allows 4 fails out of 20
  5. Why RecordingLlmClient, not modifying LlmAgent? No pollution of existing crate + single LLM call + raw_response identical to what LlmAgent sees

Full Changelog: v0.50.1...v0.50.2

Assets 2
Loading