A LegionIO cognitive architecture extension for confabulation detection and confidence calibration. Confabulation is the cognitive phenomenon of producing plausible but false narratives to fill gaps — occurring when an agent's confidence in a claim exceeds the evidence supporting it. This extension tracks cognitive claims, monitors the confidence-evidence gap, and surfaces high-risk assertions before they influence downstream decisions.
Manages a registry of claims — assertions made by the agent with an associated confidence level and an evidence strength rating. The gap between these two values is the confabulation risk.
- Register claims as the agent makes assertions, capturing confidence and evidence strength at the time
- Verify claims when supporting evidence is confirmed
- Flag confabulation when a claim is later found to be false or unsupported
- Monitor calibration — the average alignment between confidence and evidence across all claims
- Surface high-risk claims — those where confidence significantly exceeds evidence (risk >= 0.6)
A periodic Decay actor fires confabulation_report every 300 seconds for ongoing monitoring.
require 'lex-confabulation'
client = Legion::Extensions::Confabulation::Client.new
# Register a claim made by the agent
claim = client.register_claim(
content: 'The database migration completed successfully',
claim_type: :factual,
confidence: 0.9,
evidence_strength: 0.4
)
# => { id: "uuid...", content: "...", claim_type: :factual,
# confidence: 0.9, evidence_strength: 0.4,
# confabulation_risk: 0.5, risk_label: :moderate,
# verified: false, confabulated: false, created_at: "..." }
claim_id = claim[:id]
# Verify a claim when evidence is confirmed
client.verify_claim(claim_id: claim_id)
# => { found: true, claim_id: "uuid...", verified: true }
# Flag a claim as confabulation when found to be false
client.flag_confabulation(claim_id: claim_id)
# => { found: true, claim_id: "uuid...", confabulated: true }
# Retrieve all claims with high confabulation risk (risk >= 0.6)
client.high_risk_claims
# => { claims: [...], count: 1 }
# Full report across all claims
client.confabulation_report
# => { total_claims: 5, high_risk_claims: 1, verified_claims: 2,
# confabulated_claims: 1, confabulation_rate: 0.2,
# average_calibration: 0.75, overall_risk: 0.2, risk_label: :low }
# Engine summary
client.confabulation_status
# => { engine: { claim_count: 5, confabulation_rate: 0.2, average_calibration: 0.75 } }bundle install
bundle exec rspec
bundle exec rubocopMIT