74 add persistence backend sqlite for long term record history

[lxsaah]

Summary

Adds an optional persistence layer to AimDB that stores long-term record
history in SQLite. Persistence is implemented as a buffer subscriber
(tap_raw) — just like .tap() — keeping it fully within AimDB's existing
producer–consumer architecture. No new write paths or special-casing in the
core.

---

Motivation

Validation messages arrive roughly once per hour per city. Combined with
simulatable: false, this creates a UX problem: the AccuracyPanel is empty on
page refresh, on first visit, and permanently in demo mode. The in-memory ring
buffer cannot solve this because it clears on restart.

Solution: persist validation records to SQLite and query "latest per city"
on page load.

---

New Crates

aimdb-persistence

Trait definitions and extension traits that wire persistence into the AimDB
lifecycle without touching aimdb-core:

Component	What it adds
PersistenceBackend trait	Pluggable async interface (store, query, cleanup)
AimDbBuilderPersistExt	.with_persistence(backend, retention) on AimDbBuilder
RecordRegistrarPersistExt	.persist("record::name") on RecordRegistrar
AimDbQueryExt	.query_latest() / .query_range() / .query_raw() on AimDb<R>
PersistenceState	Typed state stored in Extensions TypeMap, shared between subscriber and query time
PersistenceError	NotConfigured, Backend(String), BackendShutdown, Serialization

aimdb-persistence-sqlite

Concrete SQLite backend:

• Dedicated "aimdb-sqlite" OS thread holds the rusqlite::Connection — the async executor is never blocked
• WAL mode (PRAGMA journal_mode = WAL) allows concurrent readers
• mpsc::sync_channel(64) for backpressure; tokio::sync::oneshot for async reply
• prepare_cached() on all hot paths (INSERT, DELETE, SELECT)
• Schema applied synchronously in ::new() — no Tokio runtime required at construction time
• ROW_NUMBER() OVER (PARTITION BY record_name ORDER BY stored_at DESC) window query for efficient top-N-per-group without full table scans
• Graceful shutdown when all SqliteBackend clones are dropped

---

Changes to aimdb-core

File	Change
extensions.rs	New Extensions TypeMap (HashMap<TypeId, Box<dyn Any + Send + Sync>>), exposed on AimDbBuilder and AimDb
builder.rs	on_start() hook for spawning tasks after build(); extensions() / extensions_mut() accessors; extensions field moved into AimDbInner
typed_api.rs	extensions() accessor on RecordRegistrar so .persist() can retrieve PersistenceState
remote/handler.rs	record.query AimX protocol handler delegating to a type-erased QueryHandlerFn stored in Extensions

---

API

use std::sync::Arc;
use std::time::Duration;

use aimdb_core::AimDbBuilder;
use aimdb_persistence::{AimDbBuilderPersistExt, AimDbQueryExt, RecordRegistrarPersistExt};
use aimdb_persistence_sqlite::SqliteBackend;

let backend = Arc::new(SqliteBackend::new("./data/history.db")?);

let mut builder = AimDbBuilder::new()
    .runtime(runtime)
    .with_persistence(backend, Duration::from_secs(7 * 24 * 3600));

builder.configure::<Accuracy>("accuracy::vienna", |reg| {
    reg.persist("accuracy::vienna");
});

let db = builder.build().await?;

// Latest 10 per matching record
let latest: Vec<Accuracy> = db.query_latest("accuracy::*", 10).await?;

// All rows in a time range (None = no per-record cap)
let range: Vec<Accuracy> = db.query_range("accuracy::vienna", start_ms, end_ms, None).await?;

// Capped time range
let capped: Vec<Accuracy> = db.query_range("accuracy::*", start_ms, end_ms, Some(100)).await?;

---

Key Design Decisions

• Zero coupling — aimdb-core has no dependency on persistence types; state flows through the Extensions TypeMap
• Runtime-agnostic subscriber — .persist() requires only T: Serialize; .with_remote_access() is not needed
• query_range has an explicit limit_per_record: Option<usize> — opt-in unbounded (None) vs capped (Some(n)), preventing accidental OOM on high-frequency records
• Retention via on_start() — a cleanup task runs once at startup then every 24 hours; cleanup errors are always surfaced (via tracing::warn! or eprintln!)
• record.query AimX protocol — aimdb-core stores a type-erased QueryHandlerFn registered by with_persistence(); the handler defaults to limit_per_record = Some(1) when no explicit limit is provided by the client

---

Review Fixes Applied

Two issues identified during review were fixed before merge:

1. Lossy i64 → u64 cast on reads (aimdb-persistence-sqlite/src/lib.rs)

   • Changed row.get::<_, i64>(2)? as u64 to row.get::<_, i64>(2).map(|v| v.max(0) as u64)?
   • Negative values (from DB corruption or external tampering) now clamp to 0 instead of wrapping to u64::MAX

2. Unbounded query_range (aimdb-persistence/src/query_ext.rs)

   • Added limit_per_record: Option<usize> as a 4th parameter to both the trait and impl
   • Callers must now explicitly opt into unbounded results via None

---

Test Plan

• 4 unit tests in aimdb-persistence-sqlite: store/query, time-range, retention cleanup, SQL LIKE pattern escaping
• cargo build — all affected crates
• cargo test — 4/4 pass, 0 failures
• cargo clippy -- -D warnings — zero warnings

[Copilot]

Pull request overview

Adds an optional, pluggable persistence layer to AimDB that stores long-term record history (via a buffer-subscriber) and enables historical queries, with a concrete SQLite backend and core support for type-erased extensions and remote query delegation.

Changes:

• Added new crates aimdb-persistence (traits + builder/record/query extension traits) and aimdb-persistence-sqlite (SQLite backend using a dedicated writer thread).
• Extended aimdb-core with an Extensions TypeMap, an on_start() lifecycle hook, and an AimX record.query handler delegating to a type-erased query function stored in extensions.
• Updated workspace/build tooling (workspace members + Makefile) and added design/README/changelog documentation.

Reviewed changes

Copilot reviewed 23 out of 24 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
docs/design/022-M10-persistence-backend.md	Adds a full design proposal/spec for persistence + query APIs.
aimdb-persistence/src/backend.rs	Introduces PersistenceBackend, StoredValue, and QueryParams.
aimdb-persistence/src/builder_ext.rs	Adds .with_persistence() wiring, retention task, and AimX query handler registration.
aimdb-persistence/src/error.rs	Defines PersistenceError for persistence operations.
aimdb-persistence/src/ext.rs	Adds .persist() record-registration extension (subscriber that stores values).
aimdb-persistence/src/lib.rs	Crate entrypoint + re-exports for persistence API surface.
aimdb-persistence/src/query_ext.rs	Adds AimDbQueryExt (query_latest, query_range, query_raw).
aimdb-persistence/README.md	Documents installation, usage, API, and backend implementation guidance.
aimdb-persistence/Cargo.toml	Declares new crate, features, and dependencies.
aimdb-persistence/CHANGELOG.md	Adds changelog for the new crate.
aimdb-persistence-sqlite/src/lib.rs	Implements SQLite backend actor thread, SQL schema, queries, and tests.
aimdb-persistence-sqlite/README.md	Documents runtime requirements, architecture, and API for SQLite backend.
aimdb-persistence-sqlite/Cargo.toml	Declares SQLite backend crate + deps (rusqlite bundled, tokio sync).
aimdb-persistence-sqlite/CHANGELOG.md	Adds changelog for the new SQLite backend crate.
aimdb-core/src/extensions.rs	Adds Extensions TypeMap support to core.
aimdb-core/src/builder.rs	Wires Extensions + on_start() into builder + build lifecycle.
aimdb-core/src/typed_api.rs	Exposes RecordRegistrar::extensions() for external extensions like persistence.
aimdb-core/src/remote/mod.rs	Re-exports the new query-handler types for remote delegation.
aimdb-core/src/remote/handler.rs	Adds AimX record.query method delegating to extension-registered handler.
aimdb-core/src/lib.rs	Exports Extensions module/type from aimdb-core.
Cargo.toml	Adds new persistence crates to workspace members.
Makefile	Adds build/test/clippy/doc targets for new crates; includes them in fmt loops.
_external/embassy	Updates submodule pointer.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.