Mnemolis v3.56.0

[3.56.0]

Added — the history source: time-series memory for the house (opt-in, off by default)

The second Fable Capability Extension to ship (Design Doc 5, all three of its milestones landed together in this release). The Snapshot Engine polls Home Assistant every few minutes, formats the result for diffing, and throws the numbers away — every poll carried a live office-CO2 reading, a room temperature, a battery percentage, and the moment the diff was computed they were gone. The new history source keeps them. Where changes answers "what's different since X," history answers "what were the actual values over X": highs, lows, averages, counts, and trends over real recorded samples. See the new History-and-Trends wiki page.

Sampling never degrades what exists. The sampler (app/history.py) piggybacks on the same home_assistant._get_states() fetch snapshot_ha already makes — enabling the feature adds zero HA load. Numeric sensors passing the device-class filter (HISTORY_DEVICE_CLASSES, plus HISTORY_EXTRA_ENTITIES allowlist / HISTORY_EXCLUDE_ENTITIES denylist) land in WAL-mode /app/data/history.db — one metric_samples row per sensor per tick, plus a metric_catalog upsert that tracks name/area/unit drift while preserving first_seen. One derived pair per tick parses the existing uptime text into uptime.services_up/uptime.services_total; forecast is deliberately not sampled (a prediction recorded as history would be quiet dishonesty). Retention is explicit and batched: HISTORY_RETENTION_DAYS (90) pruned in the sampler tick via a DELETE … LIMIT loop so a large purge never balloons the WAL. battery is in the default class set on purpose — draining-battery questions for free; the button-cell catalog noise is the exclude-list's job, not a reason to drop the class.

Deterministic answers, honest coverage. The routable adapter (app/sources/history.py) resolves three things by pattern against closed vocabularies — metric (friendly-name on word boundaries, longest-first → area+class → bare class → area, asking rather than guessing on ties), window (see below), aggregation (min/max/avg/trend/summary; counts go to the events leg) — with no LLM anywhere in the aggregation path. Every answer leads with the same Low/High/Average/Now line with real units and local clock times. The single most important honesty property: when the requested window reaches back further than the oldest recorded sample, the answer states its actual coverage — "this week (only the past 3 days of recorded data)" — because history only knows what it recorded while it was on. A trend is a claim, so it's gated twice: HISTORY_TREND_MIN_SAMPLES (12) below which no direction is ever asserted, and HISTORY_TREND_MIN_DELTA (0.1) — a per-window, unit-free noise floor the fitted least-squares change must clear, or the finding is "roughly flat" (a finding, not a shrug). Pure-Python slope, the temporal miner's no-numpy discipline.

Events are not duplicated. Count/when questions ("how many times did the front door open today" → 7 openings today, most recently 9:20 PM.) read the temporal feature's temporal_events table directly, read-only (?mode=ro) — exactly one thing in Mnemolis extracts events from raw state, and it's the temporal miner. Consequence stated rather than hidden: event counts require TEMPORAL_PATTERN_DETECTION_ENABLED=true, and with it off the answer says event history is unavailable instead of returning a confident zero.

The shared window owner. Both changes and history resolve "today," "last night," "since Monday" into a time window, so that logic was extracted to a single owner: timeutil.resolve_window(), returning a real bounded (start, end, label) plus the hours-since float. _resolve_changes_hours() is now a thin adapter over it, byte-identical for every phrase it previously resolved (the canonical float is carried on the result, never re-derived from the timedelta, so FP re-rounding can't drift it; the 24 existing regression tests pinning exact values pass unchanged). New bounded phrases — "last night," "yesterday morning/afternoon/evening," "over the weekend," "since {weekday}," "last N days" — enrich both sources; time-of-day resolution is now DST-correct (aware local arithmetic instead of naive hour subtraction).

Routing, endpoints, health. history registers in SOURCE_MAP/SOURCE_DESCRIPTIONS/CACHE_TTL (CACHE_TTL_HISTORY_SECONDS, 300 — the data can't advance faster than the sampler) with no fallback chain (no other source can answer recorded history) and value/trend/count intent triggers deliberately non-overlapping with changes (a census test pins the disjointness). Two new endpoints under require_api_key: GET /history/metrics (the catalog with per-metric counts and coverage — the /areas analogue) and GET /history/series?metric=&hours= (raw (value, ts) series — the read a future NOC/device-registry dashboard renders sparklines from, shipped now so that dashboard needs no follow-up release). /health gains a history block: disabled when off, else ok/stale/never_ran plus metrics_tracked, samples_24h, db_mb, and quiet_sensors — a catalog entry gone silent (a Mnemovox node dropping off) becomes visible for free. history.db joins the backup set (seven data files now).

Two documented deviations from the design doc as written:

• Bare "history" is not an intent trigger — it collides with encyclopedic phrasing ("tell me about the history of ancient rome" must stay kiwix; a regression test pins it). The specific value/trend/count phrasings carry the intent without the false positive.
• "this week" stays a rolling 168h window rather than the doc's local-Monday semantics. changes routes on a substring trigger then resolves the whole query, so redefining the phrase would silently change the shipped, regression-pinned behavior of "what changed this week" — the doc's own byte-identity constraint wins over its nicer calendar semantics. Recorded in resolve_window()'s header so it isn't re-litigated.

Changed

• Version bumped to 3.56.0. Test suite: 1645 passing (from 1518 at v3.55.2 — 127 new: 63 engine, 33 adapter, 14 shared-window incl. DST and byte-identity pins, 9 routing/census, 5 endpoint/health, 3 Hypothesis properties incl. constant-series-is-never-a-direction), ruff clean. New config: HISTORY_ENABLED, HISTORY_SAMPLE_INTERVAL_MINUTES, HISTORY_RETENTION_DAYS, HISTORY_DEVICE_CLASSES, HISTORY_EXTRA_ENTITIES, HISTORY_EXCLUDE_ENTITIES, HISTORY_TREND_MIN_SAMPLES, HISTORY_TREND_MIN_DELTA, HISTORY_STALE_GRACE_MULTIPLIER, CACHE_TTL_HISTORY_SECONDS. Wiki: new History-and-Trends page; Home, Sources, Configuration-Reference, Health-and-Observability, Roadmap, and Backup-and-Restore updated. README sources/config tables and docker-compose.example.yml updated.
• One incidental robustness fix found by the new adapter tests: catalog friendly-name matching uses word boundaries (the _detect_area discipline), so a short name like AC can't false-match inside an unrelated word.

Fixed — post-implementation audit, same release (none of this shipped broken; all caught before tag)

A deliberate adversarial pass over the freshly written feature, before release, found eleven real issues — the most important being that the implementation violated the design doc's own priority-1 constraint:

• The sampler wasn't actually piggybacking. Constraint #1 says "the same _get_states() fetch — zero additional HA API load"; the first cut registered its own scheduler job that re-fetched _get_states() (duplicating what snapshot_ha had just retrieved — the exact pattern the Full Audit Pass once removed from snapshot_ha itself, whose comment memorializes it), rendered the HA area-map template every tick, and did a live Uptime Kuma client login every 5 minutes when snapshot_uptime fetches identical text every 2. (The doc also says "register sample_metrics() on the scheduler," which conflicts with its own constraint #1; its explicit priority ordering says #1 wins.) Now: snapshot_ha()/snapshot_uptime() hand their already-fetched payloads to history.ingest_ha_states()/ingest_uptime_text() — history makes zero fetches of its own; the area map is cached, refreshed at most ~hourly, keeping the stale copy on refresh failure. HISTORY_SAMPLE_INTERVAL_MINUTES is removed (a knob that couldn't change when data arrives would be a lie); /health staleness references the real HA snapshot interval. Hook-level tests pin "exactly one states fetch per snapshot" and "a history failure cannot fail the snapshot."
• NaN/inf sensor states were stored. float("nan") parses, so a template sensor stuck on NaN would have silently poisoned every average and disabled every min/max it touched. _parse_float() now requires a finite value; property tests unaffected, new pins added.
• Bare "sun"/"sat" resolved as weekdays — "is the sun out today" became since Sunday (90h), hijacking the query's own "today". Weekday abbreviations now require a since/on/last/this prefix; full names still match bare.
• "last friday" asked on a Friday meant this morning, not a week ago. last + today's weekday now goes 7 days back; bare/since forms keep meaning today's own 00:00.
• "this afternoon" was lumped into the 18:00 evening branch — a 1 PM afternoon question looked back to yesterday evening under a mislabeled window. Now anchors at noon with its own label (no changes byte-identity concern: the pre-3.56.0 resolver had no afternoon branch, so no pinned value existed).
• Event verbs matched as raw substrings — "clock" and "blocked" matched lock, so "how many times did the clock chime" counted lock events. Word-boundary matching throughout; plural verb forms added.
• Bare "down" made metric questions count outages — "has the CO2 gone down today" answered with service outages. Bare "down" removed; a metric-class guard in query_is_event() routes any query naming a numeric metric to the metric leg (one deliberate exception: battery + battery-alert phrasing is genuinely an event count). The class vocabulary moved into the engine (app/history.py) so this guard doesn't create an import cycle; the adapter re-exports it.
• Bare "how many" dragged metric questions onto the events leg — "how many degrees was it last night" got "couldn't identify which kind of event." Count phrases are now how many times / how often only.
• Doc-proposed intent triggers were greedy under substring routing — bare trend hijacked "latest AI trends", over the past hijacked "how has the climate changed over the past century", and present/perfect count forms hijacked "how many times has brazil won the world cup" / "how often should i water succulents". Triggers narrowed to past-tense household forms (how many times did/was/were, how often did/was/were, …); trend questions reach the source via "been rising/falling" or LLM routing. Regression pins added for every false positive above.
• Time-only "most recently" over multi-day windows — "most recently 9:20 PM" across a 7-day count doesn't say which day. Timestamps past a ~26h window now carry the local date (Jul 3, 9:20 PM), on both the events leg and the metric summary's Low/High times.
• Dead code and a telemetry gap — an empty if/pass left in _trend_line() removed; the ambiguous-metric path now emits its explanation-chain event like every other exit.
• /history/series accepted nonsensical windows — negative/zero hours silently produced an inverted (empty) window and an unbounded value invited a 90-day full-table scan per request; now validated 1 ≤ hours ≤ 2160 (422 outside).
• Two cosmetic honesty bugs caught by the final smoke run — a brand-new metric's coverage disclosure read "only the past 0 hours of recorded data" (now "under an hour"), and a one-sample summary read "(1 samples)".

Changed (audit delta)

• Test suite: 1683 passing (from 1645 pre-audit — 38 new regression pins), ruff clean. Config removed: HISTORY_SAMPLE_INTERVAL_MINUTES. README, docker-compose.example.yml, and the History-and-Trends / Configuration-Reference / Health-and-Observability wiki pages corrected to describe the real ingest-hook mechanism, including an honest account of the first cut's constraint violation.

Fixed — field findings from the first real deployment (same release)

Five more, surfaced within the first fifteen minutes of the feature running on real MiniDock data — every one a behavior no test pinned because no test data had a device named after a class word, a partially-covered area, or a rolling window crossing midnight:

• A sensor literally named "Temperature" hijacked class queries — "average temperature today" silently answered with one mystery device (99.9 °F) out of five temperature sensors, because friendly-name matching runs before class resolution. Names that ARE bare class words are now skipped in the name-match step; the class logic owns the question and asks when several candidates exist.
• A named area with no matching sensor silently substituted another room — "office co2" answered with the living-room sensor, no disclosure. The fall-through is now allowed only when the class match is unambiguous, and it announces itself: "No carbon dioxide sensor recorded in office — showing LivingRoomLilyGo Room CO2." Several candidates → it asks.
• "how cold did it get last night" routed to history and then dead-ended unresolved — "cold" wasn't in the class vocabulary its own intent trigger implies. cold/hot/warm/freezing/chilly now resolve to temperature, and _detect_class switched to word-boundary matching while gaining short words (no "hot" inside photos, no "temp" inside attempts).
• Rolling windows crossing local midnight showed undated times — "1 opening today, most recently 5:25 PM" read as impossible at 2 PM; the opening was yesterday 5:25 PM, legitimately inside the pinned rolling-24h "today" window. Timestamps are now date-qualified whenever the window crosses local midnight, not just past 26 hours; bounded single-day windows ("yesterday") stay clean.
• "(only the past under an hour of recorded data)" — the audit's own sub-hour cosmetic fix produced broken grammar in the coverage template; now "(only under an hour of recorded data)".
• The ambiguity ask listed unpickable options — a candidate whose friendly name IS a bare class word (a real outdoor sensor named just "Temperature") can't be chosen by that name, since the name-collision rule skips it on the follow-up too. Such candidates (and duplicate names) are now area-qualified in the listing — "Temperature (outside)" — teaching the phrasing that actually resolves.

Test suite after field fixes: 1694 passing (11 new field-finding pins), ruff clean.