Model without recorder#15
Merged
Merged
Conversation
When cloned panels lack HA recorder data, the simulator will project one year of historical data using existing circuit/BESS/EVSE config and modulation infrastructure, stored in a companion SQLite file mirroring HA's recorder schema.
Adds: configurable lookback in RecorderDataSource, record format spec for SqliteHistoryProvider, deterministic noise model, timezone handling, BESS SOE tracking constraints, async generation boundary, weather fallback, app.py/config_types.py change notes.
_load_recorder_data now checks for a companion SQLite history DB when no HA client is configured, using SqliteHistoryProvider with a 365-day lookback. A new _resolve_history_db static method resolves the DB path via explicit panel_config.history_db or the <config_stem>_history.db convention.
…bases Generates a year of synthetic power statistics from panel config YAMLs, writing hourly rows (statistics) and 5-minute rows (statistics_short_term) into a companion _history.db file compatible with SqliteHistoryProvider. Uses solar, weather, HVAC seasonal, time-of-day, cycling, and monthly factor modulation with deterministic per-row noise seeded from (panel_serial, circuit_id, start_ts) for reproducible output.
Add TestEndToEndRoundTrip with two async tests that exercise the full generate → load → query pipeline. The lookback window is computed dynamically from the fixed anchor timestamp so the tests remain valid regardless of when they run. Coverage assertion matches the actual hourly window (365 - 10 short-term days = 355 days).
- Move datetime imports to module level in sqlite_history.py - Wrap synchronous SQLite calls in asyncio.to_thread via _sync_get_statistics - Add --years CLI flag to history_generator.py (default 1); thread through to generate() - Add BESS schedule handling in _compute_power_at: charge/discharge/idle hours respected before consumer/producer logic
…d After modeling passes
When no user modifications exist on the battery template, the After modeling pass now uses recorder data directly (matching Before exactly) instead of routing through BSEE where SOE depletion corrupted the trace. Also fix _integrate_energy to use abs(power_w) so SOE tracking handles both recorder-signed and synthetic-positive power correctly when BSEE is active after user modification.
restore_recorder required a panel_source.recorder_map entry to find the recorder_entity — template clones have no panel_source so the restore always failed silently. Fall back to the template's own recorder_entity so clicking SYN correctly clears user_modified and re-enables recorder replay for any circuit.
Reverts the skip-BSEE-when-not-user_modified approach — the After chart must always apply the BSEE schedule so users see the effect of their battery configuration. BSEE now zeros battery power during idle hours so the schedule is authoritative for all three states (charge, discharge, idle), not just when hitting SOE bounds.
Clone, Start, Stop, and Restart handlers now load the acted-on config into the editor and set config_filter before redirecting. Previously, cloning a template left the editor on the old config, so the entity list and modeling view showed the wrong panel's circuits.
There was a problem hiding this comment.
Pull request overview
This PR adds an offline recorder-replay path by generating and consuming HA-compatible statistics in a companion SQLite database, enabling cloned panels (and modeling) to work without a live Home Assistant recorder.
Changes:
- Added
SyntheticHistoryGeneratorto create<config>_history.dbfiles (hourly + short-term statistics). - Added
SqliteHistoryProviderand wiredSimulatorAppstartup to load recorder data from HA first, then fall back to a companion SQLite DB. - Updated modeling/battery handling and dashboard clone/start flows to keep configs and modeling view in sync (plus added extensive tests and design docs).
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
src/span_panel_simulator/sqlite_history.py |
New SQLite-backed HistoryProvider implementation + schema constant. |
src/span_panel_simulator/history_generator.py |
New generator that writes HA-style statistics* tables for offline replay. |
src/span_panel_simulator/app.py |
Startup logic to select HA vs companion SQLite history DB; adds _resolve_history_db. |
src/span_panel_simulator/config_types.py |
Adds panel_config.history_db for explicit companion DB override. |
src/span_panel_simulator/dashboard/routes.py |
Generates companion DB on clone flows; editor auto-switch uses redirects. |
src/span_panel_simulator/dashboard/config_store.py |
Recorder restore fallback to template’s own recorder_entity. |
src/span_panel_simulator/engine.py |
Modeling output now distinguishes battery before/after and uses recorder-signed battery in “Before”. |
src/span_panel_simulator/bsee.py |
Battery state resolution/integration adjustments (idle enforcement + signed-power integration). |
src/span_panel_simulator/dashboard/templates/partials/modeling_view.html |
Modeling chart labels updated; adds battery-before series; improves fetch error handling. |
tests/test_sqlite_history.py |
Unit tests for SQLite provider querying/filtering behavior. |
tests/test_history_generator.py |
Unit tests for generator schema/row counts/determinism and solar day/night pattern. |
tests/test_sqlite_app_integration.py |
Round-trip + _resolve_history_db + end-to-end generate→load→query tests. |
tests/test_history.py |
Protocol conformance check for SqliteHistoryProvider. |
docs/superpowers/specs/2026-03-26-synthetic-history-generation-design.md |
Design spec documenting architecture and schema. |
docs/superpowers/plans/2026-03-26-synthetic-history-generation.md |
Implementation plan capturing task breakdown and testing strategy. |
…seed - Fix log line that always counted len(circuits_to_generate) instead of actual derived mappings — use an integer counter instead of bool guard. - Replace innerHTML XSS sink with textContent when rendering fetch errors in the modeling view. - Replace Python's salted hash() with hashlib.sha256 for the weather factor seed so synthetic history is deterministic across processes. Precompute the seed per-circuit in _generate_rows to avoid recomputing SHA-256 on every timestamp.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.