RFC 0001: Shared-plan coordination via an external registry sidecar (stateless EMHASS)

[OptimalNothing90]

RFC 0001: Shared-plan coordination via an external registry sidecar (stateless EMHASS)

Status: Draft (Issue-filed)
Discussion: #931
Upstream context: #824 (EV use-cases follow-up to #789)
Verification basis: audits/2026-05-31-emhass-evcc-coordination-synthesis.md (facts F1–F16, build menu N1–N7)
Author: OptimalNothing90
Date: 2026-06-02 · Revised: 2026-06-16

Revision note (2026-06-16): in-core registry → external sidecar

The first version of this RFC (preserved at the end of this document, collapsed) proposed
making EMHASS itself stateful, with a persistent registry living inside the core. In the
#931 discussion, LesIT1, who runs a multi-deferrable production setup, argued that the state
and its lifecycle belong outside EMHASS in a thin sidecar, leaving the core stateless and
recommend-only. That avoids the concurrency, atomic-write, and lifecycle-bug surface the
project has deliberately stayed away from, and it lines up with the #789 guard. This revision
adopts that. The registry moves out of EMHASS. The only thing EMHASS-core needs is a
machine-readable plan output, which is build item N1 in the coordination synthesis.
Everything else (registration, persistence, lifecycle, recurrence) lives in the sidecar.

Motivation

In a home with several flexible loads, they all compete for one scarce resource: PV surplus
and grid import/export headroom. Today each load is optimized on its own. evcc plans EV
charging by itself; a heat pump or hot-water tank runs on its own local surplus trigger;
EMHASS optimizes the home battery and its statically-configured deferrable loads. Nobody
allocates the scarce capacity across all of them. The result is first-come-first-serve
contention: a surplus appears, several loads jump on it independently, their combined draw
exceeds the surplus or a grid limit, loads trip or back off, and the system oscillates. It
happens even in summer, where surplus is finite at any given instant, and it gets worse in
the shoulder seasons.

EMHASS already solves the whole-house MILP that arbitrates this, and it already co-optimizes
competing deferrable loads inside a single call. Two things are missing, and neither is the
solve itself:

1. No persistent shared state across cycles. EMHASS is stateless, one request at a time.
   There is no persistent notion of "the flexible loads currently in play". A multi-source
   setup (evcc, a heat pump, HA automations, later LLM agents) has no shared plan to
   contribute to. Each source rebuilds and resubmits the whole problem every MPC cycle.
2. No machine-readable plan output. /action/* returns a plain-text acknowledgement; the
   schedule is published to Home Assistant or written to opt_res_latest.csv (synthesis F11).
   Nothing hands a caller the structured plan back.

There are two ways to add the shared state. The original version of this RFC put it inside
EMHASS. The problem, raised by LesIT1 in #931 and consistent with #789, is that an in-core
registry makes EMHASS stateful: concurrency, atomic writes, file corruption, and demand
lifecycle all become core concerns the project has chosen to avoid. The alternative keeps
EMHASS stateless and puts the state in a sidecar: a small separate service that owns the
registry and renders the active demands down into an ordinary stateless optim call. The
solver never knows the registry exists. This RFC takes the sidecar route.

Architecture

  consumer events                      sidecar (external, our layer)            EMHASS (stateless core)
  (EV plugged in,        register /    ┌───────────────────────────┐
   washer started,  ───  refresh / ──▶ │  registry: active demands  │
   pool season)          withdraw      │  + lifecycle (expires_at)  │
                                       │  + recurrence rules        │
                                       └───────────┬───────────────┘
                                                   │ each MPC cycle:
                                                   │ aggregate → deferrable payload
                                                   ▼
                                       POST /action/dayahead-optim (runtimeparams)  ──▶  MILP solve
                                                   ▲                                       │
                                       GET /api/v1/plan  (structured plan, N1) ◀───────────┘
                                                   │
                                       sidecar holds the plan; consumer reads it and actuates

Placement against the #789 guard (synthesis F16): the registry and any actuation are glue,
outside EMHASS core. EMHASS stays the MILP optimizer and nothing more. The sidecar submits
demands through the existing /action/* runtime-params path — the same mechanism a
multi-deferrable operator already uses by hand today — so no new inbound API is added to
EMHASS. The one core addition is the plan output (N1) so the sidecar can read the result back
without scraping a CSV.

EMHASS-core change (the only ask): N1, structured plan output

A read-only endpoint that returns the most recent optimization result as JSON.

• GET /api/v1/plan — the latest opt_res as JSON records, carrying emhass_schema_version.
• Columns follow the existing plan-output schema (docs/plan_output_schema.md, docs(schema): publish plan-output column schema + version constant #835); the
  version follows EMHASS_SCHEMA_VERSION.
• It mirrors GET /api/v1/last-run (feat(api): add GET /api/v1/last-run JSON endpoint #851), which already returns run metadata. opt_res is
  computed and currently discarded (synthesis F11); this serializes what is already in hand.
• Additive, read-only, non-breaking. The HA publish and CSV paths are untouched.

The sidecar flow is: POST the aggregated demands to /action/dayahead-optim (or
naive-mpc-optim) via runtime params, then GET /api/v1/plan to read the structured result.
Because /action is awaited and EMHASS is a single writer, the plan returned by the GET is
the one the POST just produced.

The sidecar (external, not an EMHASS change)

Described here so the architecture is legible, but it is not part of the EMHASS ask. It owns:

• Consumer API. register / refresh / withdraw / list for demands. A required
  expires_at plus idempotent refresh means a consumer that dies just ages out, with no
  orphaned demand wedged in the plan. This API is internal to the sidecar; it never reaches
  EMHASS.
• Persistence. The registry as JSON (no pickle), single-writer with atomic temp-file +
  rename.
• Aggregation. Each cycle, fold the active demands into a standard deferrable-load
  runtime-params payload for EMHASS.
• Recurrence. A recurrence rule per registration for standing/seasonal loads (a pool pump
  whose hours shift monthly), rather than materialized instances.

A registered demand reuses the runtime-schema-family vocabulary (param_definitions.json,
runtime_params.json / #915) rather than inventing a new shape: essentially a persisted
runtime deferrable-load demand plus priority, source, and a lifecycle.

Threat model (split by layer)

Per the #808 maintainer comment, the concern is code injection. Splitting the design splits
the surface:

• EMHASS-core side: near-zero new surface. The only addition is GET /api/v1/plan. It is
  read-only, takes no input, persists nothing, and returns the same data EMHASS already
  publishes to HA. The injection-risk surface is unchanged from today.
• Sidecar side: owns the persistence risk, out of core. JSON-only (no pickle, no
  deserialization of code), validated on load with hand-rolled checks, writes confined to the
  sidecar's own data directory, ids never used to build filesystem paths, single-writer atomic
  write. A bad demand persists only until its expires_at. None of this touches EMHASS.

The pivot's headline security effect: the new EMHASS attack surface drops from "stateful
persistence with a widened blast radius" to one read-only GET.

Backward compatibility

• GET /api/v1/plan is additive. With no caller, nothing changes.
• /action/*, configuration, and the HA publish path are unchanged.
• No new runtime dependency (stdlib JSON on the sidecar side; nothing new in core).
• Default config still works unchanged.

Roadmap and sequencing (coordination synthesis N1–N7)

This RFC delivers N1 only. The rest is sequenced, not bundled:

• EMHASS-core siblings (separate RFCs/PRs, later): N2 EV-as-flexible-load with an SoC
  accumulator (closes the lossy EV→deferrable mapping; L; independent of multi-battery);
  load-priority MILP (EMHASS has no inter-load ranking today; its own sibling RFC); N5
  soft-constraint / never-infeasible behavior for drop-in fidelity.
• Sidecar features (not EMHASS RFCs): template store, MQTT intake, recurrence/seasonal
  enablement, the register/refresh/withdraw API, persistence.
• Minimal first slice: the GET /api/v1/plan PR. It is independently useful (machine-
  readable plan for any consumer, including the LLM-ready goal) regardless of the sidecar.

Known limitations (interim, with a closure path)

• EV is modeled as a deferrable load, lossy. The sidecar maps an EV charge need to a
  deferrable with an energy target and a window. It cannot express an SoC curve
  ("40% → 80% by 07:00") until N2 lands (synthesis F10). Until then the mapping is an
  approximation, not a permanent ceiling.
• No user-intent priority. When not everything fits, conflicts fall to the existing cost
  objective rather than an explicit per-load priority, until the priority sibling lands.

Open questions

• Sidecar placement. Standalone companion repo, a small community org, or something EMHASS
  references. This is the maintainer's call to steer; the only hard requirement is that it
  stays separate from and non-binding on EMHASS core.
• Recurrence model. The exact shape of the recurrence rule and seasonal enablement
  (sidecar concern).
• Per-source identity. With multiple sources registering, whether a source owns its
  entries so one can't overwrite another's. Sidecar concern; likely per-source namespacing or
  a token.
• Outcome richness. How detailed the per-demand result is (a feasibility flag vs
  deferred/reduced/rejected detail). Sidecar concern.
• MQTT intake. Whether demands are also submittable over MQTT. A sidecar transport
  feature, not an EMHASS change.

---

Appendix: superseded original (in-core registry, S2)

Original RFC 0001 as posted 2026-06-02 — proposed an in-core stateful registry. Superseded 2026-06-16 by the sidecar design above, after #931 feedback. Kept for the discussion context LesIT1's comment responds to.

RFC 0001: EMHASS as a stateful shared-plan service (persistent flexible-load registry)

Status: Issue-filed
Discussion: #931
Upstream context: #824 (EV use-cases follow-up to #789)
Author: OptimalNothing90
Date: 2026-06-02

Motivation

In a home with several flexible loads, they all compete for the same scarce resource: PV surplus and grid import/export headroom. The loads differ in nature. Some are ad-hoc (an EV charge session, a washing machine, a dishwasher), some are predictable and recurring (heat pump, hot water, heating), and some are seasonal (a pool pump). Today each one is optimized on its own:

• evcc plans EV charging by itself, using price and PV heuristics;
• the heat pump or hot-water tank runs greedily on its own local surplus trigger;
• EMHASS optimizes the home battery and its statically-configured deferrable loads.

Nobody allocates the scarce capacity across all of them. The result is first-come-first-serve contention: a surplus appears, several loads jump on it independently, their combined draw exceeds the available surplus or a grid limit, loads trip or back off, and the system oscillates. It happens even in summer, where surplus is still finite at any given instant, and it gets worse in the shoulder seasons.

EMHASS already solves the whole-house MILP that could arbitrate this, but it does so statelessly, one request at a time. Each caller has to resend the entire problem on every call, and there is no persistent notion of "the flexible loads currently in play" or "the current plan". So a multi-source setup (evcc, a heat pump, HA automations, and eventually LLM agents) can't contribute demands to one shared plan. Each source would have to own and resubmit the whole problem, which is what produces the isolated, conflicting optimizations above.

There is an obvious alternative: keep EMHASS stateless and put a stateful aggregator in front of it, one that continuously collects every demand from every source and resubmits the full problem each cycle. The trouble is that the aggregator is the hard part. It has to know about, poll, and re-capture all the sources on every tick, which makes it a complex, tightly-coupled, central component. Holding the registry inside EMHASS inverts that. Each consumer becomes a simple trigger-based integration: it fires a register or withdraw when its own event happens (the EV is plugged in, the washing machine is started), and otherwise does nothing. EMHASS already holds the optimization, so it is the natural place to aggregate. The integration burden drops from "build a stateful all-source aggregator" to "POST one demand when your event fires". That is the main reason the state belongs in EMHASS rather than in every integrator.

The stateless model also blocks the LLM-ready direction. For EMHASS to be a first-class planning interface for AI tooling, agents need to query the current plan and propose or withdraw demands against a persistent shared state, instead of reconstructing the whole optimization problem on every interaction.

Proposed change

Add an optional, persistent flexible-load registry to EMHASS, turning it into a stateful shared-plan service while keeping it recommend-only.

The core primitive is generic: any consumer registers a demand through one uniform API, whether that demand is an EV charge need, a thermal target, or a generic deferrable load. The contention example above spans several consumers, but the mechanism itself is consumer-agnostic. The consumer-specific work stays out of scope here: how a given consumer (evcc for an EV, or a heat pump for a thermal target) builds and submits its demand, and how the returned plan then gets actuated. Each consumer is its own follow-on piece, a public usage example plus that consumer's own (often private) glue. Note that a demand (what the consumer needs) and actuation (how the plan is carried out) are separate concerns; actuation, including any device-signalling protocol, stays on the consumer side.

1. Registry (the persistent state). A set of registered flexible demands. Each entry carries its constraints (energy or SoC target, deadline window, min/max power, charge-only flag), a priority, a source id, and a lifecycle status (active or expired).
2. One shared plan, full re-solve. On each optimization tick (MPC), EMHASS solves the entire registry together with the battery, base load, and forecasts into a single plan. Because every solve considers all active demands at once, arrival order no longer decides the outcome (no first-come-first-serve admission), and the combined draw can't exceed the limits, since the joint power balance forbids it. Resolving conflicts by user intent, meaning which load yields when not everything fits, needs an explicit load priority that EMHASS doesn't have yet. Until the sibling priority RFC lands, conflicts fall to the existing cost objective rather than to user priority. So this RFC on its own delivers overshoot-prevention and one shared plan; priority-based arbitration is left to the sibling.
3. Inbound admission API. Callers submit, update, or withdraw a registered demand. EMHASS re-solves and returns the new plan plus a per-demand outcome (for example: fully scheduled, deferred, reduced, or rejected as infeasible at this priority). This is the "submit; if feasible, adopt into the shared plan" behaviour, done as a full re-solve rather than a greedy incremental merge.
4. Recommend-only is preserved. EMHASS returns the plan; it never actuates. Callers (Node-RED, HA, or a glue layer) read the plan and drive the hardware. EMHASS never calls out to a device or another service; submissions are inbound only.
5. Opt-in containment. The registry is additive and optional. With an empty or unused registry, the optimization path is identical to today's stateless solve.

Notes on the demand space:

• The shared plan unifies static config with dynamic registrations. Predictable loads (hot water, heating) that EMHASS already models via thermal_config stay as configuration; ad-hoc loads (EV session, washing machine, dishwasher) arrive through the registry; seasonal ones (pool pump) are standing entries enabled per season. All of them feed the one solve. The registry is additive, not a replacement for existing config, and it doesn't have to "own" the static loads in order to arbitrate against them.
• Templates. A registered demand can be fully specified inline, or it can reference a template stored in EMHASS: a reusable named load profile (dishwasher-eco, pool-pump-summer) built on EMHASS's existing fixed-profile deferrable-load definitions, so recurring loads don't have to resend their shape every time.
• Most "started" loads are shiftable within a window, not fixed. A resident plugging in the EV or starting an appliance is a trigger to register a shiftable demand with a window, not an immovable load: EV → [now, +3h] (or by departure); washing machine → [now, +2h]; a dishwasher with a built-in night delay → [02:00, 06:00]. EMHASS schedules within the window. The genuinely immovable case (a dumb appliance with no delay, already running) is the rare committed mode, which is account-only: EMHASS can't move it but registers it so other loads yield. hybrid means run on surplus or cheap power, but guaranteed done by a deadline. Which mode applies is the EMHASS user's per-load preference.
• Every entry has a mandatory timeout and an update path. A registered load carries a required expires_at, so stale entries drop automatically and a consumer that registers but never withdraws can't leave a zombie demand. Standing or seasonal loads stay alive by refreshing via PUT, and the same PUT updates a changing demand (a new SoC, a moved deadline).

The registry is the concrete form of the long-parked "persistent flexible-load registry" idea (follow-up to #824) and the basis for the LLM-ready goal.

API / contract

New, additive endpoints (sketch; exact paths still open). They all live under the existing /api/v1 namespace, consistent with /api/v1/last-run. Transport here is REST; an MQTT intake is a possible follow-on (see Open questions).

• POST /api/v1/registry/loads — register a flexible load; returns its id and post-solve outcome.
• PUT /api/v1/registry/loads/{id} — update an existing load (new target/deadline/SoC), or refresh its timeout.
• DELETE /api/v1/registry/loads/{id} — withdraw a load.
• GET /api/v1/registry/loads — list active registered loads.
• GET /api/v1/plan (or an extension of GET /api/v1/last-run) — the current shared plan as structured JSON, i.e. the recommendation.

A registered load reuses the runtime-schema-family vocabulary rather than inventing a new one. It is essentially a persisted runtime deferrable-load demand plus priority, source, and a lifecycle:

{
  "id": "ev-loadpoint-0",          // server-validated; NOT used as a filesystem path
  "source": "evcc",
  "type": "ev | thermal | generic",
  "flexibility": "shiftable | committed | hybrid",  // user preference; committed = run-now, account-only
  "energy_target_wh": 24000,        // OR soc_target + capacity_wh
  "deadline": "2026-06-03T07:00:00+02:00",
  "window_start": "2026-06-02T20:00:00+02:00",
  "p_min_w": 1400,
  "p_max_w": 11000,
  "charge_only": true,
  "priority": 1,
  "expires_at": "2026-06-03T08:00:00+02:00"   // mandatory; stale entries auto-drop, refreshable via PUT
}

The same object shape serves every consumer. An EV (shown above), a heat-pump thermal target, and a generic load (washing machine, dishwasher, pool pump) differ only in their field values, not in structure. A demand can also carry "template": "<name>" in place of the inline fields, referencing a profile stored in EMHASS.

• Demand fields align with param_definitions.json and runtime_params.json (the runtime-schema-family; runtime_params.json shipped as feat: add runtime_params.json (machine-readable runtime param schema) #915).
• Plan output reuses the plan-output schema (docs/plan_output_schema.md, docs(schema): publish plan-output column schema + version constant #835) and is versioned by EMHASS_SCHEMA_VERSION.
• Load priority depends on a load-priority MILP capability that EMHASS doesn't have today (there is no inter-load ranking). That is proposed as a sibling RFC, not folded in here.

Threat model

Per the #808 maintainer comment, the focus is code injection. The new surface here is persistence:

• No pickle, no arbitrary deserialization. The registry is persisted as JSON and validated on load by hand-rolled checks (no schema-validator dependency, consistent with the openapi generator). A malformed or unexpected file is rejected, not executed.
• No path traversal. Writes are confined to the existing data_path, the same directory already used for opt_res_latest.csv. Load ids are server-generated or validated and are never used to build filesystem paths.
• No shell-out, no DB, no eval. Registry mutations are pure data, validated against the demand schema.
• Corruption-safe writes. A single writer (the serving process), with an atomic write via temp-file and rename.
• The API takes JSON validated against a fixed shape, the same injection-risk class as the existing /action/* runtime-params intake. Persistence does widen the blast radius, though: a bad demand persists across every solve until its expires_at, where a stateless POST only affects one solve. The mandatory timeout bounds that, and invalid entries are rejected at the API and never persisted.

Backward compatibility

• Fully opt-in. With no registered loads, the solve is functionally identical to today's; the registry contributes nothing to the optimization. The existing /action/*, configuration, and HA publish paths are unchanged.
• Stateless callers keep working. A caller that ignores the registry and POSTs the full problem (as today) is unaffected. The registry is an additional way to drive EMHASS, not a replacement.
• No new hard dependency (stdlib JSON persistence; nothing new at runtime).
• Default config still works unchanged.

Open questions

• Recurrence and seasonal enablement. Every entry now carries a mandatory expires_at (decided above). What is still open is the recurrence model for standing loads: the auto-refresh cadence, and calendar or seasonal enablement (a pool pump active only in summer, say), via explicit re-PUT or a declared schedule.
• MQTT intake. Should demands also be submittable over MQTT (devices or HA publishing to a topic) alongside REST/v1? The primitive is transport-agnostic, so the recommendation is a separate follow-on RFC for an MQTT transport adapter rather than expanding this one.
• Template store. How stored load templates (the reusable profiles referenced by template) are defined, versioned, and managed. Possibly a follow-on RFC rather than part of this one.
• Priority semantics. Strict ordering versus weighted soft-penalty, and confirming this lands as a sibling RFC (load-priority MILP) rather than here.
• Persistence location and concurrency under multi-process or HA-addon deployments (the single-writer assumption, the lock strategy).
• Identity and authorization. EMHASS's API is usually local and unauthenticated. With multiple sources registering, should a source own its entries, so one source can't overwrite or withdraw another's, by accident or otherwise? Probably simple per-source namespacing or a token; TBD.
• Outcome richness. How detailed the per-demand admission result should be for v1: a minimal feasibility flag, or the fuller deferred/reduced/rejected detail.
• Consumer adapters are separate artifacts. Each consumer (evcc/EV first, as the cleanest case with precise demand, actuation, and feedback; then heat-pump/thermal; then generic loads) is specified outside this RFC, as a public usage example plus that consumer's own (often private) glue. This RFC stays consumer-agnostic: it neither mandates nor depends on the evcc/evopt contract, nor on any device actuation protocol.
• Minimal first slice. The smallest reviewable PR that lands the registry opt-in without the priority feature (a single-demand registry, full re-solve, JSON persistence), to validate the direction before the larger surface.

[LesIT1]

@OptimalNothing90 This resonates a lot. I run EMHASS with three deferrables in production (EV charge floor, EV solar top-up, and a pool pump) on a single-phase-aware setup, plus a separate controller that actuates battery grid-charge. The contention you describe is exactly the failure mode I keep hitting: the moment you have more than one flexible load chasing the same PV surplus, isolated optimisers oscillate and step on each other.

One thing worth separating out though: EMHASS already co-optimises competing loads when you model them as deferrables in a single optim call. So for me the new capability here isn't really "solve them together", it's the two things wrapped around that:

Dynamic registration. Right now I regenerate the whole deferrable payload every MPC cycle from an external script because the loads come and go (car plugged in or not, pool hours seasonal). A register/refresh/withdraw API would let each consumer own its own demand lifecycle instead of me rebuilding the payload each run. That's the real ergonomic win.
Persistence across cycles. Useful, but this is also where the cost lives.
My one reservation is that making EMHASS itself stateful pulls in concurrency, atomic-write/corruption, and lifecycle-bug surface that the project has deliberately stayed clear of. Before committing to that, is there a lighter middle ground worth considering? A thin registry sidecar that holds the persistent load state and renders it down into a normal stateless optim call would give you the dynamic-registration API and the shared plan, while keeping the EMHASS core stateless and recommend-only. The solver doesn't need to know the registry exists; it just receives the aggregated deferrables for that cycle.

Either way I'd adopt this. Happy to be a test bed for a real multi-deferrable + external-actuator case if it helps shape the design. Two design details I'd want nailed down early: how recurrence/seasonality is expressed (my pool pump hours change month to mont

[OptimalNothing90]

@LesIT1 thanks, this is the steer I needed. The sidecar framing is better than mine and I'm switching to it.

Agreed the co-optimization isn't the new part. EMHASS already solves competing deferrables in one call. What's missing is the two things you called out: dynamic registration, so each consumer manages its own demand instead of you rebuilding the payload every cycle, and persistence between cycles. And your worry is the right one. Making EMHASS itself stateful drags in concurrency, atomic writes and lifecycle bugs the project has stayed away from on purpose, and I don't want to fight that.

So I'll rework the RFC around a thin registry sidecar that owns the state and the register/refresh/withdraw API. Each cycle it folds the active demands into a normal deferrable payload, calls EMHASS the way it already works, and reads the plan back. The solver never needs to know it's there. The one thing it needs from core is a machine-readable plan out of /action/* (right now you get an HA publish or a CSV, not a structured schedule), which is a small, broadly useful addition rather than a statefulness change.

On your two details: register/refresh/withdraw with a required expires_at and idempotent refresh, so a consumer that dies just ages out instead of leaving a stale demand wedged in the plan. For recurrence I'd lean toward storing a rule per registration rather than expanding every instance, consumer free to override, but your monthly pool-pump shift is exactly the thing I'd rather design against a real setup than guess at.

Which is why I'd take you up on the test-bed offer if it still stands. Between your setup (EV floor, solar top-up, pool pump, separate battery controller) and mine here (EV, PV, battery, and a heat pump running both heating and hot water), we'd cover most of the contention that actually matters.

One thing I'm deliberately not deciding yet: where the sidecar lives. Standalone companion, a small community org, or something EMHASS just points at. I'd rather hear @davidusb-geek's preference there than assume, since the point is to keep it separate from core and not drop anything on him he didn't ask for.

I'll revise RFC 0001 to the sidecar version next, with the JSON output as the only thing I'm asking of core.

[LesIT1]

@OptimalNothing90 glad the sidecar framing landed. Folding active demands into a normal deferrable payload and calling EMHASS through the existing stateless paths is exactly the shape I was hoping for. It keeps the core's statelessness intact and puts all the lifecycle and concurrency work in one place where it can be reasoned about on its own.

The machine-readable plan output from /action/* makes sense as the one core addition. Right now anyone consuming the plan has to either read it back out of HA or parse the CSV, and a structured schedule straight from the action endpoints would be genuinely useful well beyond this RFC. Agreed it sits in a different category from the statefulness question.

Your lifecycle and recurrence answers both work for me. A required expires_at with idempotent refresh is the right default for keeping dead consumers from leaving stale demands around, and storing the recurrence rule per registration rather than expanding instances matches how this actually behaves in practice (my pool pump hours shift month to month, so a rule the consumer can override is much cleaner than pre-expanded slots).

On where the sidecar lives, I agree that's David's call and shouldn't be pushed onto the core.

Happy to be a test bed for this once you have something to try. I'm running three deferrables plus a separate battery controller in production, so I can exercise the registration, refresh, and withdrawal paths against a real setup and feed back on the recurrence and lifecycle behaviour. Let me know if there's anything else you need from me as you revise the RFC.

[OptimalNothing90]

@LesIT1 glad it landed. I've updated the RFC in the thread above to the sidecar design (the original is collapsed at the bottom if you want to compare). Nothing needed from you right now. Once there's a sidecar MVP to try I'll ping you, so we can exercise the register/refresh/withdraw and recurrence paths against your real setup. After that it's David's call on the one core addition (the plan output) and where the sidecar lives.

On tooling: I've been getting a lot out of cross-model review lately. Claude as the main driver, Codex as a PR reviewer; having a second model go over the diff catches a surprising amount before it ships. What are you running? If we're set up similarly it'd be worth coordinating that way on the sidecar and the plan-output PR.

[LesIT1]

@OptimalNothing90 sounds good, I'll wait for the ping.

On tooling, I'm running Claude as the main driver as well, but I've gone a slightly different way on review. Instead of a second vendor's model, I lean on an adversarial pass inside the same stack: a separate reviewer agent whose only job is to try to refute the change, a heavier reasoning model on the diff and a faster one for the mechanical edits. Anything headed upstream goes through a structured contribution flow with an early adversarial deep-dive before any of the PR is written, and that early pass is where most of the this-is-actually-wrong catches land.

I am tweaking this workflow every day, using the days learnings to guide me, the idea being to make sure mistakes aren't repeated. So far it's going well, but there is always room for improvement. When I had access to Fable, I used it to do a lot of reviewing of my workflows and improved them that way too.

Net effect feels close to what you get from Codex-as-reviewer, just kept within one model family. Glad to coordinate that way on the sidecar and the plan-output work. When there's a diff, I can run my adversarial pass over it and you run yours, then we compare what each side flags before it ships.

[mschaepers]

@LesIT1 the promised ping: there's a sidecar MVP to try.

It lives in my contributions repo for now, under prototypes/flexd/. Docs and a compose bundle (flexd + EMHASS + optional Mosquitto) are in the folder; docker compose up -d plus one YAML file gets you a running pair.

What it does today:

• register/refresh/withdraw over REST (full OpenAPI) and MQTT. There's also a plain-HTTP variant for controllers that can't parse JSON — I need that for Loxone.
• each cycle it folds the active demands into one naive-mpc-optim runtime payload, calls stateless EMHASS, and reads the result back over the merged /api/v1/plan. Per-demand setpoints come back out as REST fields and retained MQTT topics.
• daily standing loads (hot water and the like) with elapsed-hours accounting that survives re-solves and restarts. This is the recurrence piece we discussed, in its simplest same-window-every-day form.
• trigger templates for appliances: "dishwasher started between 20:00 and 06:00 means finished by 06:30, but don't start before 21:30". A re-trigger is a pure refresh, so a flapping integration can't move a deadline once set.

Deliberately not in yet: priority (EMHASS can't rank loads), seasonal recurrence, per-source auth. EV is still the lossy energy-in-window mapping until an SoC-aware load type exists upstream.

To be clear about where this stands: it's a first shot, not a finished thing. 150 unit/contract tests and the compose E2E pass, but it has not run against a real household, and I'd genuinely value a second pair of eyes on whether the shape matches what you had in mind when you argued for the sidecar — API surface, lifecycle, the aggregation step, any of it. If you'd rather tear into the design before touching the code, that's just as useful to me.

Beyond that, the practical ask if the offer stands: exercising register/refresh/withdraw and the standing-load path against your multi-deferrable setup. Lifecycle works as we settled it in the RFC — mandatory expires_at, per-source ownership, consumers actuate.

Placement is still open pending David's steer; it stays in prototypes/ until then. And if a call is easier for the setup walk-through, that earlier offer stands — I can do your morning, timezone-wise.

[LesIT1]

@mschaepers good timing, the offer stands. I'll take this in two passes. First the review you asked for: I'll run my adversarial pass over prototypes/flexd and feed back what it flags, API surface and lifecycle included. Then a shadow run against my real setup: flexd plus a second EMHASS instance alongside my production stack, registering the same three deferrables (EV floor, EV solar top-up, pool pump) with actuation left off, so I can compare its plans against what my current pipeline produces. One note from setting that up mentally: my production EMHASS is driven by an external MPC loop every 15 minutes, so flexd needs its own instance or the two callers will fight over the results. Worth a line in your docs, anyone with an existing caller will hit it.

One question on sequencing: are you sending the /api/v1/plan PR upstream yourself, or do you want me to pick it up? Happy either way, just don't want it sitting unclaimed.

I'll be back with review findings within the week.

[LesIT1]

@mschaepers review's done, quicker than I expected to find time for. Short version: the shape matches what I argued for in June and a lot of the engineering held up under attack, the registry persistence, ownership guards and plan-slot attribution in particular. But I'd fix three things before this touches a real household, because they all fail silently on the primary delivery paths. Full write-up with file and line references, plus runnable repros for all three, is in OptimalNothing90/emhass-contributions#1. The headlines:

1. Standing loads accrue zero elapsed hours under real clocks. on_hours_elapsed only counts plan slots fully inside [last cycle, now], and live cycle instants always sit a few seconds past the grid, so every slot straddles a boundary. remaining_h never shrinks and the load gets re-solicited at full energy until window end. Your tests pass because they use grid-aligned timestamps that never happen live.

2. A template re-trigger rewrites expires_at to now plus default_ttl_s instead of keeping the deadline-based expiry. The flapping-start case your Loxone doc calls a harmless refresh actually gets the demand swept about an hour later, ahead of its finish_by, on any double press more than one ttl before the deadline. My repro moved expiry 10 hours earlier on a 30 second re-trigger. The test guarding this passes vacuously (the scenario pins time but refresh uses the wall clock), so it's worth threading a now parameter through refresh while fixing it.

3. JSON Infinity gets through the gt=0 field validation on every transport (the plain-HTTP one coerces power_w=Infinity to a clean 201), persists in the registry, and serializes back out into the naive-mpc-optim payload. One bad register breaks scheduling for every demand until it expires. allow_inf_nan=False plus a sanity ceiling on power and energy.

The issue has the rest: a dozen mediums (corrupt-state recovery, a not_before rule that silently drops across midnight, a couple of real races, string-compared timestamps), a tail of lows, and the list of what held up under attack, which is honestly the longer list. Two of the mediums need checking against the pinned EMHASS rather than flexd itself, I've flagged which, and I'm happy to help verify those. Once the top three are fixed I'll move to the shadow run against my setup as we discussed.

[MMicieli]

A couple of possible extensions/questions came to mind from the LLM/sidecar direction being discussed here. They may already fit within the intended scope, but I thought they might be useful use cases to consider.

Context as well as flexible demand

Some useful external information does not seem to fit naturally into a flexible-load registration.

For example:

• "The house will be unoccupied for the next three weeks, so expected base load will be much lower."
• "A smoke event is causing actual PV production to run well below the current solar forecast."
• "The battery needs to be at a certain SOC by a particular time."

These are more like temporary context or input changes than new demands.

Would it make sense for a sidecar to eventually distinguish between something like:

demand
    -> an energy requirement to schedule

context
    -> temporary information affecting an input/constraint

query/scenario
    -> inspect or simulate without changing production

For temporary context, an expiry and some basic provenance could probably be useful:

source
reason
created_at
expires_at

The sidecar could then translate that context into normal validated EMHASS runtime parameters rather than adding new optimisation logic. EMHASS would still make the actual optimisation decision.

Read-only explanation and "what-if" use cases

Another potentially useful LLM/application use case could be purely read-only.

For example:

Why did the last optimisation fail?

Why is the battery discharging now?

What would the plan look like if SOC had to be at least 80% by 17:00?

With /api/v1/plan, /api/v1/last-run and the relevant runtime inputs, an external tool could potentially explain the current result or run an isolated scenario without changing the active system.

A progression such as this might provide useful functionality without requiring autonomous control:

1. Explain / diagnose
2. Run non-actuating "what-if" scenarios
3. Propose temporary context or constraints
4. Apply only through the normal validated optimisation path

Machine-readable commissioning/debugging

There may also be some value beyond runtime orchestration in making EMHASS easier for external tools to understand.

For example, could an external tool determine, without needing to inspect the Python source:

• what capabilities the running EMHASS version supports;
• the effective configuration after defaults and runtime overrides;
• parameter units/ranges;
• the inputs associated with a particular optimisation run;
• validation errors or warnings?

That could help not only LLM-based tools but also ordinary commissioning, testing and diagnostic tooling.

Run correlation/provenance

The other piece that seems particularly useful if multiple external callers are possible is being able to reliably correlate a result with the request that produced it.

Something conceptually like:

request/run ID
generated_at
input/config provenance
associated plan

would make it much easier for an external tool to say "this is why this particular optimisation produced this result" without accidentally reasoning about a different/latest run.

None of this would require the LLM or sidecar to become an optimiser itself. The boundary could remain:

external tool / LLM
    -> interpret intent, provide context, validate, explain

EMHASS
    -> optimise

downstream integration
    -> consume/actuate the accepted result

Not sure how much of this is already implicit in the proposed sidecar direction, but these seem like useful additional interaction patterns to keep in mind if the longer-term goal includes external agents or conversational interfaces.

[LesIT1]

@MMicieli the demand/context/query split makes sense to me, and the nice part is it doesn't need new machinery. The lifecycle already settled for demands (mandatory expires_at, per-source ownership, idempotent refresh) carries straight over to context entries. They're just a different payload type the sidecar folds into validated runtime parameters instead of a deferrable entry. Source and expires_at are already the registration record, reason and created_at are cheap additions to it. So I'd treat context as a second registration type on the same registry.

One of your examples is something I already do by hand. The overnight SOC floor that has to hold until the morning peak... that's your battery-SOC-by-a-particular-time case, and today it goes in as a runtime parameter from my own scripts. Not hypothetical.

What-if is less free than it looks. EMHASS being stateless gets you part way, you can call it with modified inputs, but the solve itself writes opt_res_latest.csv and that's the file the next publish-data reads. With continual_publish it queues entities at solve time too. So "just don't actuate" isn't enough isolation, and this is the same fight-over-results problem from my flexd review: scenario solves want their own instance until results are keyed per run. Which is why your run-ID point matters more than it first looks. Key the machine-readable plan output (already pencilled in as the one core addition) by run and the explain/what-if endpoints fall out of it fairly cheaply.

Sequencing wise I'd keep the MVP demand-only until the three blockers from my review are closed out, then layer context on. Happy to exercise a context path in the same shadow run once it exists.

[MMicieli]

Thanks, that makes sense.

I had not considered the opt_res_latest.csv / continual_publish interaction with the what-if case, so I can see why run-keyed results are the prerequisite.

Treating context as a second registration type on the same lifecycle also sounds much cleaner than introducing anything separate.

Demand-only MVP first, then exercising a context entry in shadow once the current blockers are resolved, sounds like a sensible sequence.

[OptimalNothing90]

An update on RFC 0001, following the two fix batches on the sidecar prototype after LesIT1's review and the discussion in #1157 about dry_run. The revised RFC is at https://github.com/OptimalNothing90/emhass-contributions/blob/main/rfcs/0001-shared-plan-registry.md (changelog at the top, the June text kept in the appendix). Nothing here is a request for a core change beyond what #1157 already discusses; it is a sharpening of what the sidecar is accountable for, so the design can be judged on criteria rather than on the prototype.

What changed in the RFC

The sidecar is now described as the demand-lifecycle layer in front of a stateless core: registration, ownership, expiry, persistence across restarts, mapping onto runtime parameters, attribution of the result back to the demand, and an account of what the mapping had to change. It is not a second optimizer and not a coordinator between solvers. The acceptance criteria it has to meet before touching a real household are written down:

1. A withdrawn or expired demand reads OFF within one poll on every transport (REST, plain HTTP, retained MQTT), before the next solve.
2. A restart loses no active demand and no adopted plan; a registry file with one bad entry keeps the valid ones.
3. Remaining energy never produces an infeasible semi-continuous solve: whole timesteps for on/off devices, fractional hours for modulating ones, a closed window passed as inactive rather than pinned on.
4. Plan attribution survives a plan switch; elapsed energy is credited from the plan that was in force at each instant.
5. Every reduction of a request carries a machine-readable reason (today truncated, clamped, whole-slot rounding, unschedulable), so a consumer can answer "why did I get less than I asked for" from the response alone. This is the sidecar-side counterpart of the explainability request in # [Research Discussion] Exploring explainable and interrogable optimisation in EMHASS #1185.
6. Ownership by source is enforced; 7. non-finite and out-of-range input is rejected on every transport before it reaches the registry.

The first four are what the review in OptimalNothing90/emhass-contributions#1 found broken or fragile; they are now criteria rather than fixes.

The single-writer assumption is gone

The June text said that because /action/* is awaited and EMHASS is a single writer, the plan a following GET /api/v1/plan returns is the one the POST produced. That holds only with one caller per instance, which LesIT1's production loop and the run-correlation point raised here both show is not safe to assume. The RFC now names request-bound results as the target (a solve that returns its own plan, or a plan store keyed by a run id named in the response) and the current guard as the interim: adopt only if status is ok, the schema version is known and generated_at is strictly newer than the last adopted plan. The guard rejects a re-served plan; it cannot tell two fresh plans from two callers apart, so one caller per instance stays the documented operating rule until then.

Relation to #1157 and the dry_run dependency

The federated backend coordinates solvers by prices at the meter. RFC 0001 addresses a different problem: one optimizer, many consumers that come and go. The sidecar sits in front of whichever solving path the core offers, never takes part in price iteration, and if the core ever absorbs demand lifecycle it shrinks to transport adapters and the conformance tests behind the criteria above. That outcome would be fine.

dry_run as specified in #1158 (same request, real solve, plan in the response, nothing written, private optimization object) is the core primitive the sidecar needs next, independent of the backend. I have said so in #1157 and support splitting it into its own PR.

On the context and what-if proposal

@MMicieli, your demand / context / query split from August is now the sidecar's roadmap, in that order. Demands are what runs today. Context entries come after the shadow run as a second registration type on the same registry, with the same expires_at and ownership, folded into validated runtime parameters instead of a deferrable slot; the battery SOC floor example maps to soc_target and soc_target_timestep. Queries and what-if need dry_run and are not offered until it exists upstream, for the reason LesIT1 gave: a scenario solve against the live instance replaces the live plan. Two points from #1157 matter to the sidecar specifically and are recorded as dependencies: a release that does not know the flag must not run a live solve when asked for a dry one, so the sidecar has to gate on a version or capability signal, and actions that cannot honour the flag should reject it rather than ignore it. Your run-correlation point is the "request-bound results" item above.

EV and multi-battery

The June sentence calling the EV SOC model (N2) independent of multi-battery is withdrawn. Since then phase 1 of #610 shipped, phase 3 (#1150) is in design, and phase 2 is where an intermittently connected battery with a re-anchored SOC and a per-departure target belongs. N2 is therefore now an input to phase 2 rather than a competing model: docs/ev-requirements.md in the same repository lists twenty requirements with acceptance scenarios from the evcc side (arrival and departure, several windows per horizon, SOC with a timestamp and staleness, usable capacity, losses, current steps and phase switching, behaviour without telemetry, the responsibility split, feedback of charged energy, time zones and units), each mapped to the real evcc field and the current EMHASS parameter, with the unverified parts listed as open questions. @LesIT1, if that is useful as phase 2 input I would rather you shape it than me guess at the formulation; corrections welcome in either thread.

Placement of the sidecar remains @davidusb-geek's call and nothing above changes the one-caller, recommend-only, no-coupling-in-core position.