milp optimization

HSEM MILP Optimization

The MILP solver (planner/milp_optimizer.py) finds the globally optimal battery charge/discharge schedule using scipy's HiGHS linear programming solver. It is the primary planner — heuristic candidates are generated alongside it for benchmarking and fallback, but when scipy is available the MILP solution is preferred.

Pipeline

flowchart TD
    A[Engine passes slots with populated prices / PV / consumption]
    B[Identify future active slot indices past vs fixed-zero]
    C[Build per-slot data arrays: p_imp, p_exp, net_load, base_load, pv_avail]
    D{EV configs provided?}
    E[Rebuild net_load without pre-computed EV planned loads]
    F[Keep fixed EV planned load in net_load]
    G[Build objective vector c_obj: import cost, export revenue, cycle cost, conversion loss, terminal-SoC credit, SoC penalties, EV deadline penalties, EV pre-deadline benefit, EV charge-past-target benefit]
    H[Build equality constraints A_eq: energy balance per slot inc. EV charger load]
    I[Build inequality constraints A_ub: SoC recurrence soft bounds, mutual exclusion, cycle cost auxiliary m >= ec/ed, EV cumulative SOC, EV deadline target, EV post-deadline zero-charge, EV surplus-only]
    J[Build variable bounds: ec, ed capped; pv fixed to actual surplus; penalties >= 0]
    K[linprog method = highs, timeout = 2s]
    L{Solution found?}
    M[Decode solution: ec, ed, gi, ge, pv, m, penalties, EV charging]
    N[Write recommendations to output slots BatteriesChargeGrid, BatteriesChargeSolar, BatteriesDischargeMode, ForceBatteriesDischarge]
    O[Compute penalty violation diagnostics]
    P[Return slots and diagnostics]
    Q[Return None: solver failed or no future slots]

    A --> B --> C --> D
    D -->|Yes| E --> G
    D -->|No| F --> G
    G --> H --> I --> J --> K --> L
    L -->|Yes| M --> N --> O --> P
    L -->|No| Q

Variable layout

Base battery variables (8 × n slots)

For each slot t ∈ 0…n-1 the LP variable vector x contains eight decision variables:

Offset	Variable	Name	Description	Bounds
`0`	`ec[t]`	`ec_off`	Energy charged and stored in battery this slot (kWh)	`[0, max_charge_per_slot]`
`n`	`ed[t]`	`ed_off`	Energy discharged from battery this slot (kWh)	`[0, max_discharge_per_slot]`
`2n`	`gi[t]`	`gi_off`	Grid import this slot (kWh)	`[0, ∞)`
`3n`	`ge[t]`	`ge_off`	Grid export this slot (kWh)	`[0, ∞)`
`4n`	`pv[t]`	`pv_off`	PV surplus available in slot t (kWh)	`[pv_avail[t], pv_avail[t]]` (fixed)
`5n`	`m[t]`	`m_off`	Auxiliary variable ≥ max(ec[t], ed[t]) for cycle cost (kWh)	`[0, ∞)`
`6n`	`s_max_pen[t]`	`s_max_off`	SoC upper penalty — kWh by which state of charge exceeds `usable_kwh`	`[0, ∞)`
`7n`	`s_min_pen[t]`	`s_min_off`	SoC lower penalty — kWh by which state of charge drops below 0	`[0, ∞)`

The state of charge soc[t] is not an explicit variable — it is derived from the forward recurrence:

$$ soc[t] = soc[0] + \sum_{k=0}^{t} \bigl( ec[k] - ed[k] \bigr) $$

Penalty variables s_max_pen and s_min_pen prevent infeasibility when the initial SoC lies outside [0, usable_kwh]. Their objective coefficient is extremely high (max(p_imp) × 100), so they are only used when the initial state is physically out of bounds.

EV co-optimization extension

When one or more active EVs are provided, the variable vector expands to:

$$ \text{total variables} = 8n + n \cdot E + E $$

where E is the number of active EVs.

Offset	Variable	Name	Description	Bounds
`8n + i·n`	`evN_c[t]`	EV N DC-side charge per slot (kWh)	`[0, evN.max_charge_per_slot]`
`8n + E·n + i`	`evN_pen`	EV N deadline target slack (kWh shortfall)	`[0, ∞)`

The EV charger AC load entering the energy balance equation is evN_c[t] / charger_efficiency.

When an EV's charge_past_target flag is True (EV already at user-configured target SoC, allow_charge_past_target_soc enabled, SoC < 100 %):

The deadline constraint is suppressed (deadline_slot = None) — no grid import pressure
The surplus-only constraint is added (see Constraints below)
An avoided-future-import-cost benefit (future_value_per_kwh, issue #630) is added to the objective, falling back to a tiny fixed tiebreaker when no future price data is available (see Objective function below)

When an EV has a deadline and charge_past_target=False (normal mode):

Pre-deadline slots (t ≤ D): direct benefit -ev_penalty_cost on ev_c[t] forces charging
Post-deadline slots (t > D): hard constraint ev_c[t] = 0 — charging is forbidden

Fuse constraint extension (issue #567)

When main_fuse_amps > 0, the variable vector expands further:

$$ \text{total variables} = 8n + n \cdot E + E + n $$

Offset	Variable	Name	Description	Bounds
after EV vars	`gi_pen[t]`	`gi_pen_off`	Grid import fuse penalty — kWh exceeding the main fuse rating	`[0, ∞)`

The max grid import per slot is converted from amps to kWh/slot:

$$ \mathrm{max_grid_import} = \frac{\mathrm{amps} \times 230 \times 3}{1000} \times \frac{\mathrm{interval_minutes}}{60} $$

This assumes balanced three-phase load at 230 V phase-to-neutral.

The penalty uses the same high coefficient as SoC penalties (max(p_imp) × 100), ensuring the solver only exceeds the fuse limit when physically unavoidable (e.g. house base load alone exceeds the rating). When main_fuse_amps is None or 0, no variables or constraints are added — behaviour is unchanged.

Objective function

$$ \begin{aligned} \mathrm{minimise} \quad \sum_{t} \delta_t \cdot \bigg[ & p_{\mathrm{imp}}[t] \cdot gi[t] && \text{grid import cost} \\ - & p_{\mathrm{exp}}[t] \cdot ge[t] && \text{export revenue} \\ + & \alpha \cdot m[t] && \text{battery cycle cost (depreciation)} \\ + & \epsilon_{\mathrm{chg}} \cdot p_{\mathrm{imp}}[t] \cdot ec[t] && \text{charge-side conversion loss cost} \\ + & \epsilon_{\mathrm{dis}} \cdot p_{\mathrm{imp}}[t] \cdot ed[t] && \text{discharge-side conversion loss cost} \\ - & \gamma \cdot \bigl( ec[t] - ed[t] \bigr) && \text{terminal-SoC replacement credit} \\ + & p_{\mathrm{soc}} \cdot \bigl( \mathrm{s_max_pen}[t] + \mathrm{s_min_pen}[t] \bigr) && \text{SoC soft-constraint penalties} \\ + & p_{\mathrm{fuse}} \cdot \mathrm{gi_pen}[t] && \text{Main fuse grid-import penalty} \bigg] \\ \end{aligned} $$

Plus EV deadline penalties (undiscounted — deadline is a hard commitment):

$$ \sum_{v=1}^{E} p_{\mathrm{ev_pen}}^{(v)} \cdot \mathrm{ev_pen}_v $$

Where:

Symbol	Description
$\delta_t$	Time discount per slot: $\delta_t = r^{\Delta t}$ where $\Delta t$ is hours from now
$p_{\mathrm{imp}}[t]$	Grid import price (currency/kWh)
$p_{\mathrm{exp}}[t]$	Grid export price (currency/kWh), clamped to 0 when below `min_export_price`
$\alpha$	Battery cycle cost per kWh: $\alpha = \frac{P \cdot L_{pct}/100}{2 \cdot N \cdot C_u}$
$\epsilon_{\mathrm{chg}}$	Charge-side loss fraction: $\epsilon_{\mathrm{chg}} = 1 - \eta_{\mathrm{chg}}$
$\epsilon_{\mathrm{dis}}$	Discharge-side loss fraction: $\epsilon_{\mathrm{dis}} = 1 - \eta_{\mathrm{dis}}$
$\gamma$	Terminal-SoC replacement price (currency/kWh), from the engine
$p_{\mathrm{soc}}$	SoC penalty cost: $\max(p_{\mathrm{imp}}) \times 100$
$p_{\mathrm{fuse}}$	Fuse penalty cost: $\max(p_{\mathrm{imp}}) \times 100$ (same magnitude as SoC)
$p_{\mathrm{ev_pen}}^{(v)}$	EV deadline penalty for EV v: $\max(p_{\mathrm{imp}}) \cdot \max(\mathrm{energy_needed}, 1.0) \cdot 10$
$\beta_{\mathrm{ev}}^{(v)}$	EV charge-past-target benefit for EV v: `future_value_per_kwh` — avoided-future-import valuation (issue #630), or a $0.0001$ per kWh AC fallback tiebreaker when no future price data is available

Plus EV pre-deadline benefit (undiscounted, per EV $v$ with deadline, slots $t \leq D_v$):

$$ -\sum_{v=1}^{E} \sum_{t=0}^{D_v} p_{\mathrm{ev_pen}}^{(v)} \cdot \mathrm{ev_c}_v[t] $$

This direct benefit on pre-deadline slots ensures the LP always prefers charging over paying the deadline penalty. Post-deadline slots ($t > D_v$) have zero coefficient unless charge_past_target=True.

Plus EV charge-past-target benefit (discounted, per charge-past-target EV $v$):

$$ -\sum_{v \in \mathrm{past_target}} \sum_{t} \delta_t \cdot \frac{\beta_{\mathrm{ev}}^{(v)}}{\eta_{\mathrm{charger}}^{(v)}} \cdot \mathrm{ev_c}_v[t] $$

$\beta_{\mathrm{ev}}^{(v)}$ is EVConfig.future_value_per_kwh: the avoided cost of importing the same energy later, computed as confidence_factor × mean(import_price) over the next 24 hours (ev_future_charge_value_per_kwh in candidate_selector.py, mirroring replacement_price_from_next_discharge for the house battery's terminal SoC). confidence_factor defaults to 0.9 and is configurable per EV (hsem_ev_past_target_confidence_factor / hsem_ev_second_past_target_confidence_factor) to discount for uncertainty in whether the EV will actually need the extra energy before its next charge.

Because $\beta_{\mathrm{ev}}^{(v)}$ is priced in the same currency units as $p_{\mathrm{imp}}$ and $p_{\mathrm{exp}}$, charge-past-target EV charging competes fairly against both house battery charging and grid export — whichever has the higher genuine avoided-cost value wins the surplus for that slot. When no future price data is available (future_value_per_kwh is None), the MILP falls back to a tiny fixed tiebreaker ($0.0001$ per kWh AC) so surplus PV still prefers the EV over being wastefully curtailed/exported at near-zero or negative prices.

Constraints

Equality constraint — energy balance per slot

For each slot $t$:

$$ gi[t] + pv[t] + ed[t] \cdot \eta_{\mathrm{dis}} = \operatorname{base_load}[t] + \frac{ec[t]}{\eta_{\mathrm{chg}}} + ge[t] + \sum_{v=1}^{E} \frac{\operatorname{ev_c}_v[t]}{\eta_{\mathrm{charger}}^{(v)}} $$

base_load[t] = $\max(\operatorname{net_load}[t], 0)$ — demand the grid/battery must satisfy (kWh)
net_load[t] = avg_house_consumption[t] - solcast_pv_estimate[t] (when EV co-optimisation active)
pv_avail[t] = $\max(-\operatorname{net_load}[t], 0)$ — PV surplus fixed to the pv[t] variable bounds
EV charger efficiency re-scales DC-side charge to AC grid/PV load

Inequality constraints

SoC upper bound (soft):

$$ \sum_{k=0}^{t} \bigl( ec[k] - ed[k] \bigr) - \mathrm{s_max_pen}[t] \leq C_u - soc_0 $$

SoC lower bound (soft):

$$ -\sum_{k=0}^{t} \bigl( ec[k] - ed[k] \bigr) - \mathrm{s_min_pen}[t] \leq soc_0 $$

Mutual exclusion — no simultaneous charge + discharge:

$$ \frac{ec[t]}{\mathrm{max_charge}} + \frac{ed[t]}{\mathrm{max_discharge}} \leq 1 $$

Cycle cost auxiliary — forcing $m[t] \geq ec[t]$ and $m[t] \geq ed[t]$:

$$ -m[t] + ec[t] \leq 0 $$ $$ -m[t] + ed[t] \leq 0 $$

EV cumulative SoC upper bound (per EV v):

$$ \sum_{k=0}^{t} \mathrm{ev_c}_v[k] \leq \mathrm{capacity}_v - \mathrm{initial_soc}_v $$

EV deadline target (soft, per EV v):

$$ \mathrm{initial_soc}_v + \sum_{k=0}^{D_v} \mathrm{ev_c}_v[k] + \mathrm{ev_pen}_v \geq \mathrm{target}_v $$

EV post-deadline zero-charge (hard, per EV v with deadline and charge_past_target=False):

$$ \mathrm{ev_c}_v[t] = 0 \quad \forall, t > D_v $$

This hard constraint prevents any EV charging after the deadline unless charge_past_target=True (surplus-PV-only mode).

EV surplus-only constraint (per charge-past-target EV v, per slot t):

$$ \frac{\mathrm{ev_c}_v[t]}{\eta_{\mathrm{charger}}^{(v)}} \leq \max\bigl(0,; \mathrm{pv_avail}[t] - \mathrm{base_load}[t]\bigr) $$

This constraint ensures charge-past-target EVs only consume genuine PV surplus — never battery discharge or grid import. It is added for EVs where charge_past_target=True (EV already at user-configured target SoC but allow_charge_past_target_soc is enabled and SoC < 100 %). The house battery charges first (benefit ~$p_{\mathrm{imp}}$), then export at good prices (benefit $p_{\mathrm{exp}}$), and only when both are saturated does the EV get the remaining surplus.

Main fuse grid import limit (soft):

For each slot $t$, when main_fuse_amps > 0:

$$ gi[t] - \mathrm{gi_pen}[t] \leq \frac{\mathrm{amps} \times 230 \times 3}{1000} \times \frac{\mathrm{interval_minutes}}{60} $$

The penalty variable gi_pen[t] absorbs any excess at high cost (p_fuse), preventing infeasibility when house base load alone exceeds the fuse rating. When main_fuse_amps is None or 0, this constraint is not added.

Where $D_v$ is the deadline slot index for EV v.

Post-processing

After solving, the solution is decoded into slot recommendations:

flowchart TD
    A[LP solution: ec, ed, gi, ge]
    B{ec > threshold AND ed > threshold?}
    C[Numerical tolerance conflict: resolve by net profit]
    D{Round-trip profitable?}
    E[Keep ec, zero ed]
    F[Zero both]
    G{ec > threshold?}
    H{PV surplus available?}
    I[BatteriesChargeSolar]
    J[BatteriesChargeGrid]
    K{ed > threshold?}
    L{Grid export > 0 AND price >= min_export_price?}
    M[ForceBatteriesDischarge]
    N[BatteriesDischargeMode]
    O[Write EV charge decisions]
    P[Recompute estimated_net_consumption and estimated_cost per slot]
    Q[Compute penalty violation diagnostics]

    A --> B
    B -->|Yes| C --> D
    D -->|Yes| E --> G
    D -->|No| F --> G
    B -->|No| G
    G -->|Yes| H
    H -->|Yes| I --> O
    H -->|No| J --> O
    G -->|No| K
    K -->|Yes| L
    L -->|Yes| M --> O
    L -->|No| N --> O
    K -->|No| O
    O --> P --> Q

EV charging fields written to slots

Field	Source
`ev_planned_load_kwh`	AC load added when `base_load_includes_ev` is `False`
`ev_accounted_load_kwh`	AC load when already captured in house consumption
`ev_total_planned_load_kwh`	Total AC load (sum of planned + accounted)
`ev_charger_calculated_power`	Target AC power (W) for primary EV
`ev_second_charger_calculated_power`	Target AC power (W) for second EV

Engine-level post-processing (after winner selection)

After the MILP (or baseline) winner is selected, the engine runs a final pass over all slots to ensure consistency:

Power recomputation: ev_charger_calculated_power is recomputed from the actual per-slot EV AC load (ev_planned_load_kwh + ev_accounted_load_kwh). For the current (partially elapsed) slot the remaining time is used as the divisor. This ensures the power field always matches the load, even when the baseline candidate wins (bypassing the MILP's own power calculation).
Minimum power floor: If the computed AC power is below charger_min_power_w (default 1380 W = 230 V × 6 A), the charger physically cannot start. The slot's EV fields are zeroed out:
- ev_charger_calculated_power = 0
- ev_planned_load_kwh = 0, ev_accounted_load_kwh = 0, ev_total_planned_load_kwh = 0
- recommendation cleared if it was ev_smart_charging
- estimated_net_consumption_kwh and estimated_cost_currency recomputed without EV load
EV plan rebuild: When the MILP wins, the EVChargingPlan objects (used by the ev_optimal_charging_plan sensor) are rebuilt from the winning slots via rebuild_ev_plan_from_slots(). This ensures the sensor displays the MILP's actual decisions, not the EV planner's pre-MILP estimate.

Assumptions

Linear relaxation: Binary charge/discharge flags are relaxed to continuous because the mutual-exclusion constraint and per-slot power caps already prevent simultaneous charge + discharge in the optimal solution.
Deterministic inputs: All forecasts (prices, PV, load) are treated as known with certainty — no stochastic programming.
Cycle cost proxy: The m[t] = max(ec[t], ed[t]) formulation counts the larger of charge or discharge per slot, matching the 2× denominator in the cycle cost formula.
Time discount: The objective uses exponential discounting with time_discount_rate^hours_ahead to match the selector's discounted score.
Export price clamping: Negative export prices and prices below min_export_price are clamped to 0 before solving, reflecting physical inverter behaviour.
Terminal-SoC credit: Undiscounted in the objective, matching the cost function's terminal_soc_value.

Solver configuration

Parameter	Value	Rationale
Method	`highs`	scipy's HiGHS is the only supported LP method
Timeout	2.0 s	Covers 192-slot (768+ variable) problems where preprocessing reaches 200-400 ms
`pv[t]` bounds	`(pv_avail[t], pv_avail[t])`	Fixed — PV surplus is not chosen by the LP

Fallback

If scipy is unavailable, usable_kwh ≤ 0, or the solver fails (crash, timeout, or non-success status), solve_milp() returns None. The engine silently drops the MILP candidate and the heuristic candidates compete as normal. Pickup is be measured via the hsem_plan_origin metric: milp when the LP succeeds, rule_based otherwise.

HSEM Documentation

Quick Reference

Home — User-facing overview: features, FAQ, working modes, battery schedules, excess export, consumption sensors
Battery Charging Economics — How to calculate the minimum charging price for a battery schedule
Architecture Overview — System context, layered architecture, module map, planning pipeline
Planner Specification — Normative — all planner invariants, rules, and constraints
Planner Technical Guide — How the planner works with worked examples
Cost Function Math — Complete mathematical formulation of the 8-term cost function
Energy Accounting — Physical energy flow model, SoC simulation, efficiency math
Candidate Generation — How candidates are generated, assumptions, partial-SoC
MILP Optimization — Full LP formulation, variable layout, constraints, and solver pipeline
Consumption Prediction — Weighted-average model, IQR outlier detection, spike suppression
Safety Modes — Degraded mode, read-only gate, write-verify applier, runtime resolver
Price Scaling — EDS price scaling, eds_share conversion factor
Services Reference — All 4 HSEM services with examples
Sensors Reference — Complete entity reference: all sensor, select, switch, number, and time entities
Dashboard Setup — Step-by-step ApexCharts dashboard with full YAML, layout reference, and troubleshooting
Config Flow Reference — Every config/options flow step and field
EV Charge Plan Setup — EV planned load configuration guide
EV Surplus Charging Automation — Wire your physical EV charger (go-e, Easee, Zaptec) to follow HSEM surplus recommendations
EV Optimal Charging Template — Legacy Home Assistant template sensor for cost-optimal EV charging
Forecast Accuracy Tracking — Forecast vs actual tracking system
Huawei Entities — Canonical HA entity ID reference
Troubleshooting Guide — Diagnose and fix common problems: missing data, wrong prices, write failures, battery behaviour
Quality Checks — Static quality tools and CI configuration

Uh oh!

milp optimization

HSEM MILP Optimization

Pipeline

Variable layout

Base battery variables (8 × n slots)

EV co-optimization extension

Fuse constraint extension (issue #567)

Objective function

Constraints

Equality constraint — energy balance per slot

Inequality constraints

Post-processing

EV charging fields written to slots

Engine-level post-processing (after winner selection)

Assumptions

Solver configuration

Fallback

Related

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HSEM Documentation

Quick Reference

Architecture Decision Records

Clone this wiki locally