Skip to content

feat: multi-and-per-flow rates schema (v2.2.0)#65

Merged
damonrand merged 7 commits into
mainfrom
feature/multi-and-per-flow-rates
May 10, 2026
Merged

feat: multi-and-per-flow rates schema (v2.2.0)#65
damonrand merged 7 commits into
mainfrom
feature/multi-and-per-flow-rates

Conversation

@damonrand
Copy link
Copy Markdown
Contributor

Summary

Three additive schema features for simulate.yaml. All backward-compatible — existing configs unchanged. Closes the structural Axle-premium leak on two-MPAN BSCP550 sites and removes cut-and-paste duplication for multi-settlement workflows.

  • enabled: false on a simulation drops it pre-schema, so externally-managed scenarios (e.g. Monte-Carlo runs whose outputs are produced outside skypro) can sit alongside live ones in the same yaml. Removes the rebuilder's temp-yaml staging workaround.
  • Per-flow imbalanceDataSourceOverride inside RatesFiles flows. Each flow can override the rates-block-level imbalanceDataSource via a new dict shape; legacy list shape stays valid. Closes the HMCE Apr-26 BSCP550 ~£500–700/mo over-attribution where site-MPAN flows absorbed Axle premium intended only for BESS-MPAN flows.
  • Multi-finals declares N settlement variants per simulation. Mutually exclusive with the legacy final: field (peak/peaks precedent). Fanned out at parse time into one expanded sim per variant; CSV paths get an automatic variant suffix when not using $_SIM_NAME.

Commits (preserve as merge commit, not squash, so CHANGELOG SHAs resolve)

  • 7b7caa6 docs: design briefs for the two non-trivial changes
  • ea2d492 feat: accept enabled: false on simulations
  • ca77995 feat: per-flow imbalanceDataSource override on RatesFiles
  • 2188e35 feat: multi-final fan-out
  • c54c7cc chore: bump version to 2.2.0
  • 3502780 docs: CHANGELOG entry + CLAUDE.md schema reference + mark proposals IMPLEMENTED

Compatibility

  • Legacy single-final configs unchanged. Verified by integrationTestPriceCurve, integrationTestPriceCurveMultiPeak, integrationTestPerfectHindsightLP — bit-identical LP output within tolerance 0.01.
  • Legacy list-shape rate files continue to parse and behave identically.
  • ratesDB source rejects per-flow overrides with a clear error (override only supported with the YAML files source).

YAML reference

# 1. enabled:false annotation
simulations:
  hmce.202512.imb.mc-reopt:
    enabled: false                # dropped pre-schema, outputs preserved as-is

# 2. per-flow imbalance override (apply symmetrically to live + final)
files:
  gridToBatt: [ a.json, b.json ]                # legacy — inherits block-level
  solarToGrid:                                  # dict shape with override
    rates: [ a.json, b.json ]
    imbalanceDataSourceOverride: *imbSrc_plain

# 3. multi-finals fan-out (mutually exclusive with `final:`)
rates:
  live:  *ratesLive_basecase
  finals:
    fullfcl:      *ratesFinal_fullfcl
    trio_imbflex: *ratesFinal_trio_imbflex

Test plan

  • All 48 unit + integration tests pass (PYTHONPATH=src python -m unittest discover --start-directory src)
    • 34 pre-existing — bit-identical LP output, schema regression-checked
    • 4 new: FlowFiles polymorphic shape (legacy list, dict no-override, dict with override, sibling defaults)
    • 10 new: parse-time fan-out (variant suffix, expansion, deep-copy isolation, declaration-order preservation, legacy no-op)
  • Local install via pip install -e . exercised on a real HMCE scenario; output matched the pre-2.2.0 baseline once a stale market-data refresh was identified as the apparent diff source.
  • Validated the schema cheatsheet in CLAUDE.md against the actual YAML form by parsing test fixtures.

Out of scope (follow-ups, not in this PR)

  • Removing the rebuilder's _stage_simulate_yaml_for_skypro() workaround in skypro-service — needs to wait for 2.2.0 on PyPI, then skypro-service bumps the dependency.
  • Hashing imbalance-CSV directories in the rebuilder's data_hash (separate skypro-service ergonomic gap surfaced during release-day investigation; market-data refresh silently invalidates LP outputs without staleness detection).
  • The optimised single-dispatch / multi-settlement-column variant of multi-finals (deferred — see updated docs/proposals/multi-final-rates.md).

Release plan

After merge:

  1. git checkout main && git pull
  2. git tag -a v2.2.0 -m "v2.2.0 — multi-and-per-flow rates schema"
  3. git push origin v2.2.0
  4. Build (rm -rf dist/ && python -m build) and publish (uv publish dist/* --token "$(cat ~/.simt/pypi.token)")
  5. Smoke-test pip install --upgrade skypro && skypro --version

Damon Rand added 7 commits May 9, 2026 18:54
Both originated from sims work in skyprospector.com/skypro-service
(NCESF + HMCE Axle integration). HMCE 202604 is the validation case
for both — its rebuild today landed with known biases that these
two changes close.

- per-flow-imbalance-source-override: lets BSCP550 wire BESS flows
  to Axle-stacked imbalance and site flows to plain imbalance within
  a single rates block. Closes a ~£500-700/mo over-attribution bias
  on HMCE Apr-26 BSCP550 scenarios where Axle currently leaks onto
  gridToLoad and solarToGrid.

- multi-final-rates: lets one simulation run produce N parallel
  ratesFinal column-sets against the same dispatch. Avoids re-running
  the optimiser to compare settlement structures (e.g. Axle on vs
  off, Trio vs flat, OSAM on/off). Eliminates the standalone Impr0
  scenario pattern.

Both are ~3-4 hour PRs. Independent — can ship in either order.
…ation

Scenarios marked `enabled: false` in simulate.yaml are now dropped before
marshmallow validation, mirroring the rebuilder's existing skip logic. This
lets externally-managed scenarios (e.g. Monte-Carlo runs whose outputs are
produced outside skypro) coexist with live scenarios in the same
simulate.yaml — no need for the rebuilder to write a temp staged yaml that
scrubs them out.

- parse_config.py: pre-load filter walks simulations dict and removes
  entries with `enabled is False` before Config.Schema().load
- report-side parser unchanged: report.yaml has no enabled annotations
  in any current tenant config

Existing 34 unit + integration tests pass unchanged.
Closes the structural Axle-premium leak on two-MPAN BSCP550 sites where the
BESS and site MPANs settle against different imbalance signals. Previously
every flow in a rates block read from the block-level `imbalanceDataSource`,
so site-MPAN flows (gridToLoad, solarToGrid) absorbed any premium intended
only for the BESS-MPAN flows (gridToBatt, battToGrid).

Schema (backward-compatible — legacy list shape still parses):
- New `FlowFiles` dataclass: `{rates: [paths], imbalanceDataSourceOverride: ...}`
- New `FlowFilesField` marshmallow field accepts either
    `gridToBatt: [a.json, b.json]`             # legacy
  or
    `gridToBatt: {rates: [a.json], imbalanceDataSourceOverride: {price: ..., volume: ...}}`
- `RatesFiles` flows now typed `FlowFilesType` (always FlowFiles after parsing)

Plumbing:
- `parse_vol_rates_files_for_all_energy_flows` accepts
  `flow_imbalance_pricings: Optional[Dict[str, pd.Series]]` keyed by flow name;
  cache key now `(files_str, id(pricing))` so flows with same files but
  different overrides don't share rate instances.
- `_get_rates_from_config` collects unique override sources, fetches +
  normalises each (final-style for final-block overrides, live-style for
  live-block), passes per-flow pricing dict down. Block-level df continues
  to drive the canonical `imbalance_volume_*` output columns.
- Per-flow override + ratesDB combination rejected with a clear error.
- Override sources contribute to the flowsMarketData engine-needed gate.

Multiplier and OSAM safety:
- MultiplierVolRate.set_all_rates_in_set already operates per-flow, so
  `Statkraft × imbalance` resolves to whatever imbalance ShapedVolRate is
  in its flow's set — verified, no change needed.
- OSAM NCSP is dispatch-driven and computed once after the algo runs — per-flow
  overrides don't touch it.

Report-side parser unchanged: `commands/report/rates.py` still passes a single
`imbalance_pricing` and per-flow override dict defaults to `{}`.

Tests: 4 new unit tests cover legacy list shape, dict shape with/without
override, and sibling-flow defaulting. Existing 34 tests pass unchanged.
…lation

Lets a single simulation declare multiple `finals: {<name>: Rates, ...}` and
get N output CSVs back, each settling the same dispatch under a different
final-rates structure. Removes the cut-and-paste duplication for
fullfcl-vs-trio-imbflex-style scenario pairs.

Schema (mutually exclusive with the legacy `final: Rates` field, mirrors
the `peak`/`peaks` precedent in PriceCurveAlgo):

  rates:
    live:  *ratesLive_basecase
    finals:
      fullfcl:      *ratesFinal_fullfcl
      trio_imbflex: *ratesFinal_trio_imbflex

Implementation: parse-time fan-out in `parse_config`. Each multi-final sim is
replaced with N entries `<orig_name>.<variant_name>`, each carrying a deep-copied
SimulationCase with `rates.final` resolved to that variant. The simulator main
loop never sees a multi-final scenario — zero changes to `_run_one_simulation`,
`_get_rates_from_config`, `_process_final_rates`, or `generate_output_df`.
The optimiser runs N times; accepted trade-off for YAML ergonomics over compute
saving (deferred — see proposals/multi-final-rates.md for the column-multiplex
alternative).

CSV path de-clashing:
  - Paths containing `$_SIM_NAME` are left alone — the substitution loop runs
    after fan-out and naturally produces unique paths via `<orig>.<variant>`.
  - Hardcoded paths get `.<variant>` inserted before the extension to avoid
    two variants overwriting the same file.

Composes orthogonally with commit 2 — each variant is a complete `Rates` block
and can carry its own per-flow `imbalanceDataSourceOverride`.

Tests: 10 new unit tests cover variant suffix logic, fan-out expansion, deep-
copy isolation, declaration-order preservation, and the legacy single-final
no-op path. All 48 tests (34 existing + 4 FlowFiles + 10 fan-out) pass.
- pyproject.toml: 2.1.1 → 2.2.0 (minor bump for the three additive schema features
  on feature/multi-and-per-flow-rates)
- CLAUDE.md merge log: summary entry for the v2.2.0 changes

Three commits in this minor release:
  - feat: accept enabled:false on simulations (ea2d492)
  - feat: per-flow imbalanceDataSource override on RatesFiles (ca77995)
  - feat: multi-final fan-out — declare N settlement variants in one simulation (2188e35)

All 48 unit + integration tests pass. Backward-compatible — legacy list-shape
flows and single-`final` blocks unchanged.
- CHANGELOG.md: Added v2.2.0 entry with the three schema additions
  (enabled:false, per-flow imbalanceDataSourceOverride, multi-finals),
  YAML examples, and compatibility notes.
- CLAUDE.md: Added a "simulate.yaml schema reference (post-v2.2.0)"
  cheatsheet under Key Concepts, summarising all three features with
  example YAML and the symmetry rules (apply override to live AND final;
  finals is mutually exclusive with final).
- docs/proposals/per-flow-imbalance-source-override.md: Marked as
  ✅ IMPLEMENTED in v2.2.0, commit ca77995.
- docs/proposals/multi-final-rates.md: Marked as ✅ IMPLEMENTED in
  v2.2.0, commit 2188e35 — with note that the shipped fan-out approach
  differs from the brief's column-multiplex design (deferred — user
  prioritised YAML ergonomics over compute saving).
Leftover from an earlier draft that used copy.deepcopy. The current test file
uses SimpleNamespace mocks throughout. Caught by ruff in CI.
@damonrand damonrand merged commit 575b47e into main May 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant