Skip to content

Spatial complexity reduction for the run pipeline (partial) - #706

Merged
MoritzSchloesser merged 8 commits into
edisgo_run_edisgofrom
705-Integrate-spatial-complexity-reduction-into-the-run-pipeline
Jul 15, 2026
Merged

Spatial complexity reduction for the run pipeline (partial)#706
MoritzSchloesser merged 8 commits into
edisgo_run_edisgofrom
705-Integrate-spatial-complexity-reduction-into-the-run-pipeline

Conversation

@MoritzSchloesser

Copy link
Copy Markdown
Collaborator

Partial solution to #705 (spatial complexity reduction pipeline
integration — see docs_notes/issue_spatial_complexity_reduction_pipeline_integration.md
for the full issue writeup).

Summary

Adds spatial complexity reduction to the eDisGo run pipeline, as a
counterpart to the existing temporal complexity reduction
(select_timesteps). Spatial reduction merges nearby buses into
representative buses to shrink the grid the OPF runs on; reinforcement
still runs on the full topology.

  • New spatial_reduce / spatial_restore tasks bracketing optimize.
    spatial_reduce stashes a deepcopy of the full grid and spatially
    reduces the working object; spatial_restore writes the optimized
    flexible-component dispatch (charging points, heat pumps, DSM loads,
    storage) back onto the stashed full grid and recomputes reactive power.
    Both are no-ops when spatial_reduction.enabled is false.
  • New core function apply_reduced_results_to_full_grid +
    EDisGo.map_reduced_results_to_full_grid, mirroring the existing
    spatial_complexity_reduction/EDisGo.spatial_complexity_reduction
    pairing.
  • Validator integration via the existing requires/provides capability
    system (no new validator machinery): spatial_reduce provides
    reduced_grid, optimize additionally provides optimized_dispatch,
    spatial_restore requires both.
  • New top-level spatial_reduction: config block, mirroring
    timeseries_selection: exactly — same shape, same eGo injection pattern
    (global default + per-grid override).
  • New preset uc6_spatial_reduction.yaml and eGo example
    scenario_setting_uc6_example.json.
  • eGo: _build_run_edisgo_config injects spatial_reduction/
    spatial_reduction_per_grid the same way it already injects
    timeseries_selection.

This PR covers aggregation_mode=False only. aggregation_mode=True
has a known, documented gap unrelated to this PR's own logic — see
"Known limitation" below and the linked issue.

Testing

  • tests/tools/test_spatial_complexity_reduction.py::TestApplyReducedResultsToFullGrid
    — by-name write-back, disaggregation-sum exactness, the numpy-array-input
    regression, the time-index-mismatch guard, and reactive-power recompute,
    all against stub OPF results (no Julia/Gurobi dependency).
  • eGo/tests/tools/test_edisgo_integration.py — the
    spatial_reduction/spatial_reduction_per_grid injection logic.
  • Verified end-to-end against a real grid (32377), both directly through
    eDisGo (run_example_06.py) and through eGo
    (scenario_setting_uc6_example.json): spatial_reduce → optimize (Gurobi) → spatial_restore → reinforce → save all completed successfully.
  • analyse_spatial_reduction.ipynb (not part of this PR, kept local) drives
    the pipeline task-by-task and confirms the reduced grid's OPF output
    exactly matches the full grid's post-restore value for every flexible
    component (0/388 mismatches on the test run).

Known limitation (tracked, not fixed in this PR)

aggregation_mode=True merges loads at the same bus and correctly
aggregates loads_active_power, but does not aggregate
electromobility.flexibility_bands — the OPF's charging-point constraint
builder reads that separately, keyed by charging-point name, so a merged
representative has no upper_power entry and optimize raises KeyError
before spatial_restore ever runs. Only manifests with aggregation_mode: true and 2+ charging points sharing a bus. Full writeup with the exact
traceback and suggested fix in
docs_notes/issue_aggregation_mode_flexibility_bands_not_aggregated.md.
aggregation_mode: false is the default in the new preset and unaffected.

Also included

  • Fix: build_flexibility_bands now trims electromobility.flexibility_bands
    to the active time index right after building it — it previously only
    resampled to the active frequency, not the active date range, so a
    manual/short select_timesteps window left bands spanning the full
    underlying data range with nothing ever trimming them down. Unrelated to
    spatial reduction directly, but was blocking end-to-end testing of it
    (see docs_notes/issue_temporal_reduction_flexibility_bands.md).
  • Fix: flexible_* or [] in apply_reduced_results_to_full_grid raised
    ValueError on numpy-array input (task_optimize derives
    flexible_loads as edisgo.dsm.p_min.columns.values, not a list) —
    found on the first live end-to-end run.

Not in this PR

  • aggregation_mode=True fix (tracked issue, linked above).
  • Pipeline/task-level integration tests for spatial_reduce/
    spatial_restore (current scope is core-function-only, by design — see
    docs_notes/spatial_reduction_grilling_session.md).
  • Two ADR candidates flagged during design ("two bracketing tasks, restore
    optional"; "disaggregation by pre-OPF flexibility envelope") — not yet
    written up.

MoritzSchloesser and others added 8 commits July 14, 2026 11:43
Adds spatial_reduce/spatial_restore tasks bracketing optimize, so the OPF
can run on a spatially-reduced grid while reinforcement still runs on the
full topology. New core function apply_reduced_results_to_full_grid maps
optimized flexible-component dispatch (charging points, heat pumps, DSM
loads, storage) back onto the full grid, by name or disaggregated onto
old_name members per time step (weighted by each member's pre-OPF
flexibility envelope) when aggregation_mode=True. Reactive power is
recomputed on restore, mirroring how pm_optimize itself handles it.

optimize now also records its flexible-component name lists on ctx.flags
and declares provides={"optimized_dispatch"}, so the validator's existing
requires/provides system enforces spatial_restore running after both
spatial_reduce and optimize.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
get_flexibility_bands only resamples bands to the active time-series
FREQUENCY, not the active date range - so after a manual/short
select_timesteps window, flexibility_bands kept spanning the full
underlying SimBEV data range with no later step ever trimming it down.
Any consumer indexing flexibility_bands by edisgo.timeseries.timeindex
(e.g. the OPF's charging-point constraint builder) could then hit missing
time steps. build_flexibility_bands now trims electromobility data to the
active time index right after building the bands.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Standalone preset wiring spatial_reduce/spatial_restore around optimize,
disabled by default via spatial_reduction.enabled. run_example_06.py runs
it against a real ding0 grid.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CONTEXT.md gains a glossary section for spatial reduction terms (busmap,
disaggregation rule, config surface, ordering). docs_notes/ carries the
full design-session record plus two follow-up issue write-ups: the
flexibility-bands time-index gap (fixed) and the aggregation_mode=True +
charging-points gap (not yet fixed, tracked for later).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The preset's header comment and self-contained-run example still said
"UC5"/uc5_spatial_reduction from when the file was copied from
uc5_select_timesteps.yaml. Also adds a full issue description covering
what's done and what's still open (aggregation_mode=True + charging
points) for the spatial-reduction pipeline integration.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Hooks weren't run before the original commits landed. Adds the missing
license header to edisgo/run/context.py, applies ruff-format's line
collapsing across a few files, and manually wraps two docstring
:func: cross-references that were still over the line-length limit
after formatting.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@MoritzSchloesser MoritzSchloesser self-assigned this Jul 15, 2026
@MoritzSchloesser
MoritzSchloesser changed the base branch from dev to edisgo_run_edisgo July 15, 2026 15:54
@MoritzSchloesser
MoritzSchloesser merged commit b83c884 into edisgo_run_edisgo Jul 15, 2026
5 of 7 checks passed
MoritzSchloesser added a commit that referenced this pull request Jul 16, 2026
* Change database from OEP to local

* feat: spatial complexity reduction for the run pipeline

Adds spatial_reduce/spatial_restore tasks bracketing optimize, so the OPF
can run on a spatially-reduced grid while reinforcement still runs on the
full topology. New core function apply_reduced_results_to_full_grid maps
optimized flexible-component dispatch (charging points, heat pumps, DSM
loads, storage) back onto the full grid, by name or disaggregated onto
old_name members per time step (weighted by each member's pre-OPF
flexibility envelope) when aggregation_mode=True. Reactive power is
recomputed on restore, mirroring how pm_optimize itself handles it.

optimize now also records its flexible-component name lists on ctx.flags
and declares provides={"optimized_dispatch"}, so the validator's existing
requires/provides system enforces spatial_restore running after both
spatial_reduce and optimize.


* fix: trim electromobility flexibility bands to the active time index

get_flexibility_bands only resamples bands to the active time-series
FREQUENCY, not the active date range - so after a manual/short
select_timesteps window, flexibility_bands kept spanning the full
underlying SimBEV data range with no later step ever trimming it down.
Any consumer indexing flexibility_bands by edisgo.timeseries.timeindex
(e.g. the OPF's charging-point constraint builder) could then hit missing
time steps. build_flexibility_bands now trims electromobility data to the
active time index right after building the bands.


* Add uc6_spatial_reduction preset and example runner

Standalone preset wiring spatial_reduce/spatial_restore around optimize,
disabled by default via spatial_reduction.enabled. run_example_06.py runs
it against a real ding0 grid.


* docs: spatial reduction glossary and grilling-session notes

CONTEXT.md gains a glossary section for spatial reduction terms (busmap,
disaggregation rule, config surface, ordering). docs_notes/ carries the
full design-session record plus two follow-up issue write-ups: the
flexibility-bands time-index gap (fixed) and the aggregation_mode=True +
charging-points gap (not yet fixed, tracked for later).


* Change overlying_grid path

* Fix stale UC5 naming in uc6_spatial_reduction.yaml, add issue writeup

The preset's header comment and self-contained-run example still said
"UC5"/uc5_spatial_reduction from when the file was copied from
uc5_select_timesteps.yaml. Also adds a full issue description covering
what's done and what's still open (aggregation_mode=True + charging
points) for the spatial-reduction pipeline integration.


* style: apply pre-commit hooks to the spatial-reduction commits

Hooks weren't run before the original commits landed. Adds the missing
license header to edisgo/run/context.py, applies ruff-format's line
collapsing across a few files, and manually wraps two docstring
:func: cross-references that were still over the line-length limit
after formatting.


---------
joda9 added a commit that referenced this pull request Jul 29, 2026
…0.3.0

The changelog stopped at the documentation restructuring and covered none of
the work merged since: the whole edisgo.run pipeline framework (#731), the
spatial complexity reduction integration (#706), the OPF rework towards
feasible non-optimal solutions and per-interval solving, the reworked database
access, the get_flexibility_bands time-index scoping and the SPDX license
headers (#607).

Also sets the release date.
joda9 added a commit that referenced this pull request Aug 3, 2026
* docs(whatsnew): document the run pipeline, OPF and database work for 0.3.0

The changelog stopped at the documentation restructuring and covered none of
the work merged since: the whole edisgo.run pipeline framework (#731), the
spatial complexity reduction integration (#706), the OPF rework towards
feasible non-optimal solutions and per-interval solving, the reworked database
access, the get_flexibility_bands time-index scoping and the SPDX license
headers (#607).

Also sets the release date.

* chore(release): 0.3.0 metadata, authors and citation info

- setup.py: version 0.3.0 (was 0.3.0dev, which normalised to 0.3.0.dev0), and
  fill in the metadata PyPI has been missing entirely: python_requires
  (>=3.10,<3.13 -- numpy is pinned to 1.26.4, which has no 3.13 wheels, and
  the test matrix covers 3.10 to 3.12), trove classifiers and project_urls.
- authors: Jonas Danke and Moritz Schlösser are maintaining eDisGo now, so
  they lead the author field and are named as maintainers; the previous
  contributors stay listed. Maike Held was missing from it entirely despite
  being the second-largest contributor.
- AUTHORS.md and CITATION.cff are new. Neither existed, so there was no
  complete credit list anywhere outside the git history, and GitHub had
  nothing to render for "Cite this repository".
- .mailmap consolidates the identities people committed under: git counted 43
  distinct authors for 19 actual people. Two commits were authored with Jonas'
  name but Kilian's address, so that entry matches on both fields rather than
  reassigning all of Kilian's commits.
- doc/conf.py: extend the stale copyright year.

Verified: twine check passes on both artifacts, the wheel METADATA carries the
new author/maintainer/classifiers/Requires-Python, and CITATION.cff parses
with all required CFF 1.2.0 fields present.

* Update eDisGo_OPF.jl Manifest for Julia 1.12 compatibility

StaticArrays 1.5.12 (pinned in the old manifest) fails to precompile
on Julia >=1.11 due to Base's AbstractTriangular gaining a type
parameter. Pkg.update() resolved newer, compatible versions of
StaticArrays and its dependents (JuMP, PowerModels, MathOptInterface,
Gurobi.jl, etc.).

* Bump Julia to 1.12 in CI, compat and install docs

The updated eDisGo_OPF.jl Manifest was resolved with Julia 1.12 and pins
stdlibs that do not exist in earlier versions (e.g. JuliaSyntaxHighlighting,
StyledStrings, LinearAlgebra 1.12). Main.jl runs Pkg.instantiate() on that
checked-in manifest, so the CI job pinned to Julia 1.6 could not resolve it
and test_pm_optimize failed.

Keeping both 1.6 and 1.12 working is not possible with a single checked-in
manifest, so the declared Julia version is raised everywhere it appears.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant