Skip to content

docs: add memory best-practices notebook for large-scale models#624

Open
MaykThewessen wants to merge 4 commits intoPyPSA:masterfrom
MaykThewessen:docs/memory-best-practices
Open

docs: add memory best-practices notebook for large-scale models#624
MaykThewessen wants to merge 4 commits intoPyPSA:masterfrom
MaykThewessen:docs/memory-best-practices

Conversation

@MaykThewessen
Copy link

Summary

Adds a notebook documenting memory-efficient patterns for large-scale linopy models, as invited by @FBumann in #599.

Closes #599 (partial — notebook contribution specifically).

Context

The patterns are drawn from production experience with a 200-bus, 1,800-generator PyPSA power system model solved over 8,760 hourly snapshots in 168-hour weekly chunks (~1.4M LP variables and constraints per chunk, ~9 GB steady-state, ~15 GB peak).

Notebook contents (examples/memory-best-practices.ipynb)

  1. Understanding linopy's memory layout — formula table for variable/constraint/sparse-matrix storage; the two memory spike points (constraints.flat during matrices.A and COO→CSC conversion; solution unpacking); how to avoid repeated matrix builds
  2. Chunked time-horizon solving — the basic pattern; why del network.model + malloc_trim(0) / malloc_zone_pressure_relief() is essential (RSS does not drop without the OS-level trim)
  3. LP blueprint reuse across chunks — for models where LP structure is identical across chunks, extract the HiGHS object after chunk 1 and call changeRowsBounds() / changeColsCost() for chunks 2+, skipping linopy rebuild entirely; saves ~50 s and ~9 GB per chunk
  4. Efficient RHS and objective updates — numpy broadcast fill vs pandas fillna to avoid (T×N) intermediate copies; sparse RHS push via keys >= 0 masking
  5. Memory profilingtracemalloc and psutil.rss_gb() patterns; estimate_model_memory() helper for sizing RAM before solving
  6. Explicit cleanup between solvesmatrices.clean_cached_properties() (frees flat_cons cache, typically 2–3 GB); detach solver_model before del; gc.collect() + OS trim
  7. Sparse-friendly constraint formulationfilter_missings=True; groupby aggregation vs dense broadcasting; merge=False for per-snapshot constraints

All code examples are from production code paths and include both the recommended pattern and the anti-pattern it replaces.

Files changed

  • examples/memory-best-practices.ipynb — new notebook
  • doc/memory-best-practices.nblink — nblink entry pointing to the notebook
  • doc/index.rst — added memory-best-practices to the User Guide toctree

@FBumann FBumann added the documentation Improvements or additions to documentation label Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Numba/JIT-compiled constraint matrix assembly for 3-4x faster model building

2 participants