Measure notebook runtimes

[JTS22]

Some of the Jupyter-Notebooks using the new Python Interface (see #22 ) take multiple minutes for a full run-through, which causes the test pipeline to run quite long. This table aims to summarize the current run times (using the notebooks in #22 ) for evaluation and adaption of simulation run times and pipeline tests. The full notebook runtime is measured in a single run of pytest --durations=0 -v , the simulation run times are measured using the iPython %%timeit cell magic with the time calculated as the average over seven runs. Simulation run times contain the full setup and run of the dynamic simulation without PowerFlow or later data evaluation / plotting. Please note that this is not meant as a complete, accurate benchmark but rather as an overview over which notebooks should be tweaked when tested in the CI-pipeline.

Overall

61 notebooks in Circuits, Components or Grids
7 notebooks skipped in pytest(11,5%)
25 notebooks with asserts (41,0%)

Circuit-Examples

Full pytest-run in 760.53s (0:12:40)
26 passed, 2 skipped (92.9% tested)
10 of 28 notebooks with asserts (35,7%)

Notebook	Pytest runtime	Simulation runtime(s)	Asserts	Remarks
Compare_EMT_DP_Slack_PiLine_PQLoad	4.14s	1.73s (EMT), 419 ms (DP)	DP vs EMT
Compare_EMT_SynGenDQ7odTrapez_ DP_SynGenTrStab_SMIB_Fault	Skipped	48.6s (EMT), 4.25s (DP)	-	Notebook contains no error, so test could be enabled
CS_R2CL print_attributes	2.79s	12.5 µs (EMT), 12.5 µs (DP)	-	Notebook exists to show off attribute print functions, so comparisons with SL got stripped
CS_R2CL	3.36s	11.8 µs (EMT), 11.4 µs (DP)	EMT vs SL, DP vs SL
DP_Slack_PiLine_PQLoad_with_PF_Init	2.49s	71.6 µs	-	Plots currently plot the wrong time interval
DP_Slack_PiLine_VSI_Ramp	26.20s	-	-	Notebook does not use dpsimpy, plot labels are in german
DP_Slack_PiLine_VSI_with_PF_Init	47.27s	45.8s	-	wrong time interval plotted, plot labels in german
DP_SP_SynGenTrStab_3Bus_Fault	32.01s	8.94s (DP), 8.52s (SP)	-
DP_SP_SynGenTrStab_3Bus_SteadyState	12.71s	2.28s (DP), 2.11s (SP)	-
DP_SP_SynGenTrStab_SMIB_Fault	17.57s	4.06s (DP), 4.11s (SP)	-
DP_SP_SynGenTrStab_SMIB_SteadyState	12.29s	1.70s (DP), 1.45s (SP)	-
EMT_DP_Slack_PiLine_VSI_ Control_OnOff	34.25s	10.4s (EMT, control on), 11.2s (EMT, control off), 5.04s (DP, control on), 4.71s (DP, control off)	-
EMT_DP_SP_Slack_PiLine_VSI	107.11s	11.1s (EMT), 48s (DP), 47.6s (SP)	-
EMT_DP_VS_Init	3.12s	49.2ms (DP, set params), 88ms (EMT, set params), 45.1ms (DP, set attribute), 74.6ms (EMT, set attribute), 40.7ms (DP, set from node), 83.8ms (EMT, set from node)	EMT vs DP	DP vs EMT assert on set attribute commented out
EMT_DP_VS_RLC	19.35s	1.62s (EMT, VS+R), 518ms (DP, VS+R), 480ms (SP, VS+R), 2.82s (EMT, VS+RL), 785ms (DP, VS+RL), 856ms (SP, VS+RL), 2.79s (EMT, VS+RC), 758ms (DP, VS+RC), 865ms (SP, VS+RC)	EMT vs DP vs SP
EMT_Slack_PiLine_PQLoad_with_PF_Init	2.77s	493ms	-	Plots the wrong time interval
EMT_Slack_PiLine_VSI_with_PF_Init_Params	69.97s	27.9s (Kp variation), 43.7s (Ki variation), 4.33s (final choice)	-
EMT_Slack_PiLine_VSI_with_PF_Init	12.65s	11.3s	-
EMT_SynGenDQ7odTrapez_ OperationalParams_SMIB_Fault_JsonSyngenParams	93.01s	51.5s (no JSON-config), 50.4s (with JSON-config)	JSON-config vs No JSON-config
EMT_SynGenDQ7odTrapez_ OperationalParams_SMIB_Fault_ParamsModification	94.22s	50.8s (first parameter), 52.8s (second parameters)	-
EMT_SynGenDQ7odTrapez_ OperationalParams_SMIB_Fault	93.72s	52.9s (fundamental params), 54.5s (operational params)	Fundamental vs operational params
EMT_SynGenDQ7odTrapez_SMIB_Fault	50.24s	53.6s	-
RL_SourceStep_ParameterStudy	Skipped	-	-	Notebook does not use dpsimpy
SP_Slack_PiLine_VSI_with_PF_Init	3.69s	2.17s	-
VS_CS_R4	3.39s	29.8ms (EMT), 52.4ms (DP)	EMT vs SL, DP vs SL
VS_R2L3	3.47s	33.7ms (EMT), 63.1ms (DP)	EMT vs SL, DP vs SL
VS_RC1	4.22s	19.9ms (EMT), 37.5ms (DP)	EMT vs SL, DP vs SL
VS_RL1	3.77s	21.2ms (EMT), 33.9ms (DP)	EMT vs SL, DP vs SL

Component examples

Full pytest-run in 133.57s (0:02:13)
10 passed, 1 skipped (92.9% tested)
7 of 11 notebooks with asserts (63,6%)

Notebook	Pytest runtime	Simulation runtime(s)	Asserts	Remarks
Diakoptics	2.91s	44ms (VS_CS_R4), 114ms (VS_CS_R4_Diakoptics), 56.3ms (VS_R2L3), 127ms (VS_R2L3_Diakoptics)	-
Inverter_Grid_Test	49.46s	46s	DP vs ML	DP vs SL has no asserts, comparison is only plotted
Line	19.37s	904ms (SP Elements), 1.1s (SP Component), 951ms (DP Elements), 1.04s (DP Component), 2.13s (DP Diakoptics), 4.58s (EMT Elements), 6.14s (EMT Component), 79.9ms (Decoupling Line Elements), 109ms (Decoupling Line Component), 68.4ms (Decoupling Line EMT)	Elements vs Component for DP, SP, EMT; DP vs EMT component
Slack	11.18s	433ms (SP Elements), 774ms (SP Component), 481ms (DP Elements), 850ms (DP Component), 938ms (EMT Elements), 1.27s (EMT Component)	Elements vs Component for DP, SP, EMT; SP vs DP component, DP vs EMT component
SynGen_trStab_logger_test	1.40s	-	-	This example exists to show errors in logger behaviour when multiple simulations interact with each other so timing it does not make sense here
SynGen_trStab	2.15s	8.7ms (SteadyState), 17.8ms (LoadStep)	wr_gen(t=0) vs wr_gen(t=0.1)
SynGenDq7od_ParameterStudy	Skipped	1.81s (DP ODE), 1.57s (EMT ODE), 1.34s (DP Trapez), 1.07s (EMT Trapez)	-	Tries to evaluate files which are not actually generated by the simulation
SynGenDq7od_SteadyState _DP_EMT	7.67s	1.81s (DP ODE), 1.44s (EMT ODE), 1.29s (DP Trapez), 1.07s (EMT Trapez)	-	SL results are loaded and compared to in plots, but there are no asserts
SynGenDq7od_ThreePhFault_DP_EMT	8.15s	1.84s (DP ODE), 1.55s (EMT ODE), 1.25s (DP Trapez), 1.09s (EMT Trapez)	EMT vs SL
SynGenDq7od	6.77s	22.6ms (DP ODE SteadyState), 1.82s (DP ODE Fault), 1.31s (DP Trapez Fault)	DP ODE Fault vs SL, DP Trapez Fault vs SL
Trafo	12.79s	736ms (SP Elements), 950ms (SP Component), 627ms (DP Elements), 851ms (DP Component), 2.03s (EMT Elements), 3.92s (EMT Components)	Elements vs Component for DP, SP, EMT; SP vs DP component, DP vs EMT component

Grid examples

Full pytest-run in 223.14s (0:03:43)
18 passed, 4 skipped (81,8% tested)
8 of 22 notebooks with asserts (36,4%)

Notebook	Pytest runtime	Simulation runtime(s)	Asserts	Remarks
case9	1.80s	14.6ms	-
case14	1.58s	26ms	-
case145	6.60s	895ms	-
case300	8.87s	2.0s	-
CIGRE_MV_pf-interactive-dpsimpy	Skipped	-	-	There is no full simulation, only one step
CIGRE_MV_powerflow_profiles-dpsimpy	Skipped	1.81s	-	Notebook contains no error, so test could be enabled
CIGRE_MV_powerflow-dpsimpy	Skipped	24.4ms	DPsim vs NEPLAN	Notebook contains no error, so test could be enabled
CIGRE_MV_powerflow	2.54s	-	DPsim vs NEPLAN	Non-dpsimpy version of CIGRE_MV_powerflow-dpsimpy, probably obsolete
cigre-mv-pf-profiles-shmem	Skipped	-	-	Does not run at all
DP_CIGRE_MV_withDG_withLoadStep	19.76s	14.3s	-
DP_CIGRE_MV_withDG	17.84s	14.1s	-
DP_CIGRE_MV_withoutDG	6.06s	2.13s	-
EMT_CIGRE_MV_withDG_ withLoadStep	39.09s	36.0s	-
EMT_CIGRE_MV_withDG	38.16s	35.0s	-
EMT_CIGRE_MV_withoutDG	10.26s	3.92s	-
IEEE_LV_powerflow	2.91s	539ms	DPsim vs NEPLAN
PF_CIGRE_MV_withDG	2.52s	105ms	-
SP_SMIB_SynGenTrStab_KundurExample1	19.56s	-	DPsim vs PSAT
SP_WSCC_9-bus_dyn_switch	12.09s	7.38s	DPsim vs PSAT
WSCC_9-bus_dyn_switch	15.98s	9.22s	DPsim vs SL
WSCC_9-bus_dyn	13.18s	9.95s	Last value vs first value for all logged phasors
WSCC_9-bus	3.98s	998ms	Last value vs first value for all logged phasors

[rubensander]

Grid examples

Notebook	Pytest runtime	Simulation runtime(s)	Asserts	Remarks
DP_Slack_PiLine_VSI_Ramp	-	-	DP vs EMT ref	Notebook does not use dpsimpy
EMT_DP_SP_Slack_PiLine_PQLoad_FrequencyRamp_CosineFM	-	-	DP vs EMT, DP vs SP	Notebook does not use dpsimpy

[m-mirz]

Next step is to remove unnecessary examples (especially C++) and reduce the runtime of the remaining ones.