Testing procedure for FMI 2.0 interface and setting/getting state support

[ghorwin]

Slowly the number of FMI 2.0 supporting FMUs increases and quite a few provide the feature of storing/restoring the state. I suggest adding the following test procedure to the compliance checker for Co-Simulation FMUs with FMI 2.0 support:

1. store/restore state check

   • After "exitInitializationMode" ask FMU to store its state via fmi2GetFMUstateXXX (compliance checker remembers handle)
   • Simulation is done for a few steps, results are logged in file 1
   • tell FMU to restore its state to saved state via fmi2SetFMUstate
   • Simulation is done again for same number of steps, results are compared to those stored in file 1 -> exakt match is required (since even multi-step integrators will not have a history at simulation start)

2. multiple state handling checks

   • start simulation, run until say 10% of simulation time
   • store handle 1
   • continue simulation, run until say 20% of simulation time
   • store handle 2
   • continue simulation, run until say 30% of simulation time
   • restore state to handle 2
   • run again until 30% -> compare results -> close match is required in interval 20-30% compared to first run
   • restore state to handle 1
   • run again until 30% -> compare results -> close match is required in interval 20-30% compared to first run

Note about "close match" requirement:
See discussion about restoring state of error controlled, multi-step methods: When the stored state does not include full history, or when last step was not synchronized with communication interval end - the continued simulation will likely have different results; for example, the simulation could be continued with a "cold-start", i.e. method order 1, smallest time step, hereby taking completely different steps within the communication interval as in the first iteration, thus yielding (slightly) different results.

3. Serialization test (only for FMUs that support this; mainly necessary to check for correct re-initialization of external resources being used by the FMU)
   • run full simulation once
   • re-init (completely from scratch)
   • start simulation, run until say 10% of simulation time
   • retrieve FMU state and store to file
   • re-init (completely from scratch)
   • read FMU state from file and set state in FMU
   • continue simulation until end -> close match is required for results in both runs

At last, some consistency checks - may require use of valgrind or other memory leak checkers.

4. (optional) memory allocation/consistency check:
   Motivation: often, problems with resource leaks are not detected in small test cases. However, when an iterative master algorithm is used for a whole simulation, the get/set-state procedure may be repeated many 1000 times, and even a small mistake in memory/resource allocation will cause problems here.
   • procedure 1 (with short simulation interval in between) is repeated at n times, with intermittend calls to fmi2FreeFMUstateXXX
   • memory footprint of FMU model should not grow uncontrolled (a growth of memory footprint would indicate a resource leak) -> alternatively check with valgrind or other leak checker