FMU Conformance to FMI 2.0 for CS

[CThuleHansen]

PLEASE ADD LUND LABEL IF CAPABLE

What

Systematic testing of the cross-check FMUs in context of FMI 2.0 for co-simulation has revealed several shortcomings of the FMU implementations and the static description files.
The conducted systematic testing consisted of:

• A static check of the model description files
• A model-based testing of the dynamic behaviour of FMUs

A static check of the model description files implemented with VDM-SL

Some examples
invalid InitialUnknowns (missing required unknowns, or too many unknowns)
ScalarVariable inconsistencies with causality, variability, initial and start attributes

Tool is available
https://github.com/INTO-CPS-Association/FMI2-VDM-Model/releases

To be presented at
The 17th Overture Workshop, October 7. (Nick Battle, Casper Thule, Cláudio Gomes, Hugo Daniel Macedo and Peter Gorm Larsen. “Towards Static Check of FMUs in VDM-SL”)
Pre-proceedings available: https://github.com/overturetool/overturetool.github.io/raw/master/workshops/17/overture-2019-pre-proceedings.pdf
Some Results
– There are 692 FMUs in the 2.0 branch of the repository (some are duplicated for different architectures and tools)
– 294 of them pass without any @OnFail messages (ie. isValidFMIModelDescription returns true)
– 123 of them have malformed structured ScalarVariable names
– 118 of them have invalid InitialUnknowns (missing required unknowns, or too many unknowns)
– 79 of them have ModelStructure Derivatives indexes that do no match Real/derivative variables
– 67 of them have malformed floating point variables (eg. "1." or "4.e-10")
– 56 of them have aliased variables that do not all have the same units
– 33 of them have ScalarVariable attribute inconsistencies
– 27 have a Derivatives section but no derivative variables
– 24 have the reinit flag set for co-simulation models
– 18 have derivative variables defined but no Derivatives section
– 14 have Real units that are not declared in UnitDefinitions
– 13 of them have ModelStructure Outputs that do not include every "output" variable
– 4 of them have InitialUnknowns that are not sorted in ascending order

A model-based testing of the dynamic behavior of FMUs

Some examples

Tool is available
https://msdl.uantwerpen.be/git/claudio/FMIMOBSTER

To be submitted to
American Modelica Conference 2020
(Cláudio Gomes, Romain Franceschini, Nick Battle, Casper Thule, Kenneth Lausdahl, Hans Vangheluwe, Peter Gorm Larsen. “Application of Model Based Testing to Dynamic Conformance Evaluation of Functional Mockup Units”)
The paper also considers feed-through (#623 (comment))

The publications have yet to be presented.

Goal

The goal of the research is to support the FMI community in resolving possible ambiguities and improving the conformance to the standard.

Why

It is of interest that FMUs adheres to the standard entirely and correctly to improve the adoption and usability of the standard.
“The most acknowledged challenge is related to practical aspects. These include: faulty/incomplete implementations of the FMI standard, ambiguities/omissions in the specification and documentation of the FMUs, …”
(Schweiger, Gerald, Cláudio Gomes, Georg Engel, Irene Hafner, Josef-Peter Schoeggl, Alfred Posch, and Thierry Nouidui. 2019. “An Empirical Survey on Co-Simulation: Promising Standards, Challenges and Research Needs.” Simulation Modelling Practice and Theory 95: 148–63. https://doi.org/10.1016/j.simpat.2019.05.001.)

To support this process it is ideal to provide tools that tests the FMUs compatibility with the FMI standard.

This issue has been discussed and approved by @clagms

[chrbertsch]

@CThuleHansen : Do you address with this issue shortcoming of the FMI standard or of tools exporting FMUs according to the standard? My understanding when realding it is the second.

[CThuleHansen]

@chrbertsch We have discussed this quite a bit internally, and it might be both. Some due to imprecisions/ambiguities of the standard and some due to tools not fully implementing the standard

[chrbertsch]

@CThuleHansen will present this at the coffee breek of the FMI Design meeting today (at 15;30 in room "Micro".

[nickbattle]

I think it is both, as @CThuleHansen says. In order to produce the formal specification, we have to shake the ambiguities out of the standard (and we may have done so incorrectly, but the result is unambiguous). As a result, the formal spec will then identify problems in FMUs that tools have produced (which may or may not be valid, depending on how accurately we formalized the standard). But overall, the result is unambiguous; the challenge is to make the formal specification correct, given that we only have an informal basis and existing tools to go by.

[chrbertsch]

Was presented in Lund and is very much appreciated, also for FMI 3.0.