Participation in FMU Cross-Check

[chrbertsch]

It would be highly beneficial if OMSimulator would participate in the FMI Cross Check https://github.com/modelica/fmi-cross-check. For many tools this has significantly improved the maturity of their FMI support.

[lochel]

Thanks for reminding us. Actually, it has been on my list for quite some time now. I will try to prioritize it.

[lochel]

@chrbertsch Is there some kind of script to support executing all the tests? I guess not since all importing tools are very different.

[chrbertsch]

You could use the XC scripts of fmpy as a blueprint:
https://github.com/CATIA-Systems/FMPy/tree/master/fmpy/cross_check

[AnHeuermann]

See trac ticket 6010 for OpenModelica.
I can probably add OMSimulator there as well with a few additional lines.

[AnHeuermann]

I did a quick run with a Python script to see what sort of error messages we get.
I used a Docker Container with Ubuntu 20.04 with only OMSimulator and Python installed.

Basically we have three kind of errors

1. We don't support FMI 1.0.
   This is design decision and will not change.

   error:   [oms_RunFile] oms_extractFMIKind failed
   

2. JModelica has FMUs which are missing dynamic libraires.

   error:   [fmiLogger] module FMICAPI: Could not load the FMU binary: libgfortran.so.3: cannot open shared object file: No such file or directory
   error:   [instantiate] Could not load "resources/0001_fmu.fmu" which is associated with "model.root.fmu"; it may be corrupted 
   

or may not support your platform
error: [initialize] Model "model" is in wrong model state
error: [simulate] Model "model" is in wrong model state
error: [terminate] Model "model" is in wrong model state
error: [terminate] Model "model" is in wrong model state
```
We can of course install those inside our Docker image for this.

3. We get some segmentation faults for SystemModeler and JModelica FMUs or if we reject the FMU:

   Segmentation fault
   

   or

   error:   [fmiLogger] module FMI2XML: Start attribute is required for this causality, variability and initial combination
   Segmentation fault
   

   This needs to be investigated.

4. We reject a lot of FMUs because of variability of some variables.

   error:   [fmiLogger] module FMI2XML: Start attribute is required for this causality, variability and initial combination
   

   or

   error:   [fmiLogger] module FMI2XML: Start attribute is not allowed for variables with initial='calculated'
   

See this CSV file for the full list of FMUs with output and error mesages:
OMSimulator_v2.1.0-dev-177_linux64_Results.csv.zip

I will investigate on problem 3.
@lochel Any idea what's causing problem 4? My guess is that we reject to many FMUs.

[AnHeuermann]

And we need to filter and check the result files we generate.
python3 -m fmpy.cross_check.validate_vendor_repo is not checking it but complaining about the size.

[lochel]

@lochel Any idea what's causing problem 4? My guess is that we reject to many FMUs.

This is a typical problem of FMUs not following the FMI standard.

[AnHeuermann]

And integer or boolean inputs from a CSV file are apparently a problem if you use a Lua script to define the model.
See e.g. fmi-cross-check/fmus/2.0/cs/linux64/Test-FMUs/0.0.2/Feedthrough/.

error:   [addConnection] Type mismatch in connection: input.int_in -> fmu.int_in
error:   [addConnection] Type mismatch in connection: input.bool_in -> fmu.bool_in

Feedthrough_in.csv:

time,real_tunable_param,real_continuous_in,real_discrete_in,int_in,bool_in
0.0,0,0.0,0,0,0
0.5,0,0.0,0,0,0
0.5,0,2.0,0,0,0
1.0,0,1.0,1,1,1
1.5,-1,1.0,1,1,1
2.0,-1,1.0,1,1,1

[lochel]

This is correct, OMSimulator will treat all csv inputs as real.

[AnHeuermann]

TODO List

I will track the progress on this ticket in this comment.

Different OS to test

• linux64

       # ############################################# #
       #             Final results summary             #
       #       Total FMUs tested:              55      #
       #       FMUs simulated succesfully:     48 / 55 #
       #       FMUs verified:                  15 / 55 #
       # ############################################# #
  

  OMSimulator_v2.1.0-dev-188_linux64_results.csv.zip

• linux32

• windows64

       # ############################################# #
       #             Final results summary             #
       #       Total FMUs tested:              229     #
       #       FMUs simulated succesfully:     197 / 229       #
       #       FMUs verified:                  31 / 229        #
       # ############################################# #
  

• darwin64

Things to fix for the fmi-cross check:

• Add parameters to result file: Add parameters to result file #775
• Support non-real inputs from CSV files: Support non-real connection between CSV input and FMU #776
• Fix wrong results
  • Inaccurate events Inaccurate event detection for CS #781
  • oms.compareSimulationResults currently too strict, change compare tool or relaxe settings
• Fix path issue on Windows OMSimulator can't handle long relative path on Windows #789

[AnHeuermann]

We are now generating results for the FMI Cross Check: https://libraries.openmodelica.org/fmi-crosschecking/OMSimulator/
We can participate with the next release version of OMSimulator.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[AnHeuermann]

I created a PR for OMSimulator v2.1.1 at modelica/fmi-cross-check#106.

Let's see if the FMI cross check can repair it's CI so the PR will be tested eventually.

[lochel]

@AnHeuermann, @chrbertsch what more do we have to do? OMSimulator is still not listed in the fmi cross-check results.

[chrbertsch]

@t-sommer: Do you have an idea, why OMsimulator does not get listed after the successful merge of modelica/fmi-cross-check#106? Do you have to trigger a re-generation of the tables?

[lochel]

@chrbertsch @t-sommer any updates?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please feel free to create a new issue with up-to-date information.