Test FMUs with external dependencies

[lochel]

The following examples depend on shared objects which are not available or cannot be loaded on a standard target machine:

win64 (Windows 10):

• fmus/2.0/cs/win64/YAKINDU_Statechart_Tools/4.0.4/*
• fmus/2.0/cs/win64/Silver/3.3/*

linux64 (Ubuntu 20.04 LTS)

• fmus/2.0/cs/linux64/JModelica.org/1.15/*
• fmus/2.0/me/linux64/JModelica.org/1.15/*

The fmi-cross-check repository should follow strictly the FMI standard and ensure that the provided FMUs can be simulated by any importing tool that supports the FMI standard without the burden of chasing export specific libraries.

The quality of the cross-check depends highly on the quality of the provided examples. I am very much concerned that it technically is not possible to import the provided examples. It would make the test results and therwith the entire cross-check effort useless.

[lochel]

@t-sommer @chrbertsch @andreas-junghanns please comment on this issue. If you agree that it is backed by the specification, then I will go ahead and prepare a pull request for it.

[casella]

FMI Specification 2.0.2, Section 1.1

Simulator independence: It is possible to compile, link and distribute an FMU without knowing the target simulator. Reason: The standard would be much less attractive otherwise, unnecessarily restricting the later use of an FMU at compile time and forcing users to maintain simulator specific variants of an FMU. Implementation: Using a binary FMU. When generating a binary FMU such as a Windows dynamic link library (.dll) or a Linux shared object library (.so), the target operating system and eventually the target processor must be known. However, no run-time libraries, source files or header files of the target simulator are needed to generate the binary FMU. As a result, the binary FMU can be executed by any simulator running on the target platform (provided the necessary licenses are available, if required from the model or from the used run-time libraries).

FMI Specification 2.0.2, Section 2.3

Dynamic link libraries must include all referenced resources that are not available on a standard target machine [for example DLLs on Windows machines must be built with option “MT” to include the run-time environment of VisualStudio in the DLL, and not use the option “MD” where this is not the case]

FMI Cross-Check rules 4.1, Section 9.1.3

All FMUs submitted to the repository must run without license checks and contain all required files (DLLs, data files etc.) to allow running in any importing tool supporting the specified platform without additional requirements.

These rules seem very clear to me and not subject to interpretation.

Making exceptions means that some importing tools may not be able to run some FMUs only because of external dependencies, thus getting a worse score for reasons that are totally independent from their quality of implementation of the FMI standard. This is obviously unfair.

The FMI cross-check is endorsed by the Modelica Association. I would find it very odd that the MA itself does not apply its own rules when it comes to comparing the performance of different tools that support a MA standard.

[casella]

See also #57

[andreas-junghanns]

@lochel : We have checked our FMU from fmus/2.0/cs/win64/Silver/3.3/* and found nothing wrong.
What have you been testing and how?

[lochel]

@andreas-junghanns There are issues with the shared objects when running your examples in wine. However, I can confirm that it works fine natively in Windows. The corresponding pull request is already closed.

Speaking of #122: It was closed without releasing fmi 2.0.3. It is certainly not ideal to point to a not-existing release for the baseline of the cross-check project.

And even with the changes for (a potential future) version 2.0.3, the cross-check rules state:

All FMUs submitted to the repository must run without license checks and contain all required files (DLLs, data files etc.) to allow running in any importing tool supporting the specified platform without additional requirements.

This general rule seems reasonable for the cross-check. I support the relaxed requirements for the fmi specification, but I don't think it is a good idea to collect such examples in the cross-check repository.

[casella]

A further comment from my side. I fully agree that the cross-check rules should be either followed or changed, see also my comment here. Being able to actually run the FMU is not a side issue, it is central to the whole point of the XC tool.

Regarding the yet unreleased 2.0.3 specification, I understand there is consensus in the MAP-LANG group that tools can start using new features or exploit clarifications in yet unreleased versions of the specification, as long as they have already been approved and are no longer a matter of discussion. Which seems reasonable to me. We've done it several times already, e.g. with the updated conversion scripts.

Does this hold also for MAP-FMI?

[AnHeuermann]

Would UniFMI be a solution for this particular problem? See Portable runtime environments for Python-based FMUs:
Adding Docker support to UniFMU from the 2021 Modelica Conference.

With this it could be possible to specify a docker image that holds all dependencies for a specific FMU. It wont solve the underlying problem that the FMU itself doesn't have needed external dependencies, but would at least solve my problems for e.g. #57.
It could be as simple as adding a Dockerfile next to the FMU. Importing tools are free to use the Dockerfile and run it in a clean testing environment or ignore it.

Something like

FROM ubuntu:18.04

would be enough for nearly all of the FMUs in the cross check.