ModelicaUtilities for shared object / dynamic link library

[modelica-trac-importer]

Modified by beutlich on 29 Nov 2017 10:35 UTC
It is currently not (easily) possible that an external shared object / dynamic link library depends on ModelicaUtilities (e.g., for ModelicaFormatMessage or ModelicaFormatError), since ModelicaUtilities is tool-dependent, but shared object / dynamic link libraries are executable (opposed to static or import libraries).

This leads to trouble for library developers of tool-generic Modelica libraries, especially for the two tools Dymola and SimulationX on Windows, see e.g.,

• Missing ModelicaExternalC.dll in Modelica modelica-3rdparty/ExternData#2 or
• https://github.com/ClaRaLibrary/TILMedia/blob/master/Internals/redirectModelicaFormatMessage.mo
• https://github.com/CATIA-Systems/Modelica-Arduino/blob/master/Arduino/Resources/Include/ModelicaFirmata.c
  with the following workarounds:

1. Always have static libraries for Dymola and dynamic link libraries for SimulationX (by utilizing SimulationX specific ModelicaUtilities)
2. Use external code to access ModelicaFormatMessage etc. and redirect the ModelicaUtilities functions.

Both workarounds have their drawbacks and are error-prone. One possible solution would be an extension of the Modelica external function interface by providing optional callback functions for libraries. At least, this cleans up the current mess, however I am not sure if new problems are likely.

---

Modified by beutlich on 19 Oct 2017 08:33 UTC
It is currently not (easily) possible that an external shared object / dynamic link library depends on ModelicaUtilities (e.g., for ModelicaFormatMessage or ModelicaFormatError), since ModelicaUtilities is tool-dependent, but shared object / dynamic link libraries are (opposed to) static or import libraries executable.

This leads to trouble for library developers of tool-generic Modelica libraries, especially for the two tools Dymola and SimulationX on Windows, see e.g.,

• Missing ModelicaExternalC.dll in Modelica modelica-3rdparty/ExternData#2 or
• https://github.com/ClaRaLibrary/TILMedia/blob/master/Internals/redirectModelicaFormatMessage.mo
  with the following workarounds:

1. Always have static libraries for Dymola and dynamic link libraries for SimulationX (by utilizing SimulationX specific ModelicaUtilities)
2. Use external code to access ModelicaFormatMessage etc. and redirect the ModelicaUtilities functions.

Both workarounds have their drawbacks and are error-prone. One possible solution would be an extension of the Modelica external function interface by providing optional callback functions for libraries. At least, this cleans up the current mess, however I am not sure if new problems are likely.

---

Modified by beutlich on 21 Jun 2017 08:01 UTC
It is currently not (easily) possible that an external shared object / dynamic link library depends on ModelicaUtitlities (e.g., for ModelicaFormatMessage or ModelicaFormatError), since ModelicaUtilities is tool-dependent, but shared object / dynamic link libraries are (opposed to) static or import libraries executable.

This leads to trouble for library developers of tool-generic Modelica libraries, especially for the two tools Dymola and SimulationX on Windows, see e.g.,

• Missing ModelicaExternalC.dll in Modelica modelica-3rdparty/ExternData#2 or
• https://github.com/ClaRaLibrary/TILMedia/blob/master/Internals/redirectModelicaFormatMessage.mo
  with the following workarounds:

1. Always have static libraries for Dymola and dynamic link libraries for SimulationX (by utilizing SimulationX specific ModelicaUtilities)
2. Use external code to access ModelicaFormatMessage etc. and redirect the ModelicaUtilities functions.

Both workarounds have their drawbacks and are error-prone. One possible solution would be an extension of the Modelica external function interface by providing optional callback functions for libraries. At least, this cleans up the current mess, however I am not sure if new problems are likely.

---

Reported by beutlich on 20 Jun 2017 13:11 UTC
It is currently not (easily) possible that an external shared object / dynamic link library depends on ModelicaUtitlities (e.g., for ModelicaFormatMessage or ModelicaFormatError), since ModelicaUtilities is tool-dependent, but shared object / dynamic link libraries are (opposed to) static or import libraries executable.

This leads to trouble for library developers of tool-generic Modelica libraries, especially for the two tools Dymola and SimulationX on Windows, see e.g.,

• Missing ModelicaExternalC.dll in Modelica modelica-3rdparty/ExternData#2 or
• https://github.com/ClaRaLibrary/TILMedia/blob/master/Internals/redirectModelicaFormatMessage.mo
  with the following workarounds:

1. Always have static libraries for Dymola and dynamic link libraries for SimulationX (by utilizing SimulationX specific ModelicaUtilities)
2. Use external code to access ModelicaFormatMessage etc. and redirect the ModelicaUtility functions.

Both workarounds have their drawbacks and are error-prone. One possible solution would be an extension of the Modelica external function interface by providing optional callback functions for libraries. At least, this cleans up the current mess, however I am not sure if new problems are likely.

---

Migrated-From: https://trac.modelica.org/Modelica/ticket/2191

[modelica-trac-importer]

Comment by beutlich on 20 Jun 2017 17:36 UTC
Just after writing the ticket a cleaner and more appealing solution came to my mind: If a tool supports loading of external shared objects / dynamic link libraries, it needs to provide a ModelicaExternalC.so/ModelicaExternalC.dll such that library developers can resolve the external ModelicaUtilities symbols (either at build-time or at run-time by dlopen / LoadLibrary(Ex)) in a tool-generic way.

Thus the ticket reduces to a tool issue, actually, however, the specification could be improved for clarification.

[modelica-trac-importer]

Comment by sjoelund.se on 20 Jun 2017 17:47 UTC
You don't need dlopen to do the lookup at runtime. Just link allowing unresolved symbols (if necessary, specify which symbols). This way, a tool can generate a shared object (not an executable) like SimulationX and have the object automatically find the missing symbols when loaded by the simulation engine (which would either be part of the engine or loaded in a different shared object using the OS options to expose the ModelicaUtilities symbols to future loaded shared objects). So yes, a tool issue to handle it.

The case when generation an executable is simpler: just link in the tool-specific symbols so they are found by the library (for GCC, add the library after all user libraries on the command line).

[modelica-trac-importer]

Comment by beutlich on 20 Jun 2017 18:09 UTC
According to https://stackoverflow.com/q/2326460 it is possible to build MSVC DLLs with unresolved symbols using /FORCE:UNRESOLVED, however it will crash as soon as one of the unresolved symbols is called. Thus, your proposal does not seem to apply to the MSVC world.

[modelica-trac-importer]

Comment by sjoelund.se on 20 Jun 2017 20:16 UTC
Seems the MSVC world does handle it, but it does it slightly differently (avoiding the source code using dllimport in the DLL and dllexport in the exe): https://stackoverflow.com/questions/9950607/unresolved-external-symbol-when-creating-a-dll

Which would be a problem since I believe the header in the specification doesn't have any dllimport in it...

[modelica-trac-importer]

Comment by beutlich on 20 Jun 2017 20:42 UTC
Good answer. Thanks. Thus, if dymosim.exe and simx.exe export the ModelicaUtilities functions, they could load the tool-generic external DLL. However, we would need a dymosim.lib and simx.lib to link the external DLL with, which makes the library linking tool-specific again. Therefore, having a ModelicaExternalC.lib that resolves the ModelicaUtilities symbols in a ModelicaExternalC.dll would be the tool-generic solution (as proposed by comment:1).

[modelica-trac-importer]

Modified by beutlich on 21 Jun 2017 08:01 UTC

[modelica-trac-importer]

Modified by beutlich on 6 Oct 2017 06:12 UTC

[modelica-trac-importer]

Comment by c.schulze on 6 Oct 2017 07:38 UTC
Please keep in mind:

• dependencies: Every executable or FMU had to provide this ModelicaExternalC.dll
• availability of ModelicaExternalC: The dll should not be available in the path environment, to enable the parallel installation of e.g. OpenModelica and Dymola.
• multiple FMUs use the same library: Multiple FMUs used by the same application refer to the same external library instance. Still error messages of the external library should be output by the FMU in which the error occurred.
• FMU file size: If this dll is linked with /mt the file size is > 0

[modelica-trac-importer]

Comment by hansolsson on 19 Oct 2017 08:08 UTC
Should investigate if:

• Adding dllexport in ModelicaUtilities.h
• Can then use dllimport in external libraries using ModelicaWarning: include instructions for linking
• That this works
• And can handle the error of missing function
• Should we specify name of module?

Hans to test in Dymola for next meeting.

[modelica-trac-importer]

Modified by beutlich on 19 Oct 2017 08:33 UTC

[modelica-trac-importer]

Comment by beutlich on 20 Nov 2017 20:17 UTC
One more candidate:

• Requirements for new release of ExternalMedia and action plan modelica-3rdparty/ExternalMedia#4 (comment)

[modelica-trac-importer]

Comment by beutlich on 20 Nov 2017 20:26 UTC
Replying to [comment:8 c.schulze@…]:

Please keep in mind:

• dependencies: Every executable or FMU had to provide this ModelicaExternalC.dll

No, ModelicaExternalC.dll is tool-specific and would be part of the installation of the Modelica simulation environment.

• availability of ModelicaExternalC: The dll should not be available in the path environment, to enable the parallel installation of e.g. OpenModelica and Dymola.

Right, OpenModelica would use its appropriate ModelicaExternalC.dll, Dymola another one from its installation directory.

• multiple FMUs use the same library: Multiple FMUs used by the same application refer to the same external library instance. Still error messages of the external library should be output by the FMU in which the error occurred.

Currently, all (error) messages go to a single log window only. Different instances cannot be differentiated other by its instance name if part of the (error) message.

• FMU file size: If this dll is linked with /mt the file size is > 0

For static libraries (LIB) /MT for MSVC compilation is pretty established since the Dymola build scripts build with this flag. But for dynamic link libraries (DLL) you are free to use /MD as well.

[modelica-trac-importer]

Comment by c.schulze on 20 Nov 2017 21:20 UTC
Replying to [comment:12 Thomas Beutlich]:

Replying to [comment:8 c.schulze@…]:

Please keep in mind:

• dependencies: Every executable or FMU had to provide this ModelicaExternalC.dll

No, ModelicaExternalC.dll is tool-specific and would be part of the installation of the Modelica simulation environment.

FMUs can be executed without the Modelica simulation environment. The external library using the error function and the ModelicaExternalC.dll have to be part of the FMU to make it portable.

• availability of ModelicaExternalC: The dll should not be available in the path environment, to enable the parallel installation of e.g. OpenModelica and Dymola.

Right, OpenModelica would use its appropriate ModelicaExternalC.dll, Dymola another one from its installation directory.

Yet still it needs to be available. So copying it to the executable or a tool specific search algorithm with delayed loading of the dll would be possible solutions. It would be nice if exported executables could be executed on computers without an installed Modelica simulation environment.

• multiple FMUs use the same library: Multiple FMUs used by the same application refer to the same external library instance. Still error messages of the external library should be output by the FMU in which the error occurred.

Currently, all (error) messages go to a single log window only. Different instances cannot be differentiated other by its instance name if part of the (error) message.

The only solution I can think of is an implementation similar to the fmiComponent approach. This structure can be constructed with external objects. I have to think about that. No idea how to realize that with Modelica.Media.

• FMU file size: If this dll is linked with /mt the file size is > 0
  For static libraries (LIB) /MT for MSVC compilation is pretty established since the Dymola build scripts build with this flag. But for dynamic link libraries (DLL) you are free to use /MD as well.

Agree. MD might be more appropriate.

[modelica-trac-importer]

Comment by beutlich on 20 Nov 2017 21:34 UTC
Replying to [comment:13 c.schulze@…]:

Replying to [comment:12 Thomas Beutlich]:

Replying to [comment:8 c.schulze@…]:

Please keep in mind:

• dependencies: Every executable or FMU had to provide this ModelicaExternalC.dll

No, ModelicaExternalC.dll is tool-specific and would be part of the installation of the Modelica simulation environment.
FMUs can be executed without the Modelica simulation environment. The external library using the error function and the ModelicaExternalC.dll have to be part of the FMU to make it portable.

I believe we cannot force FMI import tools to provide a ModelicaExternalC.dll. Well, then it only will work out if we apply the same mechanisms from FMI also for Modelica external C interface, that is, have logging/error functions set by callbacks.

[modelica-trac-importer]

Comment by sjoelund.se on 20 Nov 2017 23:59 UTC
Replying to [comment:14 Thomas Beutlich]:

I believe we cannot force FMI import tools to provide a ModelicaExternalC.dll. Well, then it only will work out if we apply the same mechanisms from FMI also for Modelica external C interface, that is, have logging/error functions set by callbacks.

Furthermore, some FMI importing tools (FMIL comes to mind) import FMUs with flags set to ignore functions found in its own DLL, preferring already loaded versions of for example ModelicaExternalC instead (meaning if you load a Dymola FMU and then an OpenModelica FMU you could possibly have the OpenModelica FMU using Dymola's version of ModelicaError).

[modelica-trac-importer]

Comment by beutlich on 21 Nov 2017 06:57 UTC
Replying to [comment:14 Thomas Beutlich]:

Replying to [comment:13 c.schulze@…]:

Replying to [comment:12 Thomas Beutlich]:

Replying to [comment:8 c.schulze@…]:

Please keep in mind:

• dependencies: Every executable or FMU had to provide this ModelicaExternalC.dll

No, ModelicaExternalC.dll is tool-specific and would be part of the installation of the Modelica simulation environment.
FMUs can be executed without the Modelica simulation environment. The external library using the error function and the ModelicaExternalC.dll have to be part of the FMU to make it portable.

I believe we cannot force FMI import tools to provide a ModelicaExternalC.dll. Well, then it only will work out if we apply the same mechanisms from FMI also for Modelica external C interface, that is, have logging/error functions set by callbacks.

Or in other - more radical - words: Because of the incompatibilities (w.r.t. to logging and error reporting, i.e., ModelicaUtilities) of shared objects / dynamic link libraries (DLL) used with the Modelica external C interface, we declare this interface as deprecated and recommend library developers to use the FMI instead. Of course, the FMI ist the heavier interface and we need to address issues like msl:#2389 and #1626 first (and even should guide library devs how to build an FMU), but FMI is more mature, widely supported and progressing in development.

[modelica-trac-importer]

Comment by fcasella on 21 Nov 2017 13:46 UTC
There is one lightweight and portable workaround for this issue: the external function calls a wrapper function which is directly defined in the include annotation (so it gets statically linked with the simulation code by construction) and calls ModelicaError if that's the case.

The actual external function is called by the wrapper function and returns an error string, which is empty if there is no error, or contains the error message otherwise.

This is a small example demonstrating the approach

package TestExternal
  function f_sin
    input Real x;
    output Real y;
  protected
    String error;
    external "C" f_sin_int(x, y);
    annotation(Include="
  #include <math.h>
  #include \"TestExternal.h\"
   
  void f_sin_int(double x, double *y)
  {
    char *error;
    f_sin_lib(x, y, &error);
    if(*error)
      ModelicaError(error);
  }   
  ",Library="TestExternal");
  end f_sin;

  model Test
    Real x(start = 0, fixed = true);
  equation
    der(x) = 1 + f_sin(x);
  annotation(experiment(StopTime = 1));
  end Test;
end TestExternal;

/* TestExternal.c */
#include <math.h>
void f_sin_lib(double x, double *y, char **error)
{
  *y = sin(x);
  if (x > 1)
    *error = "Error: f_sin(x) called with x > 1";
  else
    *error = "";
}          

/* TestExternal.h */
void f_sin_lib(double x, double *y, char **error);

I guess this approach should be fine in most cases, without the need using FMI, which has a much broader scope and overhead.

[modelica-trac-importer]

Comment by beutlich on 21 Nov 2017 13:55 UTC
SimulationX would look for function f_sin_int in TestExternal.dll here and fail since it is not exported.

[modelica-trac-importer]

Modified by beutlich on 29 Nov 2017 10:35 UTC

[modelica-trac-importer]

Comment by beutlich on 29 Nov 2017 10:40 UTC
I added the https://github.com/CATIA-Systems/Modelica-Arduino library as another example of utilizig external code for passing the ModelicaUtilities function pointers to the DLL.

[modelica-trac-importer]

Modified by beutlich on 23 Mar 2018 08:05 UTC

[casella]

Based on recent experience with the ExternalMedia library and on expert input from @fedetftpolimi, I'd like to reopen the discussion and hopefully lead it to port.

Analysis

External C functions can be called by a Modelica simulation runtime, but also by the Modelica tool itself at compile time, e.g. if the dimensions of some arrays are determined by the output of external functions that read from some data tables, so they need to be called at compile time before performing structural analysis. Proper management of anomalous situations requires function from ModelicaExternal to be available to those external functions, in particular ModelicaError, which is needed to handle exceptions properly, e.g. by re-trying with a shorter time step at runtime, or issuing an error message at compile time.

Existing tools are unable to call into static libraries as part of evaluating package constants, so any solution involving static libraries solves the problem for the run time part, but not for the compile time part. This includes the previously proposed approach of having external functions call a wrapper function which is directly defined in the include annotation.

It appears that the only way to make an external C function callable both by the Modelica simulation runtime and by the Modelica tool itself is to wrap it in a dynamic/shared library, which brings us to the question of how to make the symbols of ModelicaUtilities (such as ModelicaError) available to shared/dynamic libraries in a tool-independent way.

All the discussion in this thread so far focused on the need to put those symbols in an additional dynamic/shared library such as ModelicaExternalC.so/ModelicaExternalC.dll library and make it available, but actually it is possible to do better for these reasons:

1. Both on Windows, Linux and Mac OS, not only dynamic libraries can export symbols, but also executable programs can. Thus the simulation runtime itself and the Modelica tool itself can provide and export the required ModelicaUtilities symbols.
2. On Linux and Mac OS, shared libraries can be built in a way to allow undefined symbols without having to specify in which library they reside. When the library is loaded, the undefined symbols are searched in the executable program that loaded them, as well as in all other dynamic libraries that were loaded before. Thus, it is possible to build a shared library that calls ModelicaError or any other ModelicaExternal function without hardcoding where this symbol should be. It could be in the executable, or in any of the libraries already loaded by the executable. If the symbol could not be found, the library will correctly fail to load. This is important, as building a shared library without specifying where the symbols of ModelicaUtilities have to be leaves maximum flexibility to the tool vendors to provide those symbols however they like, either in the executable or in one of their runtime libraries, and does not require them to agree on the library name that must have those symbols nor on its path on the filesystem.
3. While Windows does not do this symbol resolution automatically, it is possible to write code to do so, as demonstrated by this code of ExternalMedia: https://github.com/modelica-3rdparty/ExternalMedia/blob/master/Projects/Sources/importer.h#L17: first, it searches for a ModelicaUtilities symbol (e.g. ModelicaError) in the executable that loaded the library and, if not found, in the already loaded libraries.

The solution outlined in 2. works well with both OpenModelica and Dymola on Linux, where the task is performed transparently by the dynamic linker. It also works well with OpenModelica on Windows with the solution proposed in 3., since the OpenModelica compiler executable happens by chance to already export the required symbols

Unfortunately, this solution currently doesn't work with Dymola on Windows, as Dymola does not export ModelicaUtilities symbols, for the reason that this is not required by the specification. This required us to get into some ugly workarounds into ExternalMedia to pass pointers to ModelicaError to the external functions, which only work at runtime and are unnecessarily complicated.

Proposal

We think that the MSL should be amended to require that the Modelica compilers and their simulation runtimes both export the symbols of all ModelicaUtilities functions either in the executable or in a runtime library, before trying to load any user-supplied dynamic/shared library.

This would allow for a clean solution that works across all tools and all operating systems.

We can provide a stripped-down MWE that demonstrates the concept, based on what we learned with ExternalMedia.

@HansOlsson, @henrikt-ma, @gkurzbach what do you think?

[henrikt-ma]

I am not the right person to analyze the proposal, but perhaps @otronarp has something to say?

Existing tools are unable to call into static libraries as part of evaluating package constants, so any solution involving static libraries solves the problem for the run time part, but not for the compile time part.

It might not be the most efficient approach, but I thought a simple solution for compile time evaluation would be to compile a small external function evaluator program for evaluating any of the external functions of a library. This program would be built automatically by the tool, and the external functions can be handled similarly to when building a simulator, except that the implementation of ModelicaUtilities.h doesn't necessarily have to be the same. Since the evaluator program is built automatically on demand, I guess it would be safe to use a binary data format for passing the arguments and receiving results, and by letting the evaluator program run as a server during the model translation, one could avoid program startup overhead with each evaluation. As far as I can tell, this approach requires nothing beyond what is specified today.

[fedetftpolimi]

The following comment is written from the perspective of a library writer who is using the Modelica external functions interface for a number of research projects, including ExternalMedia.
Your solution is likely technically feasible (but see below!). However, it is as technically feasible now as it was in 2017 when the bug was opened. Meanwhile, 7 years later, no tool vendor that I know of made the required software infrastructure to call into static libraries at compile time. On the other hand, with dynamic libraries we are "almost there", as on Linux everything works on the two major Modelica tools, and on Windows only Dymola is still not exporting the required symbols.
I guess whatever solution(s) end up being chosen, I think it would nonetheless take some convincing the tool vendors, such as making it explicit in the standard, before library developers are given a way to call into ModelicaError and related functions in a tool and platform independent way.

Moreover, static libraries bring their set of hassles too. For example, ExternalMedia depends on CoolProp that in turn depends on libstdc++. How to tell the Modelica tool that, when linking the static library libExternalMedia.a it needs to link with libstdc++ as well? Of course libstdc++ is just an example, another static library may need to link with an arbitrary set of other libraries and that set is not encoded in the static library itself (unlike with dynamic libraries), it's usually hardcoded by the library developer in their build system.
You could tell me that to be used with Modelica one must provide a "self contained" static library that itself links statically with all its dependencies, but that would not work either, as coming back to the libstdc++ case imagining libExternalMedia.a links statically with libstdc++ (is there even a clean way to do that?), what if the Modelica tool uses libstdc++ as well (like OpenModelica, which does)? What would happen is that at the linking stage multiple versions of the same symbols of the common libraries, possibly from incompatible versions, would be linked together, and that's a recipe for disaster... Again, libstdc++ is just an example, the issue could pop up with libblas, libssl, etc.
If you want an example, see
OpenModelica/OpenModelica#8738
OpenModelica used to have problems with external functions due to the fact their internal build system linked with a few libraries statically, and the issue was fixed by moving to shared libraries.

The cool thing with dynamic libraries isn't just that they can be loaded at runtime into a running program without the need for a software infrastructure involving calling the compiler and launching a separate program and adding some sort of IPC, but also that they contain enough information to load all their dependencies in a way that is transparent to the tool that loads them.

[fedetftpolimi]

mwe.zip

I'm also attaching a minimum working example with a model Test that calls a foo function at translation time and a bar function at runtime, both of which can be made to fail by calling ModelicaError.

I only tested it with OpenModelica on Linux, but it uses the same importer code of ExternalMedia so it should also work on Windows.

Note: the C/C++ code is only provided in source form, you need to build it yourself to try the example.