Binaries in the Modelica Standard Library

[modelica-trac-importer]

Reported by fcasella on 3 Jun 2013 09:13 UTC
The Modelica Standard Library depends on external C and FORTRAN code for its operation.

Some functionality has a well-defined Modelica interface, but its implementation in terms of source code needs to be tool-specific. It is agreed that this functionality is confined to the ModelicaServices library, that defines the interface towards the MSL and other libraries. Each tool vendor is then responsible for providing an implementation which works with its tool.

Other functionality is provided by external C code contained in Modelica/C-sources/. In this case, the source code is not tool-specific, bu the very same C source code should be compiled in different tools and also on different OS platforms.

The current situation is that released versions of the MSL only contain the C sources and tool vendors are responsible for compiling and linking these sources with each new tool release. During development of the MSL, the development version of the library on the SVN trunk contains the Dymola-Windows compiled
version of new libraries (ModelicaStandardTables at the moment), which are then meant to be removed from the release version of MSL, which will instead refer to the libraries built-in by the tool vendors, and only contain the source code.

This situation is not optimal for several reasons:

1. It is not possible for any user or developer to checkout any
   specific revision of the MSL and get it to work out of the box
   on their Modelica tool, because it lacks the binaries. This is
   particularly weird for a library that claims to be "standard".
2. Developers of the MSL depend on tool vendors to provide them
   with updated binaries (in-between official releases),
   whenever the C-sources are changed on the trunk. This can
   happen many times during development and testing.
3. When a development version is ready for release, it needs to be
   modified in order to strip it from the binaries and change some
   include files, and this could potentially cause problems.
4. During development and debugging of new external C code, the
   MSL trunk contains binaries only for one specific tool and
   one specific OS, which is unfair towards users of other tools
   and other OSs.

We need to find a long-term sustainable solution that solves this problem also for future similar cases, while allowing to continue the development and testing of MSL 3.2.1 without causing inconveniences.

This is my proposal. It is formulated for the addition of the new tables implementation, but it remains valid, mutatis mutandis, for any future addition to the MSL that requires adding external C code.

1. The code for the new tables is compiled in a new library
   ModelicaStandardTables, making sure that its exported functions
   have different names than the ones of the old tables
   implementation, so the very same tool can support MSL < 3.2.1
   and MSL >= 3.2.1 at the same time without ambiguities and
   symbol conflicts at link time.
2. The ModelicaStandardTables binary will be placed in
   Modelica/Resources/Library (not under
   ToolXY/bin/lib!), which will be referenced by
   LibraryDirectory annotations in the Modelica functions.
3. Each tool vendor provides library compilation scripts for
   each OS they support, that can be executed by a single command
   typed at the OS prompt without any special software installed
   except the Modelica tool itself (e.g., .bat files for Windows,
   makefiles for Linux). These scripts will be stored under
   Modelica/Resources/BuildLibraries with appropriate names,
   e.g. BuildLibrary_Toolname_xxbit), together with a howto.txt
   file with instructions.
4. The compilation scripts must also handle dependencies, based on
   how required libraries such as zlib may or may not be already
   present in the tool installation and on which C compiler the
   tool uses.
5. When shipping a tool for a specific OS (including nightly
   builds), each tool vendor runs the appropriate makefile, then
   removes all the unneeded stuff from Modelica/Resources to avoid
   wasting bandwidth and disk space.
6. MSL developers can download any version of the tool from
   modelica.org or checkout any revision from SVN, run the
   appropriate script with and get an up-to-date, fully functional
   version of MSL work out of the box. The script only needs to be
   run at checkout and whenever the sources are updated.

To my understanding, this proposal solves all the problems listed above. Furthermore, it allows to test models with the very latest version of the C sources, without waiting for tool vendors to provide the updated binaries, and without tweaking the tool installation, which might not be allowed by the OS settings.

Yes, there will be tool-specific code on the MSL trunk (the scripts to build the binaries from sources), but this is not meant to provide unfair advantage for a tool over another one, but rather to ease the development effort of the MA community.

My proposal is that this development model is also adopted for any future addition to the MSL besides the new tables, as well as for the new Table maintenance. We cannot get rid of ModelicaInternalC for backwards compatibility, but we can at least stop adding new code to it.

Tool developers are recommended not to patch the C sources locally on their tools, but rather to contribute to the development of more general and robust versions of the code on the SVN trunk, in the same spirit the rest of the MSL is developed. It might also be useful for other libraries that depend on external C code, such as ExternalMedia.

My recommendation is that this process is tried out immediately on MSL 3.2.1, since it fits the main requirement of this release of being fully standard and tool-neutral.

---

Migrated-From: https://trac.modelica.org/Modelica/ticket/1155

[modelica-trac-importer]

Modified by sjoelund.se on 3 Jun 2013 09:33 UTC

[modelica-trac-importer]

Modified by fcasella on 4 Jun 2013 08:03 UTC

[modelica-trac-importer]

Modified by fcasella on 4 Jun 2013 08:04 UTC

[modelica-trac-importer]

Comment by fcasella on 4 Jun 2013 08:09 UTC
I see that this issue is mentioned in the minutes of the 79th Design Meeting as related to Modelica Specification 3.2 rev2.

As far as I understand, my proposal doesn't need any change or clarification to the language specification, and only uses annotations which were standardized long ago in section 12.9.4.

What needs to be decided is an agreed upon policy for external C sources in the Modelica Standard Library.

[modelica-trac-importer]

Comment by sjoelund.se on 4 Jun 2013 08:57 UTC
It is (3) which would be nice to specify. It can be of course specific to MSL for 3.2r2. But I think it is of general interest for user libraries and should be standardised in the future.

[modelica-trac-importer]

Comment by otter on 5 Jun 2013 13:49 UTC
Discussion at the meeting (agreement to try this out whether this is sufficient):

Solution 1:
Provide build scripts for different environments (Windows, Visual C++ DLL / lib, Minwg; Linux …). Usually there is just a project definition for several versions (32/64 bit, DLL/lib). Store the project-build information here:

Modelica/Resources/BuildProjects/VisualStudio   // for Dymola, SimulationX
                                /AutoTools      // for OpenModelica

This helps tool vendors (and interested library developers) to build the binaries.

Solution 2:
Have two libraries:

ModelicaExternalC
ModelicaStandardTables

In the Table blocks only use library annotation ModelicaStandardTables.

Solution 3:
For the development and for the tools needed, the Resources/libraries directory is provided.
For a tagged/released version, this directory is deleted.

[modelica-trac-importer]

Comment by mtiller on 5 Jun 2013 14:05 UTC
Just a comment. Solution 1 above implies that the autotools information will only be used by OpenModelica. But there is a Linux version of Dymola as well (and probably other tools like CyModelica, JModelica). So many tools (and, ultimately, Linux users like me) would benefit from a autotools support.

[modelica-trac-importer]

Comment by sjoelund.se on 5 Jun 2013 14:08 UTC
That's true and was also commented at the meeting. It is of course not an exhaustive list, but as a start, VisualStudio+autotools would be enough for almost all platforms.

[modelica-trac-importer]

Comment by thiele on 5 Jun 2013 15:08 UTC
The Modelica_DeviceDrivers library uses plenty of external C-code and supports win32/64 and linux32/64.

I found that for me CMake (http://www.cmake.org/) works pretty well to maintain the sources for all the platforms. I therefore think that it might be worth to consider its use also for the MSL.

The external code setup used in Modelica_DeviceDrivers can be found at https://github.com/modelica/Modelica_DeviceDrivers/tree/master/Resources.

In order to make it convenient to users I keep binaries for all the platforms specified in
"12.9.4 Annotations for External Libraries and Include Files" in the repository (i.e. Resources/Libraries/[win32|win64|linux32|linux64]). If the binaries don't work (e.g., due to compiler version incompatibilities), a user has to resort to CMake. CMake can then generate VS projects or linux makefiles as needed. It's quite straight forward, but I suppose it could be even more automatized by using intelligent scripts if "one click" solutions are deemed to be necessary.

[modelica-trac-importer]

Comment by sjoelund.se on 7 Jun 2013 05:41 UTC
Well. cmake requires you to actually install cmake in order to use it (configure the project). With autotools the files will generate the configure script (as well as sane makefiles), which you can then check into subversion.

Since this is supposed to be possible to run as the user (not tool developer), it is good if it can configure everything without needing (a specific version of) cmake installed.

[modelica-trac-importer]

Comment by thiele on 7 Jun 2013 08:06 UTC
Replying to [comment:10 sjoelund.se]:
Yes, cmake would need to be installed, though it might be possible to provide "one click"-scripts that initiate an automatic downloading/installation if cmake is not found. However, not sure whether this would really help.

Regarding the need for users to install cmake to do the build, I also think that autotools are not installed by default in some (many?) linux distributions (at least I think it is not the case for Ubuntu, but I might be wrong), not to speak from other third-party libraries (or linux xxx-dev packages) that may be needed to do the build.

Nice thing of cmake for my taste is the possibility to choose between various project build generators, but only need to maintain one set of build files. Also the use of cmake for supporting cross-compiling tool chains is not bad (nice if non-standard platforms, e.g., real-time simulation platforms, should be targeted).

I mean, it's a matter of taste how external C-code should be maintained and at the end the people who take responsibility to do that for the MSL need to find a mode that they think works best for them (and at the moment I'm of course not part of that group, but I think you are).

[modelica-trac-importer]

Comment by sjoelund.se on 7 Jun 2013 08:27 UTC
From experience, all cmake projects I have ever tried to use failed. This is because they used features that required the very latest version of cmake installed. The main problem here is that Linux distributions will not ship this version of cmake, but an older one. So you had to go online to find cmake and install it from source. If cmake was more mature and didn't add new features all the time (or if the people who used it restricted themselves to use what was available 3 years ago), there would not be any problems.

autotools on the other hand do not need to be installed on the client machine. You run the generation script and get a configure script + a Makefile. It is even portable to targets that autotools themselves do not run on.

As for cross-compilation using autotools... Let's say I want to compile for 32-bit Windows MinGW using my 64-bit Ubuntu as the host:

[martin@mega ModelicaExternalC]$ ./configure  --host=i586-mingw32msvc 
checking whether build environment is sane... yes
checking for i586-mingw32msvc-strip... i586-mingw32msvc-strip
checking for i586-mingw32msvc-gcc... i586-mingw32msvc-gcc
...
[martin@mega ModelicaExternalC]$ make
/bin/bash ./libtool  --tag=CC   --mode=compile i586-mingw32msvc-gcc -DPACKAGE_NAME=\"Modelica\ Standard\ Library\ Tables\" -DPACKAGE_TARNAME=\"libmodelicastandardtables\" -DPACKAGE_VERSION=\"3.2.1\" -DPACKAGE_STRING=\"Modelica\ Standard\ Library\ Tables\ 3.2.1\" -DPACKAGE_BUGREPORT=\"https://trac.modelica.org/Modelica\" -DPACKAGE_URL=\"https://modelica.org\" -DPACKAGE=\"libmodelicastandardtables\" -DVERSION=\"3.2.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DLT_OBJDIR=\".libs/\" -I.   -DDUMMY_FUNCTION_USERTAB  -g -O2 -O0 -g -MT ModelicaInternal.lo -MD -MP -MF .deps/ModelicaInternal.Tpo -c -o ModelicaInternal.lo `test -f '../../../libraries/Modelica\ 3.2.1/Modelica/Resources/C-Sources//ModelicaInternal.c' || echo './'`../../../libraries/Modelica\ 3.2.1/Modelica/Resources/C-Sources//ModelicaInternal.c
...

[modelica-trac-importer]

Comment by sjoelund.se on 7 Jun 2013 08:50 UTC
Replying to [comment:12 sjoelund.se]:

autotools on the other hand do not need to be installed on the client machine.

I will clarify this a little. autotools can generate everything that is needed for compilation, so you can ship this.

This does, however, grow the project size from ~3kB to ~2MB. As such, one would usually keep the smaller project in subversion and only ship the generated, more portable, project in the release zip.

[modelica-trac-importer]

Comment by beutlich on 7 Jun 2013 12:43 UTC
Replying to [comment:8 sjoelund.se]:

That's true and was also commented at the meeting. It is of course not an exhaustive list, but as a start, VisualStudio+autotools would be enough for almost all platforms.

It is finally done.

[modelica-trac-importer]

Comment by thiele on 7 Jun 2013 17:47 UTC
Replying to [comment:12 sjoelund.se]:
Alright, I'm fine with anything that works. Just as a little rehabilitation for CMake I should note that my experiences with CMake-projects from "other people" have certainly not been as devastating as yours and I wouldn't like to scare people away to give it a try if they search a tool for cross-platform development.

[modelica-trac-importer]

Comment by sjoelund.se on 7 Jun 2013 17:52 UTC
I am not saying no to using cmake either. No harm having more choices for library developers.
Specifying the name of the directory and the (default) order of commands needed to install the binary into Resources/Library should not be that much for cmake.

[modelica-trac-importer]

Comment by msielemann on 10 Jun 2013 09:19 UTC
Dear all,
I am not a specialist in this area, but I have something to add. I have been using the Trilinos library for some time, and building this libary on Windows was a pain using Autotools. Since version 10, they switched to CMake and things became much better.

In order to avoid establishing the foundations now based on the wrong solution, I recommend the following reading: http://www.ornl.gov/~8vt/TrilinosCMakeEvaluation.pdf

Possibly, all the good reasons to use CMake for Trilinos do not apply to our problems. But for my usage of Trilinos I would hate to step back to Autotools....

[modelica-trac-importer]

Comment by dietmarw on 10 Jun 2013 09:26 UTC
Michael, as far as I understand the system now, there is no reason to have "autotools" or "cmake" configurations. We can provide both configuration setups and the developers/tool vendors can then decide which one they use for compilation. This will of course also depend on who is willing to maintain the different variations. So just because OM contributed the autotools configuration (and will most likely also keep it up to date) does not mean you (or anybody else) can not contribute the "cmake" configuration under BuildProjects/cmake as well. :-)

[modelica-trac-importer]

Comment by otter on 18 Jul 2013 12:20 UTC
Summary of discussion:

• If MSL relies on C-source, the C-code is provided under Resources, if practical (e.g. not practical for Lapack since much too big).
• Additionally to the-code, build scripts for various compilers and operating systems are provided in order to help a tool vendor to generate a binary library from the C-code.
• Due to the many different possibilities and the specialities of tools, in a release version of MSL no binary library is provided. It is the tool vendors responsibility to provide binaries.

[modelica-trac-importer]

Comment by fcasella on 18 Jul 2013 13:16 UTC
Replying to [comment:19 otter]:

• Additionally to the-code, build scripts for various compilers and operating systems are provided in order to help a tool vendor to generate a binary library from the C-code.

In fact, I understand the idea is that tool vendors provide these scripts to help MSL developers and advanced users to build the binaries on their own from the latest (or possibly modified) version of the source code.