New ND-table based on SDF to be integrated in MSL

[modelica-trac-importer]

Reported by ahaumer on 14 Jun 2014 11:28 UTC
Modelon Germany developed a new ND-table implementation based on Scientific Data Format (see https://modelica.org/events/modelica2014/proceedings/html/submissions/ECP14096885_AndresDiehlSommer.pdf). The tools (not only the teh ND-table block, but also an editor and integration of the same table lookup n other tools, e.g. Matlab/Simulink) are available at [​https://svn.modelica.org/projects/Modelica/branches/development/SDF/trunk]. This tickes is intended to summarize issues required to integrate the new ND-table implementation into the MSL.

---

Migrated-From: https://trac.modelica.org/Modelica/ticket/1516

[modelica-trac-importer]

Comment by beutlich on 16 Jun 2014 07:13 UTC
My concerns (also given as part of the review process for MoCo 2014):

• The interpolation of N-dimensional table data is already part of issue Ideas for a new table call #1153. Thus there should be some agreement on Ideas for a new table call #1153 first before integrating such a special solution.
• HDF5 is widely discussed in the Modelica community, especially for managing simulation results. Importing HDF5 based tables is already available with MSL 3.2.1 that supports reading (in fact also writing) all MAT-file formats (v4, v6, v7 and v7.3) by means of the ModelicaMatIO module. The v7.3 is the latest MAT-file format version from The MathWorks and is based on HDF5. However, from MSL side there is no need that simulation tools also support v7.3. The reason why HDF5 was left out for MSL tables are due to the
• larger number of files to be compiled, thus long compilation time,
• additional dependencies (like szip),
• large size of binary, escpecially static lib,
• long linking time (if compiled/linked with /LTCG flag on Visual C++ 2010 (default)),
• lot of overhead that is not of aplication use. Actually, in his introduction of http://www.ep.liu.se/ecp/096/113/ecp14096113.pdf Mike Tiller calls HDF5 over engineered.
• I would like to sperate (N-dimensional) interpolation from data file format. For MSL tables we have ASCII text and MAT-file data file formats and univariate/bivariate interpolation. HDF5 is a format extension of table input data but not a general replacement.
• In Interpolation options #477 there also is the idea of allowing different extrapolation/interpolation options by every table dimension.
• SimulationX installs its database files as .sdf, which is SQL Server Compact Edition Database File. Therefore it is confusing to reuse the file extension .sdf for HDF5 based table data. I propose to call it what it actually is (either .h5 or .hdf5) or use a less confusing extension (like .sdf5).
• For Akima splines of MSL 3.2.1 I implemented a special linear extrapolation that does not calculate the slope from the first/last two sample points but rather uses the boundary slope of the spline polynomial. See Modelica ticket Tables: CombiTimeTable with spline interpolation and linear extrapolation #1028 and Discontinuous change in Modelica.Blocks.Tables.CombiTable2D #1465. This kind of extrapolation shall also be considered for N-dimensional tables.

[modelica-trac-importer]

Comment by dietmarw on 17 Jun 2014 10:35 UTC
Reply by Torsten Sommer (Modelon)

Replying to [comment:1 beutlich]:

My concerns (also given as part of the review process for MoCo 2014):

* The interpolation of N-dimensional table data is already part of issue #1153. Thus there should be some agreement on #1153 first before integrating such a special solution.
* HDF5 is widely discussed in the Modelica community, especially for managing simulation results. Importing HDF5 based tables is already available with MSL 3.2.1 that supports reading (in fact also writing) all MAT-file formats (v4, v6, v7 and v7.3) by means of the ModelicaMatIO module. The v7.3 is the latest MAT-file format version from The MathWorks and is based on HDF5. However, from MSL side there is no need that simulation tools also support v7.3. The reason why HDF5 was left out for MSL tables are due to the
* larger number of files to be compiled, thus long compilation time,
* additional dependencies (like szip),
* large size of binary, escpecially static lib,
* long linking time (if compiled/linked with /LTCG flag on Visual C++ 2010 (default)),
* lot of overhead that is not of aplication use. Actually, in his introduction of http://www.ep.liu.se/ecp/096/113/ecp14096113.pdf Mike Tiller calls HDF5 over engineered.
* I would like to sperate (N-dimensional) interpolation from data file format. For MSL tables we have ASCII text and MAT-file data file formats and univariate/bivariate interpolation. HDF5 is a format extension of table input data but not a general replacement.

The interpolation is completely decoupled from the file format. Currently the implementation supports HDF5 and compiled-in data ("UserTab.h"). Additional file formats e.g. a text based format could easily be added. The issue with .mat and .csv files is that there is no proper support for multi-dimensional data and meta info like quantities, units and scales.

* In #477 there also is the idea of allowing different extrapolation/interpolation options by every table dimension.

The implementation could easily be changed to support different inter- / extrapolation methods for every dimension. However the user would have to set them all which can be confusing. Maybe a boolean flag "use for all dimensions" that disables the separate selection would be a solution.

* SimulationX installs its database files as .sdf, which is SQL Server Compact Edition Database File. Therefore it is confusing to reuse the file extension .sdf for HDF5 based table data. I propose to call it what is actually is (either .h5 or .hdf5) or use a less confusing extension (like .sdf5).

The file extension ".sdf" is used to link the files with the SDF Editor and to indicate that certain meta information is available (i.e. quantities, units, ...) which might not be present in a "standard" HDF5 file. The users can store their files with whatever file extension they want.

* For Akima splines of MSL 3.2.1 I implemented a special linear extrapolation that does not calculate the slope from the first/last two sample points but rather uses the boundary slope of the spline polynomial. See Modelica ticket #1028 and #1465. This kind of extrapolation shall also be considered for N-dimensional tables.

Additional inter- / extrapolation methods can easily be integrated. We should put #1028 on the todo list.

[modelica-trac-importer]

Comment by beutlich on 17 Jun 2014 13:53 UTC
Replying to [comment:2 dietmarw]:

* SimulationX installs its database files as .sdf, which is SQL Server Compact Edition Database File. Therefore it is confusing to reuse the file extension .sdf for HDF5 based table data. I propose to call it what is actually is (either .h5 or .hdf5) or use a less confusing extension (like .sdf5).

The file extension ".sdf" is used to link the files with the SDF Editor and to indicate that certain meta information is available (i.e. quantities, units, ...) which might not be present in a "standard" HDF5 file. The users can store their files with whatever file extension they want.

I know that I can call it anyhow but please do not call it .sdf which is already taken and increases confusion. If you call it e.g. .sdf5 you can still link it to your SDF5 editor.

[modelica-trac-importer]

Comment by leo.gall on 17 Jun 2014 16:14 UTC
Just a quick link to the project website, if you want to inform people outside the Modelica Association:
http://simdevtools.org

[modelica-trac-importer]

Comment by sjoelund.se on 24 Jun 2014 12:02 UTC
My concern is that:

• The editor is Windows-based (although if it is an eclipse plugin it could work cross-platform; there are just no instructions for setup of an update site to install it from...)
• There are dependencies to DataFiles 1.0, which I have no clue what it is (ships with Dymola though).
• There are only Includes in the Modelica library. No C sources. Just the binary libraries (the sources are hidden in a directory called VisualStudio, for some reason).
  • The C sources in that directory cannot possibly correspond to the binaries since they should not compile (one C-file refers to an undeclared, static variable in another file; functions are used without prototype; etc)
  • There are temporary copies of object files, etc in subversion (always cleanup before svn add)
• The used library annotation says libzlib, but MSL conventions suggest z should be used instead (libz.so). Better would be to only put HDF5Table in the annotation and let the dynamic linker get the rest of the dependencies himself (there is this new thing called shared objects or dll's).
• I saw some odd-looking Modelica code like Integer dims[:] = { size(values, 1)} instead of Integer dims[1] = size(values) (the number of dimensions is always known), etc.

In short, it all looks like a mess to me and would be premature to include it in MSL. I was unable to test it due to the flaws.

Note: Binaries should not be part of the source repository. They can be provided elsewhere, like a zip uploaded to trac.

[modelica-trac-importer]

Comment by pharman on 24 Jun 2014 15:56 UTC
Due to the same issues Martin S has reported I could not test executing the models, however there are some compliance issues with the Modelica code:

• in HDF5Table.NDTable the evaluation of the interpolation is performed with y = evaluate(datasetID, u, interpMethod - 1, extrapMethod - 1);. Here interpMethod and extrapMethod are both enumerations, so subtracting 1 from each is not a valid expression.
• in HDF5Table.Examples.Components.Cell there is a reference to an absolute file on someone's machine parameter String filename = "C:/Users/markus.andres/Desktop/RRC_Battery.sdf";
• in HDF5Table.Examples.Battery there is a usage of an unspecified function classDirectory() in filename=classDirectory() + "RRC_Battery.sdf". Resources embedded in the library should be loaded with Modelica.Utilities.Files.loadResource().

There are also some style issues:

• the Battery example and associated components are duplicated in a directory HDF5Table/Examples/Battery, possibly a version control merge error
• I find it a little strange that all the function definitions are in HDF5Table/package.mo even though the classes NDTable and Recorder are in their own files in the same directory, not that there is anything specifically forbidden about this

Regarding the library overall I agree with the assessments by Martin S and Thomas B. It requires the MA to adopt the associated filetype and accompanying editor, which I don't think is practical, but without which I think the implementation is too closely linked with the filetype (and by filetype I don't just mean HDF5 but also the specific schema required on top of it). If there are elements of the implementation here that can be used to enhance the existing tables implementations this may suit the MSL better.

I'm interested what the practical limitations are of using "dense" data for interpolation is, I have generally used Kriging interpolation for higher numbers of dimensions, at some number of dimensions one technique becomes more practical than the other, and this would be a useful efficiency for the MA to study if we include N-dimensional interpolation in the MSL.

[modelica-trac-importer]

Modified by beutlich on 23 Oct 2014 08:37 UTC

[modelica-trac-importer]

Comment by dietmarw on 19 Dec 2014 07:01 UTC
Ticket retargeted after milestone closed

[modelica-trac-importer]

Comment by beutlich on 26 Nov 2015 07:40 UTC
I am going to close this as wontfix for the following reasons.

• The authors neglected most of the reviewer's comments and, apparently, did not show any more interest to add this proposal to the standard library.
• The authors rather decided to release the current implementation along with Dymola.
• The interpolation of N-dimensional table data is already part of open ticket Ideas for a new table call #1153. Thus the general idea is not closed by accident.

[modelica-trac-importer]

Comment by otter on 26 Nov 2015 09:53 UTC
I will include this topic to the (MSL group) agenda of the next Modelica Design Meeting

[modelica-trac-importer]

Comment by beutlich on 26 Nov 2015 10:21 UTC
Replying to [comment:10 otter]:

I will include this topic to the (MSL group) agenda of the next Modelica Design Meeting

Hopefully the MSL group !(?) will take notice of the comments given here.

[modelica-trac-importer]

Comment by dietmarw on 8 Jan 2016 19:07 UTC
Ticket retargeted after milestone closed

[modelica-trac-importer]

Modified by tsommer on 7 Apr 2016 05:24 UTC

[modelica-trac-importer]

Comment by @t-sommer on 7 Apr 2016 12:22 UTC
I think most of the above issues are resolved as of version 0.2 of the SDF library (shipping with Dymola 2017).

• the functions now ship as a shared library (no need to link the model against HDF5)
• linear extrapolation (with Akima interpolation) now uses the boundry slope approach
• the SDF Editor is now a native executable (included in Dymola 2017)
• the NDTable is now also available in the included Python package

Planned improvements are to...

• provide the NDTable as a Simulink S-function
• add Linux support for all included libraries / apps
• add "Steffen" & "Butland-Fritsch" interpolation methods for 1D tables
• add support for event generation for time tables
• add support for different inter- and extrapolation methods per dimension
• move the interpolation code to a source file (to support platforms wihtout file system e.g. HiL)

3DS GmbH (formerly Modelon GmbH) is still commited to make SDF and the accompaining
libraries (except the SDF Editor) available to the open source community.

Please contact me if you want to try out the new version and have no access to the Dymola Beta program.

[beutlich]

Apart from the previous arguments, I am going to close this as wontfix for the following reasons.

• https://github.com/modelica-3rdparty/netCDF-DataReader and https://github.com/modelica-3rdparty/ApproxSpline are alternative libraries for bivariate interpolation
• It was decided once to not include any more 3rd-party C code dependecies (e.g. HDF5 sources)
• There is https://github.com/ScientificDataFormat/NDTable to access this library under a permissive license.

[chrbertsch]

Apart from the previous arguments, I am going to close this as wontfix for the following reasons.

* https://github.com/modelica-3rdparty/netCDF-DataReader and https://github.com/modelica-3rdparty/ApproxSpline are alternative libraries for bivariate interpolation

* It was decided once to not include any more 3rd-party C code dependecies (e.g. HDF5 sources)

* There is https://github.com/ScientificDataFormat/NDTable to access this library under a permissive license.

I would agree to these arguments, if we had a package manager for 3rd party Modelica libraries well integrated in all Modelica tools ...
So either MA should work in that direction (couldn't this be payed work to provide the framework), or we need nd-tables in the MSL ...

[beutlich]

Nothing happend on ND-tables since 2016-04-06 (see #1153 (comment)) when we got lost on the requirements for the file format(s).

Since this issue is closed, please use #1153 for your updates/requirements on ND-tables. Thank you.

[beutlich]

Comment by leo.gall on 17 Jun 2014 16:14 UTC
Just a quick link to the project website, if you want to inform people outside the Modelica Association:
http://simdevtools.org

@GallLeo That site no longer exists.

[beutlich]

For cross-reference, see also https://stackoverflow.com/q/55162261/8520615

[GallLeo]

Comment by leo.gall on 17 Jun 2014 16:14 UTC
Just a quick link to the project website, if you want to inform people outside the Modelica Association:
http://simdevtools.org

@GallLeo That site no longer exists.
I think that has been replaced by https://github.com/ScientificDataFormat