Read 1D or 2D arrays

[jfulem]

If I take the JSON example:

 model JSONTest "JSON file read test"
    extends Modelica.Icons.Example;
    parameter String setName="set1" "Parameter set name" annotation(choices(choice="set1" "First parameter set", choice="set2" "Second parameter set"));
    inner parameter ExternData.JSONFile jsonfile(fileName=Modelica.Utilities.Files.loadResource("modelica://ExternData/Resources/Examples/test.json")) "JSON file" annotation(Placement(transformation(extent={{-80,60},{-60,80}})));
    Modelica.Blocks.Math.Gain gain1(k=jsonfile.getReal(setName + ".gain.k")) annotation(Placement(transformation(extent={{-15,60},{5,80}})));
    Modelica.Blocks.Math.Gain gain2(k=Modelica.Utilities.Strings.scanReal(jsonfile.getString(setName + ".gain.k"))) annotation(Placement(transformation(extent={{-15,30},{5,50}})));
    Modelica.Blocks.Sources.ContinuousClock clock annotation(Placement(transformation(extent={{-50,60},{-30,80}})));
    final parameter Integer m = jsonfile.getArrayRows2D("table1") "Number of rows in 2D array";
    Modelica.Blocks.Sources.TimeTable timeTable(table=jsonfile.getRealArray2D("table1", 3, 2));
...
  end JSONTest;

Is it possible to use table=jsonfile.getRealArray2D("table1", m, 2) instead of (table=jsonfile.getRealArray2D("table1", 3, 2)?

It looks Dymola fails to expand the variable. Is there any workaround, please?
Thank you.

[beutlich]

Here's a minimal failing example for Dymola 2021.

model JSONTest "JSON file sizing test"
  parameter ExternData.JSONFile jsonfile(fileName=Modelica.Utilities.Files.loadResource("modelica://ExternData/Resources/Examples/test.json")) "JSON file"
    annotation(Placement(transformation(extent={{-80,60},{-60,80}})));
  parameter Integer m = jsonfile.getArrayRows2D("table1") "Number of rows in 2D array";
  parameter Real table[:,2] = jsonfile.getRealArray2D("table1", m, 2);
  annotation(uses(ExternData));
end JSONTest;

I raised this topic in modelica/ModelicaSpecification#2425 when @GarronFish from @Claytex invented a workaround to hide the C implementation of the external object in another set of C functions which are then exposed using the Modelica external function interface. Thus, the main idea was to not use external objects but rather external functions. In their case they simply acted from necessity to read array (dimensions) from XML files.

@HansOlsson It would be very appreciated to get your feedback either here or in modelica/ModelicaSpecification#2425 why it is not supported in Dymola (but works in other Modelica tools) and if it can be enhanced/clarified at all. Thank you.

[jfulem]

Well, it works fine with JModelica (Optimica Compiler Toolkit).

[Marvin-TMG]

Hi, if you want to load a dynamically sized table from a json file and set this at runtime, this can be done. I've recently implemented such a table in my own json library. The work around for me was to not assign the data from the json file directly to a matrix in Modelica, but rather pass it directly to the MSL table functions in C.

To achieve this, I did the following:

• Create your own copy of a CombiTable, modify it to accept a data access object and a stable name, see code snippet below
• Create a duplicate of the ExternalCombiTable1D, which instead references your own CombiTable data object
• Create a c-function which reads the data directly from the json and stores it into a table object, which is then passed to the MSL function ModelicaStandardTables_CombiTable1D_init2, see snippet below

Usage of Dynamic table:
DynamicTables.CombiTable1Ds combiTable1Ds(columns={20},tableName="bodyAero.Cz2D_table",jsondata=json);

Standard table:
Modelica.Blocks.Tables.CombiTable1Ds combiTable1Ds1(table=table2d, columns={20});

block CombiTable1D
  "Table look-up in one dimension (matrix/file) with n inputs and n outputs"
  extends Modelica.Blocks.Interfaces.MIMOs(final n=size(columns, 1));

  parameter String tableName=""
    "Table name on file or in function usertab (see docu)"  annotation (Dialog(group="Table data definition"));

  parameter DataAccess.JsonFile jsondata "Reference to JSON data access object";

  parameter Integer columns[:]=2:size(table, 2)
    "Columns of table to be interpolated"  annotation (Dialog(group="Table data interpretation"));

  parameter Modelica.Blocks.Types.Smoothness smoothness=Modelica.Blocks.Types.Smoothness.LinearSegments
    "Smoothness of table interpolation"  annotation (Dialog(group="Table data interpretation"));

  parameter Modelica.Blocks.Types.Extrapolation extrapolation=Modelica.Blocks.Types.Extrapolation.LastTwoPoints
    "Extrapolation of data outside the definition range"  annotation (Dialog(group="Table data interpretation"));

  parameter Boolean verboseExtrapolation=false
    "= true, if warning messages are to be printed if table input is outside the definition range"  annotation (Dialog(group="Table data interpretation", enable=extrapolation == Modelica.Blocks.Types.Extrapolation.LastTwoPoints or extrapolation == Modelica.Blocks.Types.Extrapolation.HoldLastPoint));

  final parameter Real u_min=Modelica.Blocks.Tables.Internal.getTable1DAbscissaUmin(tableID)
    "Minimum abscissa value defined in table";
  final parameter Real u_max=Modelica.Blocks.Tables.Internal.getTable1DAbscissaUmax( tableID)
    "Maximum abscissa value defined in table";

protected 
  parameter Modelica.Blocks.Types.ExternalCombiTable1D tableID = Types.ExternalCombiTable1D(
      jsondata.json,
      tableName,
      columns,
      smoothness,
      extrapolation) "External table object";

equation 
  assert(tableName <> "", "no table name given");

  if verboseExtrapolation and (
    extrapolation == Modelica.Blocks.Types.Extrapolation.LastTwoPoints or 
    extrapolation == Modelica.Blocks.Types.Extrapolation.HoldLastPoint) then
    for i in 1:n loop
      assert(noEvent(u[i] >= u_min), "
Extrapolation warning: The value u["
                                   + String(i) +"] (=" + String(u[i]) + ") must be greater or equal
than the minimum abscissa value u_min (="
                                        + String(u_min) + ") defined in the table.
",
 level=AssertionLevel.warning);
      assert(noEvent(u[i] <= u_max), "
Extrapolation warning: The value u["
                                   + String(i) +"] (=" + String(u[i]) + ") must be less or equal
than the maximum abscissa value u_max (="
                                        + String(u_max) + ") defined in the table.
",
 level=AssertionLevel.warning);
    end for;
  end if;

  if smoothness == Modelica.Blocks.Types.Smoothness.ConstantSegments then
    for i in 1:n loop
      y[i] = Modelica.Blocks.Tables.Internal.getTable1DValueNoDer(
        tableID,
        i,
        u[i]);
    end for;
  else
    for i in 1:n loop
      y[i] = Modelica.Blocks.Tables.Internal.getTable1DValue(
        tableID,
        i,
        u[i]);
    end for;
  end if;

end CombiTable1D;

class ExternalCombiTable1D
        "External object of 1-dim. table defined by matrix"
        extends ExternalObject;

        function constructor "Initialize 1-dim. table defined by matrix"
          extends Modelica.Icons.Function;
          input DataAccess.Types.ExternJSONFile json;
          input String name;
          input Integer columns[:];
          input Modelica.Blocks.Types.Smoothness smoothness;
          input Modelica.Blocks.Types.Extrapolation extrapolation=Modelica.Blocks.Types.Extrapolation.LastTwoPoints;
          output ExternalCombiTable1D externalCombiTable1D;

          // ModelicaStandardTables_CombiTable1D_init2_variable(cJSON *json, char *name, _In_ int* cols, size_t nCols, int smoothness, int extrapolation)
        external"C" externalCombiTable1D = ModelicaStandardTables_CombiTable1D_init2_variable(
                json,
                name,
                columns,
                size(columns, 1),
                smoothness,
                extrapolation) annotation (Include="#include \"include/jsonModelicaInterface.h\"");

        end constructor;

        function destructor "Terminate 1-dim. table defined by matrix"
          extends Modelica.Icons.Function;
          input ExternalCombiTable1D externalCombiTable1D;
        external"C" ModelicaStandardTables_CombiTable1D_close(externalCombiTable1D);
        end destructor;

      end ExternalCombiTable1D;

void * ModelicaStandardTables_CombiTable1D_init2_variable(cJSON *json, char *name, int* cols, size_t nCols, int smoothness, int extrapolation) {
	int nRow = 0; int nColumn = 0;
	get_size(json, name, &nColumn, &nRow);
		
	// ModelicaFormatMessage("nrows: %d - ncols: %d\n", nrows, ncols);
	double *table;

	// Allocate memory for data
	table = (double*) malloc(sizeof(double)*nColumn*nRow);
	if (table) {
		int ncolsout = 0; int nrowsout = 0;
		get_real_2d(json, name, &ncolsout, &nrowsout, table, nRow, nColumn);

		// ModelicaFormatMessage("nrowsout: %d - ncolsout: %d\n", nrowsout, ncolsout);
	} else {
		ModelicaError("Memory allocation error\n");
	}

	return ModelicaStandardTables_CombiTable1D_init2("NoName", "NoName",
        table, nRow, nColumn, cols, nCols, smoothness, extrapolation, 1);
}

[vsskanth]

@Marvin-TMG Thank you for sharing this. I went ahead and implemented your suggestion adapting it for ExternData JSON functions. The model compiles fine. However, I found that the custom CombiTable1D block does not initialize unless I have an instance of Modelica CombiTable1D in the model. If you faced this issue, how did you overcome this ?

More details and code to replicate this issue here:
https://stackoverflow.com/questions/66433865/cannot-initialize-model-with-custom-combitable1d-without-instance-of-modelica-co

[Marvin-TMG]

Hi vsskanth,

Yes, you are correct. I ran into that issue as well and it was quite frustrating. It has been a while since I looked into this. I have just checked and I have two example models one with tables and one without anything, both are working properly, so I believe I solved it.

If I remember correctly, you need to somehow force Dymola to load the libraries needed for the CombiTable1D blocks, even if you dont actually use these blocks in your model.

This is my constructor for the extern data object, maybe the annotations or the IncludeDirectory did the trick, I am not sure:

function constructor "Parse JSON file"
  extends Modelica.Icons.Function;
  input String filename;
  output ExternJSONFile json;
  external "C" json = init_json(filename) annotation(uses(Modelica), Library={"ModelicaStandardTables", "ModelicaIO", "ModelicaMatIO", "zlib"},
  IncludeDirectory={"modelica://DataAccess/Resources"},Include="#include \"include/jsonModelicaInterface.h\" ");
end constructor;

Can you somehow verify that your json object is accessible and that the table is actually calling the external C function, which should initialize the table data? The modelica format functions can help you to get some feedback from the C code, while debugging the issue.

EDIT: Sorry, my assumption was incorrect. I see that your problem is different and I have the same issue.
This works:

• Having a CombiTable1D and a dynamic table duplicate of this
• Have a model without a dynamic table and CombiTable1D table (compileation check)
  This does not work:
• Have the same modle without the CombiTable1D block

I am not sure how to solve this, sorry. I also dont have the time to investigate further at this point.

[tbeu]

FYI, I am working to solve the issue originally reported by @jfulem. But I have to admit I do not understand the use case and workaround posted by @Marvin-TMG .

[Marvin-TMG]

The usecase is to have a variable size table which is not predefined at compile time. The work around I posted will read the data from the JSON file, directly in C with variable size determined at runtime. This is then used to initialize a modelica table directly from the external C function (by calling ModelicaStandardTables_CombiTable1D_init2).
There are some problems with this approach as highlighted by vsskanth.

[vsskanth]

Thank you @Marvin-TMG for taking the time to respond.

To add some more context for @tbeu:

As you probably already know, Modelica array parameter sizes have to be known at compile time. This means the size of the array is fixed post-compile, even if you somehow manage to get jsonfile.getArrayRows2D() to set the array parameter size.

What I am trying to do here is to be able to read 2D arrays of unknown size from JSON, mainly for use in Modelica lookup tables. Modelica components such as Modelica.Blocks.Tables.CombiTable1Ds achieve this by directly reading the table and doing the interpolation, lookup etc. in C using an ExternalObject class (ex. Modelica.Blocks.Types.ExternalCombiTable1D), thereby completely bypassing the need to use an array parameter. However, Modelica lookup tables only support reading from TXT or MAT files and I would like to be able to do the same from JSON.

What @Marvin-TMG suggested is to duplicate the implementation of Modelica ExternalCombiTable1D and wrap their constructor function in C to feed in a pointer to a double array created using JSON read functions, which would then return a table object pointer of type Modelica.Blocks.Types.ExternalCombiTable1D and can be used in a duplicate of Modelica.Blocks.Tables.CombiTable1Ds to create a JSON based variable array lookup table.

However, this approach only works if you have atleast one other instance of Modelica.Blocks.Tables.CombiTable1Ds in the model.

Code to replicate this: https://github.com/vsskanth/ExternData.CustomTable

[tbeu]

OK, I see. It is solely for the purpose of lookup-tables read from Json file. This is not the main use-case of ExternData, but rather should be implemented in https://github.com/tbeu/ModelicaTableAdditions. The table blocks of ModelicaTableAdditions are drop-in replacements of the Modelica Standard Library, but with support of additional file formats (currently CSV). What do you think?

[vsskanth]

Thanks @tbeu I didnt know this existed. ModelicaTableAdditions is exactly what I want to be able to do. I will check it out.

[tbeu]

Well, it would not be a big deal to add Json read support there, too. Do you use MSL 3.2.x or MSL 4.0.0?

[vsskanth]

I use MSL 4.0.0, Dymola 2021x, Windows 10.

It would be great to have JSON support as we are currently making our custom table with a dummy Modelica.Blocks.Types.ExternalCombiTable1D inside the table to make this work with ExternData JSON Objects.

[tbeu]

OK. Does your Json array look much different than the one on https://github.com/modelica-3rdparty/ExternData/blob/master/ExternData/Resources/Examples/test.json?

[vsskanth]

Its the same. The code I shared uses test.json from ExternData as a reference file.

[tbeu]

OK, I've just updated https://github.com/tbeu/ModelicaTableAdditions (main branch) to read tables from JSON files. You can give it a try, if you like.

[vsskanth]

That worked. I also added an ExternData JSON object to the test experiment to try reading the array from the same file and it works without issues.

Thank you! Much appreciated.

[tbeu]

Yes, that should work,too, since both external objects link with the (same) parson dependency.

[tbeu]

Feel free to check my sponsor page at https://github.com/sponsors/tbeu. Thank you.

[overtaker]

Resolved in f68d4ee by introduction of separate read functions especially for Dymola: readArraySize1D, readArraySize2D, readArrayRows2D and readArrayColumns2D where applicable.