Reference results of model with very high frequency oscillations cannot be reliably reproduced

[casella]

We are testing the results obtained on MSL 3.2.3 models with StopTime annotation in OpenModelica, comparing the results to the reference files produced by MAP-LIB using the CSV compare tool.

Two variables of the Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMEE_Generator_MultiPhase have sinusoidal oscillations with a period of less than 0.02 s, while the simulation lasts 30 seconds.

I understand it is nearly impossible to avoid a verification error on those two variables unless an impossibly small tolerance is used. The fact that the remaining 9 are verified successfully suggests that the model is simulated correctly and that those two variables are not reliable to check the results across different tools and solvers.

@christiankral , @AHaumer, can you please confirm?

@beutlich, @GallLeo, please remove those two variables from the reference results of MSL 3.2.3.

@beutlich, please make sure the same problem is not present in the reference results of MSL 4.0.0.

[christiankral]

Two variables of the Modelica.Magnetic.FundamentalWave.Examples.BasicMachines.SMEE_Generator_MultiPhase have sinusoidal oscillations with a period of less than 0.02 s, while the simulation lasts 30 seconds.

I am afraid I do not exactly understand this issue. When I simulate the MSL 3.2.3 with Dymola 2020 I get the following result:

The period of these oscillations is exactly 0.02 s close to the stop time of the simulation.

If I simulate the same example with OpenModelica 1.16.0~dev-210-g53a3f5f and the standard simulation settings of the experiment the simulation fails:

[christiankral]

One more note: I noticed that you simulate this model with a tolerance of 1E-6 whereas the description of the simulation is based on 1E-7

[maltelenz]

The model clearly states it should be simulated with 1E-6 in the experiment annotation. If the result cannot be reproduced with that experiment annotation, maybe the experiment annotation needs to be changed?

[christiankral]

@GallLeo I assume that the reference results are created with the tolerance 1E-7. When comparing a particular version of the library with the reference results, the default tolerance 1E-6 is used. Is the way how the comparison is supposed to work?

[maltelenz]

The baseline generation uses more intervals and a tighter tolerance (as you can see in the creation.txt linked above). This is generally a good thing, I think. However, in my opinion, this has to be combined with an experiment setting in the model that makes it possible to reproduce the reference result accurately.

For this specific example, we (Wolfram SystemModeler) have to run the simulation with the same number of output points as the baseline generation to make it pass correctness checking against the baseline. In my opinion, that means the number of intervals is not high enough in the experiment annotation.

We didn't have to touch the tolerance in this case.

[HansOlsson]

But (ideally) the trajectory with fewer output points should match the reference in these output points it generates, so for verifying the reference results it is more a matter of exactly how the verification is done.

For the experiment-annotation it is more a matter of whether the result "looks good". If there are high-frequency oscillations we should avoid obvious aliasing-effect that completely removes the oscillations, but less obvious ones is to me more a matter of taste.

[maltelenz]

If the output points are a multiple, comparing them only in the points the trial result has data may work, but this is not generally true.

This is the beginning of the simulation for the model in question:

As you can see, we exactly match the baseline results if we have the exact same number of intervals. As soon as we deviate however, using the experiment annotation (blue) or some intermediate number of intervals (red), large correctness errors appear in this high frequency signal.

To me, this indicates that the number of intervals in the experiment annotation is to small for this example.

I strongly disagree that the experiment annotation is about whether the result "looks good". It should be a matter of whether the simulation generates accurate and predictable results.

[HansOlsson]

If the output points are a multiple, comparing them only in the points the trial result has data may work, but this is not generally true.

Correct, I was only thinking of a multiple of the base-line - since I would assume that people usually do rather large changes of the interval.

And in this case the interval was multiplied by 0.5 according to creation.txt; so a clear multiple.

I strongly disagree that the experiment annotation is about whether the result "looks good". It should be a matter of whether the simulation generates accurate and predictable results.

I don't see a major difference between our two positions, as one normally wants to evaluate something based on the trajectory - and whether that will be accurate or not depends on additional details. In this simulation result I don't think it matters that the 50Hz sine-signal is "only" sampled at 1ms, it seems accurate enough.

[casella]

If I simulate the same example with OpenModelica 1.16.0~dev-210-g53a3f5f and the standard simulation settings of the experiment the simulation fails:

@christiankral, the fact that the models simulates in the CI (which runs on Linux) and fails in Windows is odd, thanks for reporting that. FYI, I opened #5880 on that topic.

[casella]

I had a closer look at our CI report plots. In fact, I don't think the problem is that we don't have enough data points, the problem is that the reference signal and our signal are slightly different. In some intervals (e.g. around 0.1) they coincide at the minimum and maximum, in other intervals (e.g. around 6.7) they coincide around zero.

Also, if you switch off the high, low, and error signals and only keep reference (blue) and actual (red), you'll see that OMC's simulation result (in red) has the same amplitude throughout the simulation, while the reference one (in blue) has a bulge in the amplitude around time = 22.

It seem to me that the problem is probably that a very strict tolerance is needed in order to obtain the same results different tools and solvers. The experiment annotation has 1e-6, but I understand at least 1e-7 is probably needed to avoid this issue.

We have been discussing having different tolerances for "normal" experiments and for "MSL tool comparison" experiments. However, this is not yet standardized, so we can't use it for MSL 4.0.0.

As far as I know, @LeoGall generated the reference results by using whatever tolerances are declared in the experiment annotation. In the past, the MSL mostly had 1e-4 as default, but then we understood it was not good enough to compare results across tools, so we explicitly put it at 1e-6 everywhere. It seems this is really not enough for this test case.

@christiankral, may I suggest that you simply change the tolerance in the experiment annotation of this example to 1e-7 (also for the corresponding MSL 4.0.0)? I guess this may resolve the issue, and also generate a more correct simulation result in general.

Thank you!

[christiankral]

I confirm, using the simulation tolerance 1E-7 leads to a successful simulation result in OpenModelica 1.16.0~dev-211-g31c432b: