Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression Testing MSL 3.2.2-beta.2 #1924

Closed
modelica-trac-importer opened this issue Jan 15, 2017 · 15 comments
Closed

Regression Testing MSL 3.2.2-beta.2 #1924

modelica-trac-importer opened this issue Jan 15, 2017 · 15 comments
Assignees
@dietmarw @wischhusen @christiankral @rfranke @AHaumer @MartinOtter @beutlich @hubertus65 @GallLeo @ECClauss
Labels
bug Critical/severe issue
Milestone
MSL3.2.2

Comments

@modelica-trac-importer
Copy link

Reported by leo.gall on 25 Feb 2016 09:17 UTC
This ticket collects result deviations of MSL 3.2.2-beta.2.
Library officers please inspect.

Reference:

  • MSL v3.2.1+build.4
  • Dymola 2014 FD01

To be tested:

  • MSL v3.2.2+build.0-beta.2
  • Dymola 2017 Dev 4

Methods for regression testing: Own mos-scripting + MA compare.exe for comparison.

Number of test cases:

  • Modelica: 366 compared to 278 (88 added)
  • ModelicaTest: 543 compared to 440 (106 added, 3 removed)
    For new test cases, results have been generated, which can be used as a baseline for comparing to other tools.

Blocks

ModelicaTest.Blocks.Mean

Electrical.Machines

Modelica.Electrical.Analog.Examples.AmplifierWithOpAmpDetailed

Magnetic

Modelica.Magnetic.QuasiStatic.FundamentalWave.Examples.BasicMachines.InductionMachines.IMC_DOL

  • No stop time annotation. Please add experiment(StopTime=1.0)

Mechanics

Modelica.MultiBody.Joints.JointUSP2

Modelica.Mechanics.MultiBody.Examples.Systems.RobotR3.fullRobot

  • compare.exe stopped working, due to problems when comparing discrete variables:
2016-01-28Z15:55:32 [     Warning ] axis1.gear.bearingFriction.startBackward is invalid! 2 errors have been found during validation. 
2016-01-28Z10:34:05 [       Error ] Exception during run: Der Index war außerhalb des Arraybereichs.”
  • Comparing continuous variables manually, no big deviations could be found.
  • Visible differences for mode signals
  • Maybe differences originate from new Dymola version?

Modelica.Mechanics.Rotational.Examples.HeatLosses

  • compare.exe stopped working, due to problems when comparing discrete variables:
2016-01-28Z10:46:41 [     Warning ] brake.startForward is invalid! 30 errors have been found during validation.
2016-01-28Z10:46:41 [       Error ] Exception during run: Der Index war außerhalb des Arraybereichs.
  • Comparing continuous variables manually, no big deviations could be found.
  • Visible differences also for oneWayClutch.stuck at 0.035 s
  • Maybe differences originate from new Dymola version?

Fluid

ModelicaTest.Fluid.Dissipation.Verifications.HeatTransfer.Plate.kc_overall

  • No stop time annotation. Please add experiment(StopTime=1.0)
    ModelicaTest.Fluid.Dissipation.Verifications.HeatTransfer.StraightPipe.kc_laminar_KC
  • No stop time annotation. Please add experiment(StopTime=1.0)

Access to full comparison reports

Access comparison reports of all test cases (HTML generated by MA/ITI compare.exe v2.0.0-rel):
Modelica
ModelicaTest

Basic documentation of the testing process is available here:
https://trac.modelica.org/Modelica/wiki/HowTo/MAP-LIB/ModelicaStandardLibrary/RegressionTesting

Access to result files

MA file server (including mat files):
[sftp://modelica.org:20022/files/RegressionTesting/ReferenceResults/MSL/tags/v3.2.2%2Bbuild.0-beta.2/]

Readme for accessing file server:
https://svn.modelica.org/projects/RegressionTesting/AccessInfo/read-access/README.html

SVN (csv files, r9071)
https://svn.modelica.org/projects/RegressionTesting/ReferenceResults/MSL/trunk
Attention: I only added results of new test models.
I did not yet replace old results, because it seems awkward to use beta.2 as a new reference.

If you have any questions replicating the test results, please don't hesitate to contact [mailto:leo.gall@ltx.de me]!

Migrated-From: https://trac.modelica.org/Modelica/ticket/1924
@modelica-trac-importer modelica-trac-importer added this to the MSL3.2.2 milestone Jan 15, 2017
@modelica-trac-importer modelica-trac-importer added the bug Critical/severe issue label Jan 15, 2017
@modelica-trac-importer
Copy link
Author

Comment by beutlich on 25 Feb 2016 10:52 UTC
I really wonder why I provide you the signals under test for Modelica tables if you do not use them in the end? See comment 2 of #1867.

@modelica-trac-importer
Copy link
Author

Comment by leo.gall on 25 Feb 2016 13:38 UTC
Replying to [comment:1 beutlich]:

I really wonder why I provide you the signals under test for Modelica tables if you do not use them in the end? See comment 2 of #1867.

My fault, I missed your attachment! I downloaded it now and it will be used for the next run.

@modelica-trac-importer
Copy link
Author

Comment by ahaumer on 25 Feb 2016 21:46 UTC
ModelicaTest.Fluid.Dissipation.Verifications.HeatTransfer.Plate.kc_overall and ModelicaTest.Fluid.Dissipation.Verifications.HeatTransfer.StraightPipe.kc_laminar_KC already have an annotation(experiment(StopTime=1.01, Tolerance=1e-005)).

Added an annotation(experiment(StopTime=1.5, Interval=0.001) to Modelica.Magnetic.QuasiStatic.FundamentalWave.Examples.BasicMachines.InductionMachines.IMC_DOL (1904de3)

@modelica-trac-importer
Copy link
Author

Comment by ahaumer on 25 Feb 2016 21:59 UTC
ModelicaTest.Blocks.Mean is clear:
The signal source has been changed from sine to cosine (and new blocks have been added for testing: rectifiedMean, rootMeanSquare, harmonic).
Please produce new reference results - thanks in advance!

@modelica-trac-importer
Copy link
Author

Comment by ahaumer on 25 Feb 2016 22:02 UTC
Just summarizing the 4 issues left:

  • Modelica.Electrical.Analog.Examples.AmplifierWithOpAmpDetailed
  • Modelica.MultiBody.Joints.JointUSP2
  • Modelica.Mechanics.MultiBody.Examples.Systems.RobotR3.fullRobot
  • Modelica.Mechanics.Rotational.Examples.HeatLosses

@modelica-trac-importer
Copy link
Author

Comment by otter on 26 Feb 2016 01:27 UTC
Modelica.Electrical.Analog.Examples.AmplifierWithOpAmpDetailed is discussed in #1741 (model is sensitive to tolerances). When comparing the results it is obvious that the new simulation results with Dymola 2017 Dev.4 and MSL 3.2.2 is the correct one: The result should give three (identical) periods and with Dymoal 2014 FD01, MSL 3.2.1, there is a small devication in the third period leading to the regression failure, whereas with Dymola 2017 Dev.4 and MSL 3.2.2 the three periods are identical.

To summarize, the new reference files are correct and this issue is clarified

@modelica-trac-importer
Copy link
Author

Comment by otter on 26 Feb 2016 01:32 UTC
Modelica.Mechanics.Rotational.Examples.HeatLosses is discussed in https://trac.modelica.org/Modelica/ticket/1741#comment:4

Short summary: The issue is the friction element where the internal discrete state of the friction is sensitive to small deviations (the continuous-time variables are o.k.).

To summarize: This issue is clarified.

@modelica-trac-importer
Copy link
Author

Comment by otter on 26 Feb 2016 01:33 UTC
Replying to [comment:3 ahaumer]:

ModelicaTest.Fluid.Dissipation.Verifications.HeatTransfer.Plate.kc_overall and ModelicaTest.Fluid.Dissipation.Verifications.HeatTransfer.StraightPipe.kc_laminar_KC already have an annotation(experiment(StopTime=1.01, Tolerance=1e-005)).

This was already added after Beta.2 was released (so was already fixed).

@modelica-trac-importer
Copy link
Author

Comment by otter on 26 Feb 2016 01:35 UTC
Modelica.Mechanics.MultiBody.Examples.Systems.RobotR3.fullRobot is discussed in https://trac.modelica.org/Modelica/ticket/1741#comment:14

Again, the issue is the friction element, where the internal discrete state "mode" is sensitive to small deviations (however, the continuous-time signals are fine).

To summarize: This issue is clarified.

@modelica-trac-importer
Copy link
Author

Comment by otter on 26 Feb 2016 01:49 UTC
Modelica.MultiBody.Joints.JointUSP2:
compare.exe reports a relative error of 1e-5 (and the tolerance is 1e-4, so the deviation is uncrictical; in the compare.exe plot the deviation is also not visible). I compared with the two Dymola versions and MSL versions and could not reproduce this tiny deviation (anyway it is uncritical if the relative error is 1e-5; since this occurs around phi=0, this might be an issue with the compare.exe tool).

To summarize: This issue is clarified.

@modelica-trac-importer
Copy link
Author

Comment by otter on 26 Feb 2016 01:50 UTC
Since all issues are explained, regression testing is successful and I close this ticket.

@modelica-trac-importer
Copy link
Author

Changelog modified by otter on 26 Feb 2016 01:50 UTC
Regression testing issues have been all explained

@modelica-trac-importer
Copy link
Author

Comment by sjoelund.se on 26 Feb 2016 21:08 UTC
I was a bit unsure where to put OM testing results against these reference files. I guess they can go here:

Modelica:
BuildModel Results: 342/366 succeeded
Simulation Results: 329/366 succeeded
Verified Results: 310/366 succeeded

ModelicaTest:
BuildModel Results: 502/515 succeeded
Simulation Results: 485/515 succeeded
Verified Results: 450/515 succeeded

Still a few problems with impure functions (Component x of variability PARAM has binding ... of higher variability VAR usually means ... is impure). And I saw a few things that are OM problems for sure.

I guess DoublePendulumInitTip revolute2.phi is one of the variables that does not make much sense to compare.

@modelica-trac-importer
Copy link
Author

Comment by hubertus on 26 Feb 2016 21:40 UTC
These are good numbers, but the really interesting part would be to see models that worked in 3.2.1, but did not in 3.2.2, and also an overview over newly added models that don't work.

@modelica-trac-importer
Copy link
Author

Comment by beutlich on 1 Mar 2016 10:42 UTC
Replying to [ticket:1924 leo.gall]:

2016-01-28Z10:34:05 [       Error ] Exception during run: Der Index war außerhalb des Arraybereichs.

FYI: I fixed the exception. As long as it's not merged yet you might want to use https://github.com/tbeu/csv-compare/tree/issue16 for a repeated run.

dietmarw pushed a commit to dietmarw/ModelicaStandardLibrary that referenced this issue Oct 2, 2017
Partly resolves modelica#1924 by adding an annotation(experiment(Stop…
1904de3 
…Time=1.5, Interval=0.001) to Modelica.Magnetic.QuasiStatic.FundamentalWave.Examples.BasicMachines.InductionMachines.IMC_DOL

git-svn-id: https://svn.modelica.org/projects/Modelica/trunk@9072 7ce873d0-865f-4ce7-a662-4bb36ea78beb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dietmarw dietmarw

@wischhusen wischhusen

@christiankral christiankral

@rfranke rfranke

@AHaumer AHaumer

@MartinOtter MartinOtter

@beutlich beutlich

@hubertus65 hubertus65

@GallLeo GallLeo

@ECClauss ECClauss

Labels
bug Critical/severe issue
Projects
None yet
Milestone
MSL3.2.2
Development

No branches or pull requests