Support for stochastic simulation added #54

hemilpanchiwala · 2020-07-28T17:54:20Z

Updates from this PR:

The new test file StochasticTestSuiteTest is added to SBSCL which simulates all the stochastic tests from the SBML Test Suite.
Complete output format of FERN is now updated according to the test suite result format and all the tests now simulate fine.
36 out of 39 tests now pass successfully (others fail as currently there is no support for rules and species with no substance units).
Also, as stochastic tests can fail sometimes due to randomness, so for passing the Travis CI, seed values for all the models have been collected and currently, tests are ran using these seed values, so tests run fast and pass successfully.
Minor fixes in CSVImporter and issues in test cases with the event (there was a triggering issue) that has been resolved now.
Repetitive initialization in the SBMLinterpreter class is removed.

With these updates and PR #53, stochastic simulation support is now added to SBSCL.
All the updates of the project are present at my blog posts at this link.

…tionSystem class

zakharc · 2020-08-01T10:47:02Z

@hemilpanchiwala: Please clarify:

36 out of 39 tests now pass successfully (others fail as currently there is no support for rules and species with no substance units).
Also, as stochastic tests can fail sometimes due to randomness, so for passing the Travis CI, seed values for all the models have been collected, and currently, tests are ran using these seed values, so tests run fast and pass successfully.

I see 5 errors (not fails) with NPE in the build log:

[[1;31mERROR[m] testBasicSEDMLExecutorForMiriamURNDefinedModel(org.simulator.sedml.SEDMLExecutorTest)  Time elapsed: 2.067 s  <<< ERROR!
java.lang.NullPointerException
	at org.simulator.sedml.SEDMLExecutorTest.testBasicSEDMLExecutorForMiriamURNDefinedModel(SEDMLExecutorTest.java:137)

[[1;31mERROR[m] testLeloupSBML(org.simulator.sedml.SEDMLExecutorTest)  Time elapsed: 1.024 s  <<< ERROR!
java.lang.NullPointerException
	at org.simulator.sedml.SEDMLExecutorTest.testSpecificationExample(SEDMLExecutorTest.java:228)
	at org.simulator.sedml.SEDMLExecutorTest.testLeloupSBML(SEDMLExecutorTest.java:153)

[[1;31mERROR[m] testRepressilator1(org.simulator.sedml.SEDMLExecutorTest)  Time elapsed: 0.138 s  <<< ERROR!
java.lang.NullPointerException
	at org.simulator.sedml.SEDMLExecutorTest.testSpecificationExample(SEDMLExecutorTest.java:228)
	at org.simulator.sedml.SEDMLExecutorTest.testRepressilator1(SEDMLExecutorTest.java:249)

[[1;31mERROR[m] testRepressilator(org.simulator.sedml.SEDMLExecutorTest)  Time elapsed: 0.057 s  <<< ERROR!
java.lang.NullPointerException
	at org.simulator.sedml.SEDMLExecutorTest.testSpecificationExample(SEDMLExecutorTest.java:228)
	at org.simulator.sedml.SEDMLExecutorTest.testRepressilator(SEDMLExecutorTest.java:200)

[[1;31mERROR[m] testIkappab(org.simulator.sedml.SEDMLExecutorTest)  Time elapsed: 0.03 s  <<< ERROR!
java.lang.NullPointerException
	at org.simulator.sedml.SEDMLExecutorTest.testSpecificationExample(SEDMLExecutorTest.java:228)
	at org.simulator.sedml.SEDMLExecutorTest.testIkappab(SEDMLExecutorTest.java:147)

Also, tons of warnings of this type:

WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.5: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.6: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation
WARNING: [VIOLATION]	(k1 < 10) && (k1 > 0) at time 6.4: 
Jul 30, 2020 3:28:58 AM org.simulator.sbml.SimpleConstraintListener processViolation

Or this:

 WARN (SBMLReader.java:1096) - Cannot find a parser for the  namespace
 WARN (SBMLReader.java:1324) - !!! event.isEndElement: there is a problem in your SBML file !!!!
 WARN (SBMLReader.java:1325) - This should never happen, there is probably a problem with the parsers used.
 Try to check if one needed parser is missing or if you are using a parser in development.
 WARN (SBMLReader.java:1096) - Cannot find a parser for the  namespace
 WARN (SBMLReader.java:1096) - Cannot find a parser for the  namespace
 WARN (SBMLReader.java:808) - Some characters cannot be read: type
 WARN (SBMLReader.java:1324) - !!! event.isEndElement: there is a problem in your SBML file !!!!
 WARN (SBMLReader.java:1325) - This should never happen, there is probably a problem with the parsers used.
 Try to check if one needed parser is missing or if you are using a parser in development.
 WARN (SBMLReader.java:808) - Some characters cannot be read:  Status report
 WARN (SBMLReader.java:1324) - !!! event.isEndElement: there is a problem in your SBML file !!!!
 WARN (SBMLReader.java:1325) - This should never happen, there is probably a problem with the parsers used.
 Try to check if one needed parser is missing or if you are using a parser in development.

Is this expected?

log.txt

hemilpanchiwala · 2020-08-01T10:55:21Z

@zakharc, the reason for Travis CI failing currently is that for the simulation of the SED-ML models, it downloads the models via a URN from biomodels. But currently, the site from which it downloads is down, so the tests are failing.

This is what gets downloaded currently as a SEDML model

<title>Apache Tomcat/8.0.24 - Error report</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}</style>
HTTP Status 404 - /biomodels-main/download
type Status report
message /biomodels-main/download
description The requested resource is not available.
Apache Tomcat/8.0.24

So, once the site runs fine, these tests will pass fine. Also, the SBMLReader warnings are due to this as it is interpreting the above code as a model.

hemilpanchiwala · 2020-08-01T10:59:12Z

For the tons of warnings of [VIOLATION]...., I don't have any idea about them currently as I have not worked on them. I will see that test file and try to figure out why this warnings come.

zakharc · 2020-08-01T11:37:49Z

@hemilpanchiwala:

so the tests are failing.

As I said in my previous post (and highlighted in bold text):
I see 5 errors (not fails) with NPE.
Fail: Our code does not meet its test criteria. That means that the test case fails due to assertions not being fulfilled.
Error: We have an unexpected situation during test execution.
Do we talk about the same tests?

the reason for Travis CI failing currently is that for the simulation of the SED-ML models, it downloads the models via a URN from biomodels. But currently, the site from which it downloads is down.

I am quite sure, that throwing NPE on this scenario is not the best idea. If this is the reason for the errors, we need a meaningful exception for that.

So, once the site runs fine, these tests will pass fine.

I think re-triggering the build would be a good idea. I would like to see it pass :)

For the tons of warnings of [VIOLATION]...., I don't have any idea about them currently as I have not worked on them. I will see that test file and try to figure out why these warnings come.

👍

hemilpanchiwala · 2020-08-01T12:26:13Z

@zakharc,

As I said in my previous post (and highlighted in bold text):
I see 5 errors (not fails) with NPE.
Fail: Our code does not meet its test criteria. That means that the test case fails due to assertions not being fulfilled.
Error: We have an unexpected situation during test execution.
Do we talk about the same tests?

Yes, I am talking about these tests only. I get the model as a string which I sent in my last message. Then this model is passed to SBMLReader which reads and only gives warnings (but simulation runs further). And at a point, when any field is accessed from this, it gets a null value. This is the reason why NPE is given.

I have run the tests again but currently, also the tests are failing. But I am sure that the Travis will pass once this thing is running as the fern branch has a passing Travis in my forked repository.

matthiaskoenig · 2020-08-02T20:22:13Z

src/main/java/fern/network/sbml/SBMLEventHandlerObserver.java

-	}
-
-}
+package fern.network.sbml;


I am confused on this file. It says everything changed, but this is clearly code which existed before somewhere ?! Same with the example/Dsmts.java above.

@matthiaskoenig, this happened due to different line separators in Windows (CRLF) and Ubuntu (LF). The file content remains the same only the line separators are different (as FERN is developed in Windows and I am working on Ubuntu).

…pdated download_bigg_models script

src/main/java/fern/simulation/Simulator.java

src/main/java/fern/simulation/algorithm/GillespieEnhanced.java

src/main/java/org/simulator/comp/CompSimulator.java

src/test/java/org/simulator/stochastic/StochasticTestSuiteTest.java

hemilpanchiwala · 2020-08-09T05:40:28Z

@zakharc, I have updated most of the requested changes. I have some questions in few which I have commented, once you answer that I will update that changes too.

…MLinterpreter

src/main/java/org/simulator/sedml/SedMLSBMLSimulatorExecutor.java

src/test/java/org/simulator/distance/MaxAbsDistanceTest.java

src/test/java/org/simulator/distance/MaxDivergenceToleranceTest.java

src/test/java/org/simulator/sbml/SBMLTestSuiteTest.java

src/test/java/org/simulator/sbml/SBMLTestSuiteWrapper.java

shalinshah1993

Added small changes but otherwise LGTM!

…tochasticTestSuiteTest

src/main/java/org/simulator/fba/FluxBalanceAnalysis.java

hemilpanchiwala and others added 12 commits July 21, 2020 23:46

fix issue in comparing stochastic test suite results

3a5b7dc

resolve event trigger issue

17de582

resolve an issue while executing events

4db0231

add predefined seeds to pass the sstochastic tests properly

65b83fc

add timeout for StochasticTestSuiteTest

fb49a71

add print results while testing the stochastic test suite

ba5f3c7

travis fix

fb06675

minor fix in CSVImporter

05df68d

remove repetitive initializations in SBMLinterpreter

2e05196

refactor SBMLinterpreter by separating the initialization to new Equa…

52a7b1b

…tionSystem class

minor variable names update

5b51adb

updated documentation comments

ce25e93

hemilpanchiwala added the gsoc label Jul 28, 2020

hemilpanchiwala requested review from matthiaskoenig, draeger, shalinshah1993 and zakharc July 28, 2020 17:54

hemilpanchiwala self-assigned this Jul 28, 2020

hemilpanchiwala added 3 commits July 28, 2020 23:26

minor fix

4b8e3f2

minor refactoring

05bc582

refactor SBMLTestSuiteRunnerWrapper

d185e5c

hemilpanchiwala added 3 commits August 1, 2020 19:19

minor fix

5814243

add output processed results as simulation proceeds in SimulatorExample

70c5866

add propertyChangeListener parameter to all solve() methods in solver

a0b216e

matthiaskoenig reviewed Aug 2, 2020

View reviewed changes

hemilpanchiwala added 3 commits August 6, 2020 21:46

minor updates

6ebeea1

update BiGG models download script and BiGGTest

9b92496

BiGGTest updated to verify reference solutions | minor fix in FBA | u…

c40ef21

…pdated download_bigg_models script

This was referenced Aug 7, 2020

FBA gives incorrect objective_value 0 for bigg models iMM1415 & iRC1080 #1

Closed

Run FBA tests on all BiGG models #4

Closed

This was linked to issues Aug 7, 2020

FBA gives incorrect objective_value 0 for bigg models iMM1415 & iRC1080 #1

Closed

Run FBA tests on all BiGG models #4

Closed

hemilpanchiwala mentioned this pull request Aug 8, 2020

Refactor SBMLTestSuiteRunnerWrapper hemilpanchiwala/SBSCL#3

Closed

hemilpanchiwala added 3 commits August 8, 2020 16:26

update multiple instances of SBMLinterpreter to one in FERN

5c2adab

remove SBMLNetwork dependency from some classes by using SBMLinterpreter

ddc9171

minor updates

41fb960

zakharc requested changes Aug 8, 2020

View reviewed changes

hemilpanchiwala added 6 commits August 9, 2020 11:14

update changes requested by @zakharc

56a1442

refactor SBMLTestSuiteTest

2644424

add method overloading for the solve() method in DESolver

11d14ea

remove SBMLNetwork dependency from other classes updating with the SB…

bbed49a

…MLinterpreter

refactor StochasticTestSuiteTest

088d929

minor updates

7555e15

shalinshah1993 approved these changes Aug 12, 2020

View reviewed changes