Add noise scaling tests. #232
Conversation
|
On a side note, I have been trying to figure out why there's occasional crashes in the tests. I have been rerunning the tests in a long My hypothesis have been that there's some obscure case in the SDE tests where the solution goes into negative and causes an error (this should have been accounted for when the test was constructed). However, when I run the SDE tests just a lot of time, and eventually get a crash, and investigate the parameters which yielded the crash, it is fine. When I got a crash in |
|
https://travis-ci.com/github/SciML/Catalyst.jl/builds/176849991#L526 Is that the "crash" you're talking about? Error reports should be more specific, because just saying crash could mean anything. If it's this, then yeah something went negative and the next MTK release should handle that. |
|
Ok, that travis report is more helpful, yes, that was what I guessed went wrong. The reason I used the very ambiguous "crash" was that when I wrapped the runtests files, or the solve_SDE tests file, in a |
|
It might be good to pare down the shear number of networks we test some though. Test really take a long time currently, with some test files allocating > 50GB of memory. |
|
We can split tests too. But yes, using one network to test many features is usually a good idea unless we're testing network generation itself. |
|
Or maybe there is a way to setup a standard vs full set of tests, where standard is what runs normally and is more compact, but we can occasionally trigger full to run more networks. |
|
If we split test groups, then we could do it like OrdinaryDiffEq where the full tests are only ran on CI and a subset are ran locally. |
|
Trying to reduce the rather excessive number of tests would be good. They were quite useful when written since they caught a few things. But at this stage, they mostly extend the test time. A shame to remove them though might come in useful. |
|
At a closer look, it turned out that this error was actually in the new tests that I added, and that these had a small chance to go into negative. I have modified to ensure that the chance of this happening is practically zero. |
|
Ok, I think I might finally be picking up a trail on what is going on here. Hopefully, I will have it nailed down soon. |
|
Ok, I think I have narrowed it down to something odd going on for certain SDE solvers SciML/StochasticDiffEq.jl#344 |
|
@TorkelE Looks like tests have now passed. OK to merge? |
|
yeah it should be good. |
At the risk of overdoing the tests...
This basically tests the noise scaling functionality recently added in MTK, but "in action". That is, it simulates a network with different levels of noise scaling and tests that the variation of the distributions scales properly.