Add the ability to save the state of a simulation and return to it later

[andrewbriand]

One of the features we want to implement this summer is the ability to save the state of a simulation and return to it later, as if nothing had happened.

I'm thinking that saving the objects associated with RoadRunner with pickle might work, but I'm not sure how pickle would interact with rr::Variant and C++ types in general, so I'll look into that.

[hsauro]

Is pickle a python feature? If it is we can’t use it. Saving state should handled by the c++ code because this summer I have a student that will also port roadrunner to Julia. Also some people drive roadrunner from their own c/c++ and they’ll need the same functionality. Herbert

[andrewbriand]

Yes, pickle is a python feature so then we'll have to go with some other format.

It seems that the relevant data for the simulation is stored in rr::RoadRunnerImpl, so if I could create a function there that dumps and loads the model and state data in a format like JSON, I could create a constructor in rr::RoadRunner to rebuild a RoadRunner object and a method to save it to a file or a string.

The difficult part will definitely be saving the ExecutableModel, the integrator and steady state solver can simply be saved as strings, as the setIntegrator and setSteadyStateSolver methods will create the objects if they don't exist.

[andrewbriand]

In order to support the editing and creation of models later, the save data should probably include the model's rules, initial conditions, etc. as well as the dynamic state of the simulation.

[hsauro]

Yes it will have to include everything that is necessary. Make sure loading of a snap shot is fast. H

[matthiaskoenig]

Please on C/C++ level. Pickle is very inefficient, only working on python, can only handle simple data structures (will have lots of issues with C++ bindings and objects behind this bindings), and does not even work between python versions. I.e. things pickled with python 3.5 will most likely not work with python3.6. - Also have in mind if you want compatibility between roadrunner versions. I.e. should the stored objects be read by newer roadrunner versions (if you do some memory dump of objects this will most likely not work), or is this only a short lived object dump which will only be used within a single roadrunner version.

[hsauro]

I mentioned to Andrew not to use pickle but it has to be built in to Roadrunner. Herbert

[andrewbriand]

@hsauro @matthiaskoenig How do we want the interface for state saving to work? One way would be to pass roadrunner a number of steps or a certain duration to run the simulation for before pausing and then allow the user to save to a file afterwords.

[hsauro]

I was thinking, a user would simulate from 0 to 10 time units, then could opt to save the state at that point. eg m = r.simulate (0, 10, 100) r.saveState (filename) do other stuff r.loadState (fileName) m = r.simulate (10, 50, 100)

…

On Fri, Jun 21, 2019 at 10:09 AM Andrew Briand ***@***.***> wrote: @hsauro <https://github.com/hsauro> @matthiaskoenig <https://github.com/matthiaskoenig> How do we want the interface for state saving to work? One way would be to pass roadrunner a number of steps or a certain duration, or e to run the simulation for before pausing and then allow the user to save to a file afterwords. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#550?email_source=notifications&email_token=AAIBSDTRJOERLPXEUPEYMALP3UDGHA5CNFSM4HM3Z5B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJA7BI#issuecomment-504500101>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIBSDR7LEKB4IQXERBTPLTP3UDGHANCNFSM4HM3Z5BQ> .

-- Herbert Sauro, Associate Professor University of Washington, Bioengineering 206-685-2119, www.sys-bio.org hsauro@uw.edu Books: http://books.analogmachine.org/

[luciansmith]

ISTR that there's some function in roadrunner already that will save the model state itself as SBML. That seems like the best way to save that part of the information. The other information that would need to be saved is all the settings that the user set in roadrunner itself (tolerances, etc.), as well as the current value of the 'time' variable. Then you'd bundle up all of that into a single file (perhaps a combine archive with an SBML model and a 'roadrunner state' document?)

[hsauro]

You could do it by saving and loading the sbml plus other info but the loading of SBML is slow and we want to make loading and saving states fast. Herbert

…

On Fri, Jun 21, 2019 at 10:40 AM Lucian Smith ***@***.***> wrote: ISTR that there's some function in roadrunner already that will save the model state itself as SBML. That seems like the best way to save that part of the information. The other information that would need to be saved is all the settings that the user set in roadrunner itself (tolerances, etc.), as well as the current value of the 'time' variable. Then you'd bundle up all of that into a single file (perhaps a combine archive with an SBML model and a 'roadrunner state' document?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#550?email_source=notifications&email_token=AAIBSDT4ZIXQ556Z3XPBTI3P3UHBNA5CNFSM4HM3Z5B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJDQ4A#issuecomment-504510576>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIBSDVI5IWHYM5QCARYKEDP3UHBNANCNFSM4HM3Z5BQ> .

-- Herbert Sauro, Associate Professor University of Washington, Bioengineering 206-685-2119, www.sys-bio.org hsauro@uw.edu Books: http://books.analogmachine.org/

[matthiaskoenig]

Yes, this should be fast and simple syntax as Herbert proposed, i.e.,

r.saveState(f_out)
r.loadState(f_in)

HDF5 could be a good option which has very fast IO, supports hierarchical data, storing of arrays/matrices, and things like hashmaps (settings and integrator settings maps).
You could just store all the internal arrays to HDF5 in addition with the settings maps. This should scale very well to very large models, because you basically store the memory in a binary format.
This is extremely fast and compresses also very nice (I store most of my time course results as HDF5).

Also there are HDF5 libraries for most programming languages and one can inspect the data format. I.e. I could have a look at my state file and what is stored in it (important for debugging).

[andrewbriand]

I've finished implementing a way to save a roadrunner instance to a binary file and reload it (to avoid the overhead of parsing an sbml file). Currently there is no way to pause a simulation part way through though, @matthiaskoenig @hsauro if this is a feature we want it wouldn't be too hard to implement.