New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitoring: allow for flexible storage options #298

Open
mstimberg opened this Issue Aug 4, 2014 · 5 comments

Comments

Projects
None yet
4 participants
@mstimberg
Member

mstimberg commented Aug 4, 2014

For the background, see #168. It would be nice to have some generic storage option for all monitors that would allow to switch a monitor from writing to memory to writing to disk. The implementation can hopefully make use of our Function mechanism.

@mstimberg mstimberg added this to the 2.0 milestone Aug 4, 2014

@romainbrette

This comment has been minimized.

Show comment
Hide comment
@romainbrette

romainbrette Aug 8, 2014

Member

Following up on the email from Johannes Rieke: I agree it would be nice to use a format that can be used by other software. This way we could imagine of using other tools (developed by other people) to visualize or analyze the data. But: we want to keep it as simple as possible and avoid over complicated standards.

Member

romainbrette commented Aug 8, 2014

Following up on the email from Johannes Rieke: I agree it would be nice to use a format that can be used by other software. This way we could imagine of using other tools (developed by other people) to visualize or analyze the data. But: we want to keep it as simple as possible and avoid over complicated standards.

@mstimberg

This comment has been minimized.

Show comment
Hide comment
@mstimberg

mstimberg Aug 8, 2014

Member

I think there are two issues here that we should probably deal with in separate issues: one is the issue of storage and the other is the one of representation. I was thinking of the first issue here, i.e. during a simulation, where do we write the monitor data. Writing it to memory might not be possible for long-running simulations and many recorded values. The representation issue is something more general and does not only apply to monitors. Even if the data is internally always stored as numpy arrays (as it currently is), we might want to allow to expose it in different ways, e.g. in a pandas DataFrame or a neo object. I think what Johannes had in mind was more about this question?

Member

mstimberg commented Aug 8, 2014

I think there are two issues here that we should probably deal with in separate issues: one is the issue of storage and the other is the one of representation. I was thinking of the first issue here, i.e. during a simulation, where do we write the monitor data. Writing it to memory might not be possible for long-running simulations and many recorded values. The representation issue is something more general and does not only apply to monitors. Even if the data is internally always stored as numpy arrays (as it currently is), we might want to allow to expose it in different ways, e.g. in a pandas DataFrame or a neo object. I think what Johannes had in mind was more about this question?

@mstimberg

This comment has been minimized.

Show comment
Hide comment
@mstimberg

mstimberg Aug 8, 2014

Member

I opened a new issue about exporting data to different formats as #306.

Member

mstimberg commented Aug 8, 2014

I opened a new issue about exporting data to different formats as #306.

@dabliss

This comment has been minimized.

Show comment
Hide comment
@dabliss

dabliss Apr 2, 2015

Contributor

I've learned the hard way that, as of now, monitors cannot be pickled. (Or am I doing something wrong?) Is there an easy way right now to save all of a monitor's data to disk?

In [14]: pickle.dump(me_pfc, open('me_pfc_spikes.pkl', 'w'))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-050720fd43f8> in <module>()
----> 1 pickle.dump(me_pfc, open('me_pfc_spikes.pkl', 'w'))

/usr/lib/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)

TypeError: can't pickle function objects

In [15]: type(me_pfc)
Out[15]: brian2.monitors.spikemonitor.SpikeMonitor
Contributor

dabliss commented Apr 2, 2015

I've learned the hard way that, as of now, monitors cannot be pickled. (Or am I doing something wrong?) Is there an easy way right now to save all of a monitor's data to disk?

In [14]: pickle.dump(me_pfc, open('me_pfc_spikes.pkl', 'w'))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-050720fd43f8> in <module>()
----> 1 pickle.dump(me_pfc, open('me_pfc_spikes.pkl', 'w'))

/usr/lib/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)

TypeError: can't pickle function objects

In [15]: type(me_pfc)
Out[15]: brian2.monitors.spikemonitor.SpikeMonitor
@mstimberg

This comment has been minimized.

Show comment
Hide comment
@mstimberg

mstimberg Apr 2, 2015

Member

Note that this github issue is not about storing the contents of a monitor after a run, it's about writing it directly to disk during the run.

Anyway, pickling the monitor objects is not the way to go in your case, you only want their data, not the complete objects including all their functionality. To export the stored data, you can use the get_states method. For a SpikeMonitor, this will return you a dictionary with the keys i and t, corresponding to the indices and times of the spikes. For StateMonitor, this is not working smoothly, unfortunately (fixing this is on my list...), the recorded values are named e.g. _recorded_v instead of just v, and a simple get_states() won't get them (as these are "internal" names). To get a useful dictionary out of a state monitor mon, you can use the following for now:

states = mon.get_states(['t'] + ['_recorded_' + v for v in mon.record_variables])

After you retrieved the data dictionaries from the monitor objects, you can of course pickle them to disk.

Member

mstimberg commented Apr 2, 2015

Note that this github issue is not about storing the contents of a monitor after a run, it's about writing it directly to disk during the run.

Anyway, pickling the monitor objects is not the way to go in your case, you only want their data, not the complete objects including all their functionality. To export the stored data, you can use the get_states method. For a SpikeMonitor, this will return you a dictionary with the keys i and t, corresponding to the indices and times of the spikes. For StateMonitor, this is not working smoothly, unfortunately (fixing this is on my list...), the recorded values are named e.g. _recorded_v instead of just v, and a simple get_states() won't get them (as these are "internal" names). To get a useful dictionary out of a state monitor mon, you can use the following for now:

states = mon.get_states(['t'] + ['_recorded_' + v for v in mon.record_variables])

After you retrieved the data dictionaries from the monitor objects, you can of course pickle them to disk.

@thesamovar thesamovar removed this from the 2.0 milestone Jul 21, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment