making a recording json serializable?

[khl02007]

@alejoe91 this is related to the old spikeextractors, sorry if this isn't the place to post. basically I'm trying to load an old recording extractor, convert it to a new recording (si >0.90), and then save it using the new API. Please see the example and error message below (of course in this case I should just open the file with the new API; my real problem is that some of the other stuff that we use rely on a custom extractor based on the older spikeextractors, which gives the same error upon save or extract_waveforms). how would I make this json serializable? I assume this means storing all the metadata in json format?

import spikeextractors
import spikeinterface as si
old_r = spikeextractors.BinDatRecordingExtractor(file_path='./r/traces_cached_seg0.raw',
                                                 sampling_frequency=30000,
                                                 numchan=4,
                                                 dtype='int16')
new_r = si.create_recording_from_old_extractor(old_r)
new_r.save(folder='r2', n_jobs=5, total_memory='5G')

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_2784207/555477451.py in <module>
      6                                                  dtype='int16')
      7 new_r = si.create_recording_from_old_extractor(old_r)
----> 8 new_r.save(folder='r2', n_jobs=5, total_memory='5G')

~/repos/spikeinterface/spikeinterface/core/base.py in save(self, **kwargs)
    517             return self.save_to_memory(**kwargs)
    518         else:
--> 519             return self.save_to_folder(**kwargs)
    520 
    521     def save_to_memory(self, **kwargs):

~/repos/spikeinterface/spikeinterface/core/base.py in save_to_folder(self, name, folder, dump_ext, verbose, **save_kwargs)
    581         provenance_file = folder / f'provenance.{dump_ext}'
    582         if self.check_if_dumpable():
--> 583             self.dump(provenance_file)
    584         else:
    585             provenance_file.write_text(

~/repos/spikeinterface/spikeinterface/core/base.py in dump(self, file_path, relative_to, folder_metadata)
    394         """
    395         if str(file_path).endswith('.json'):
--> 396             self.dump_to_json(file_path, relative_to=relative_to, folder_metadata=folder_metadata)
    397         elif str(file_path).endswith('.pkl') or str(file_path).endswith('.pickle'):
    398             self.dump_to_pickle(file_path, relative_to=relative_to, folder_metadata=folder_metadata)

~/repos/spikeinterface/spikeinterface/core/base.py in dump_to_json(self, file_path, relative_to, folder_metadata)
    420         file_path = self._get_file_path(file_path, ['.json'])
    421         file_path.write_text(
--> 422             json.dumps(check_json(dump_dict), indent=4),
    423             encoding='utf8'
    424         )

~/miniconda3/envs/nwb_datajoint/lib/python3.8/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    232     if cls is None:
    233         cls = JSONEncoder
--> 234     return cls(
    235         skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    236         check_circular=check_circular, allow_nan=allow_nan, indent=indent,

~/miniconda3/envs/nwb_datajoint/lib/python3.8/json/encoder.py in encode(self, o)
    199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
--> 201             chunks = list(chunks)
    202         return ''.join(chunks)
    203 

~/miniconda3/envs/nwb_datajoint/lib/python3.8/json/encoder.py in _iterencode(o, _current_indent_level)
    429             yield from _iterencode_list(o, _current_indent_level)
    430         elif isinstance(o, dict):
--> 431             yield from _iterencode_dict(o, _current_indent_level)
    432         else:
    433             if markers is not None:

~/miniconda3/envs/nwb_datajoint/lib/python3.8/json/encoder.py in _iterencode_dict(dct, _current_indent_level)
    403                 else:
    404                     chunks = _iterencode(value, _current_indent_level)
--> 405                 yield from chunks
    406         if newline_indent is not None:
    407             _current_indent_level -= 1

~/miniconda3/envs/nwb_datajoint/lib/python3.8/json/encoder.py in _iterencode_dict(dct, _current_indent_level)
    403                 else:
    404                     chunks = _iterencode(value, _current_indent_level)
--> 405                 yield from chunks
    406         if newline_indent is not None:
    407             _current_indent_level -= 1

~/miniconda3/envs/nwb_datajoint/lib/python3.8/json/encoder.py in _iterencode(o, _current_indent_level)
    436                     raise ValueError("Circular reference detected")
    437                 markers[markerid] = o
--> 438             o = _default(o)
    439             yield from _iterencode(o, _current_indent_level)
    440             if markers is not None:

~/miniconda3/envs/nwb_datajoint/lib/python3.8/json/encoder.py in default(self, o)
    177 
    178         """
--> 179         raise TypeError(f'Object of type {o.__class__.__name__} '
    180                         f'is not JSON serializable')
    181 

TypeError: Object of type BinDatRecordingExtractor is not JSON serializable

[alejoe91]

Hi @khl02007

The problem is that the old extractor also has the is_dumpable field but the dumping mechanism is slightly different. To solve this you can set the is_dumpable to False in the conversion class (just to test you can try to add new_r.is_dumpable=False before the save function)

[khl02007]

@alejoe91 thanks, that solved the problem. Should I submit a PR to set is_dumpable = False in the OldToNewRecording class?

[alejoe91]

@alejoe91 thanks, that solved the problem. Should I submit a PR to set is_dumpable = False in the OldToNewRecording class?

Yes please :)

[khl02007]

@alejoe91 another Q: when I load the old recording and turn it into a new recording, and then apply whitening prior to e.g. saving, I find that I have to set is_dumpable=False again. Should anything be changed to make is_dumpable persist?

[alejoe91]

Ideally the check on is_dumpable should be recursive, but I'll double check!

[samuelgarcia]

is_dumpable is an attribute. I wanted to rewrite it since long time and turn it into a function since long time but never find time to do it. With function approach it would be more easy maybe to handle the parent/children propagation of this when preprocess.