Bug: Multisorting - Extraction from pickle

[marcbue]

After dumping a multisorting using (multisorting.dump_to_pickle(output_folder / 'sorting.pkl')). It can't be revovered using se.load_extractor_from_pickle(output_folder / 'sorting.pkl'). The following error occurs:

---

KeyError Traceback (most recent call last)
in
----> 1 multisorting = se.load_extractor_from_pickle('/data/sarah/MEA/20210504/C25/829/low/1/merged/sorting.pkl')

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeextractors-0.9.6-py3.8.egg/spikeextractors/extraction_tools.py in load_extractor_from_pickle(pkl_file)
747 The loaded extractor object
748 """
--> 749 return BaseExtractor.load_extractor_from_pickle(pkl_file)
750
751

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeextractors-0.9.6-py3.8.egg/spikeextractors/baseextractor.py in load_extractor_from_pickle(pkl_file)
497 with open(str(pkl_file), 'rb') as f:
498 d = pickle.load(f)
--> 499 extractor = _load_extractor_from_dict(d['serialized_dict'])
500 if 'properties' in d.keys():
501 extractor._properties = d['properties']

KeyError: 'serialized_dict'

Looking at the keys of the pickled file it didn't create the 'serialized_dict' necessary for instantiating the extractor:

import pickle
with open(output_folder / 'sorting.pkl', 'rb') as f:
        d=pickle.load(f)
        print(d.keys())

gives:

dict_keys(['class', 'module', 'kwargs', 'dumpable', 'version', 'relative_paths', 'annotations', 'properties'])

[alejoe91]

Hi,

This is strange because when dumping to pickle the first thing that is dumped is the serialized_dict (https://github.com/SpikeInterface/spikeextractors/blob/master/spikeextractors/baseextractor.py#L211).

How did you obtain the multisorting object? A simple solution is to cache it first (sorting_cached = se.CacheSortingExtractor(multisorting, "my-path.npz") and then dumping sorting_cached to pickle.

[marcbue]

Hi,

so I reinstalled from master.

Here is how I created the multisorting:
Checking that I noted that I might have hacked something that might be the issue...

for exp in file_df_filtered_out_faulty.experiment.unique():
    filtered_file_df = file_df_filtered_out_faulty[file_df_filtered_out_faulty.experiment==exp].sort_values('file_code', axis=0)
    # list of data files you want to spike sort
    recording_files = filtered_file_df.destination_path.to_list()
    # list of intermediate folder where tmp and output files are saved
    intermediate_folders = filtered_file_df.destination_dir.apply(lambda x: x / 'intermediate').to_list()
    output_folder = filtered_file_df.destination_path.iloc[0].parents[3] / 'merged'
    faulty_multisortings = []
    recordings = []
    for (rec_path, inter_path) in zip(recording_files, intermediate_folders):
        cache_folder = inter_path / 'cache' / sorter
        print("Loading cached recording")
        rec_cache = se.load_extractor_from_pickle(cache_folder.parent / 'recording.pkl')
        # This is a bit awkward but otherwise this throws an error when building the multirecording
        rec = si.BinaryRecordingExtractor(file_paths=cache_folder.parent / 'recording.dat', 
                                          sampling_frequency=rec_cache.get_sampling_frequency(), 
                                          num_chan=rec_cache.get_num_channels(), 
                                          dtype='float32', 
                                          channel_ids=rec_cache.get_channel_ids(), 
                                          gain_to_uV=rec_cache.get_channel_gains()
                                         )
        rec.set_channel_locations(rec_cache.get_channel_locations())
        recordings.append(rec)
    multirecording = si.append_recordings(recordings)
    probe = Probe(ndim=2, si_units='um')
    probe.set_contacts(positions=rec_cache.get_channel_locations(), shapes='circle')
    probe.set_device_channel_indices(np.arange(rec_cache.get_num_channels()))
    multirecording = multirecording.set_probe(probe)
    multisorting = si.run_sorter(sorter, multirecording, output_folder=output_folder, verbose=True, remove_existing_folder=True)
    multisorting.dump_to_json(output_folder / 'sorting.json')

What worked for me so far is dumping it to json. Then I can get it back with se.load_extractor_from_json().
Getting a multisorting back like this I tried again:

ms = se.load_extractor_from_json(sorting_path)
ms.dump_to_pickle(sorting_path.parent / 'sorting.pkl')
ms1 = se.load_extractor_from_pickle(sorting_path.parent / 'sorting.pkl')

This throws:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-17-85c32ec59ab1> in <module>
----> 1 ms1 = se.load_extractor_from_pickle(sorting_path.parent / 'sorting.pkl')

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeextractors-0.9.6-py3.8.egg/spikeextractors/extraction_tools.py in load_extractor_from_pickle(pkl_file)
    747         The loaded extractor object
    748     """
--> 749     return BaseExtractor.load_extractor_from_pickle(pkl_file)
    750 
    751 

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeextractors-0.9.6-py3.8.egg/spikeextractors/baseextractor.py in load_extractor_from_pickle(pkl_file)
    497         with open(str(pkl_file), 'rb') as f:
    498             d = pickle.load(f)
--> 499         extractor = _load_extractor_from_dict(d['serialized_dict'])
    500         if 'properties' in d.keys():
    501             extractor._properties = d['properties']

KeyError: 'serialized_dict'

Trying this:
sorting_cached = se.CacheSortingExtractor(ms, sorting_path.parent / 'sorting_cache.npz')
Throws:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-19-d7f63a423ffd> in <module>
----> 1 sorting_cached = se.CacheSortingExtractor(ms, sorting_path.parent / 'sorting_cache.npz')

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeextractors-0.9.6-py3.8.egg/spikeextractors/cacheextractors.py in __init__(self, sorting, save_path)
    129             self._is_tmp = False
    130             self._tmp_file = save_path
--> 131         NpzSortingExtractor.write_sorting(self._sorting, self._tmp_file)
    132         NpzSortingExtractor.__init__(self, self._tmp_file)
    133         # keep Npz kwargs

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeextractors-0.9.6-py3.8.egg/spikeextractors/extractors/npzsortingextractor/npzsortingextractor.py in write_sorting(sorting, save_path)
     57         spike_labels = []
     58         for unit_id in units_ids:
---> 59             sp_ind = sorting.get_unit_spike_train(unit_id)
     60             spike_indexes.append(sp_ind)
     61             spike_labels.append(np.ones(sp_ind.size, dtype='int64')*unit_id)

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeinterface-0.90.0.dev0-py3.8.egg/spikeinterface/core/basesorting.py in get_unit_spike_train(self, unit_id, segment_index, start_frame, end_frame)
     53                              end_frame: Union[int, None] = None,
     54                              ):
---> 55         segment_index = self._check_segment_index(segment_index)
     56         S = self._sorting_segments[segment_index]
     57         return S.get_unit_spike_train(unit_id=unit_id, start_frame=start_frame, end_frame=end_frame)

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeinterface-0.90.0.dev0-py3.8.egg/spikeinterface/core/base.py in _check_segment_index(self, segment_index)
     61                 return 0
     62             else:
---> 63                 raise ValueError()
     64         else:
     65             return segment_index

ValueError: 

If I do not rebuld the recording with the infos from rec_cache following this code:

for exp in file_df_filtered_out_faulty.experiment.unique():
    filtered_file_df = file_df_filtered_out_faulty[file_df_filtered_out_faulty.experiment==exp].sort_values('file_code', axis=0)
    # list of data files you want to spike sort
    recording_files = filtered_file_df.destination_path.to_list()
    # list of intermediate folder where tmp and output files are saved
    intermediate_folders = filtered_file_df.destination_dir.apply(lambda x: x / 'intermediate').to_list()
    output_folder = filtered_file_df.destination_path.iloc[0].parents[3] / 'merged'
    faulty_multisortings = []
    recordings = []
    for (rec_path, inter_path) in zip(recording_files, intermediate_folders):
        cache_folder = inter_path / 'cache' / sorter
        print("Loading cached recording")
        rec_cache = se.load_extractor_from_pickle(cache_folder.parent / 'recording.pkl')
        recordings.append(rec_cache)
    multirecording = si.append_recordings(recordings)
    probe = Probe(ndim=2, si_units='um')
    probe.set_contacts(positions=rec_cache.get_channel_locations(), shapes='circle')
    probe.set_device_channel_indices(np.arange(rec_cache.get_num_channels()))
    multirecording = multirecording.set_probe(probe)
    multisorting = si.run_sorter(sorter, multirecording, output_folder=output_folder, verbose=True, remove_existing_folder=True)
    multisorting.dump_to_json(output_folder / 'sorting.json')

I get the following error (should I open another issue for this?):

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<timed exec> in <module>

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeinterface-0.90.0.dev0-py3.8.egg/spikeinterface/core/segmentutils.py in append_recordings(*args, **kwargs)
     69 
     70 def append_recordings(*args, **kwargs):
---> 71     return AppendSegmentRecording(*args, **kwargs)
     72 
     73 

~/miniconda3/envs/spikesorting/lib/python3.8/site-packages/spikeinterface-0.90.0.dev0-py3.8.egg/spikeinterface/core/segmentutils.py in __init__(self, recording_list)
     36         sampling_frequency = rec0.get_sampling_frequency()
     37         dtype = rec0.get_dtype()
---> 38         channel_ids = rec0.channel_ids
     39 
     40         # check same carracteristics

AttributeError: 'BinDatRecordingExtractor' object has no attribute 'channel_ids'

[alejoe91]

@marcbue

I think your are mixing things up. In the new master branch (0.90.0), spikeextractors does not exist anymore. To be sure you're doing things right, I recommend uninstalling spikeextactors. Please take a look to the new docs for changes: https://spikeinterface.readthedocs.io/en/latest/

[marcbue]

Hi @alejoe91

sorry you are right I was mixing packages. I just tried the same approach using the si.load_extractor() function and loading from pickle works! Thanks for the clarification!

[alejoe91]

Closing the issue then :)