Problem when running `se.NumpyRecording.dump()`

[marcoistasy]

Hello!

I am running into the follwing AssertionErrorr when running se.NumpyRecording.dump(/path/to.json):

AssertionError                            Traceback (most recent call last)
~/d/n2b/n2b-visification/scripts/io.py in <module>
     25 # save RecordingExtractor
     26 print('Saving recording to {}'.format(OUTPUT_PATH))
---> 27 recording.dump(OUTPUT_PATH)
     28 print('Finished saving recording.')

~/anaconda/envs/visification/lib/python3.7/site-packages/spikeinterface/core/base.py in dump(self, file_path, relative_to, folder_metadata)
    394         """
    395         if str(file_path).endswith('.json'):
--> 396             self.dump_to_json(file_path, relative_to=relative_to, folder_metadata=folder_metadata)
    397         elif str(file_path).endswith('.pkl') or str(file_path).endswith('.pickle'):
    398             self.dump_to_pickle(file_path, relative_to=relative_to, folder_metadata=folder_metadata)

~/anaconda/envs/visification/lib/python3.7/site-packages/spikeinterface/core/base.py in dump_to_json(self, file_path, relative_to, folder_metadata)
    412             If not None, file_paths are serialized relative to this path
    413         '''
--> 414         assert self.check_if_dumpable()
    415         dump_dict = self.to_dict(include_annotations=True,
    416                                  include_properties=False,

AssertionError: 

However, the final AssertionError is not printing anything, so I am not quite sure what the issue is. The OUTPUT_PATH is 161.json, by the way.

[samuelgarcia]

NumpyRecording is not dumpable because it is a in memory object.
But you can save it
NumpyRecording.save(folder='/path/to/save')
dump() is an a alis for the to_dict() which callect kwargs to reconstrust the object but for numpy array it is not possible.

[marcoistasy]

'Lazy' loading was a priority for me, so i used se.BinaryRecordingExtractor.write_recording() to write the recording, then read it in with se.BinaryRecordingExtractor(), and then finally ran recording.dump().

[alejoe91]

'Lazy' loading was a priority for me, so i used se.BinaryRecordingExtractor.write_recording() to write the recording, then read it in with se.BinaryRecordingExtractor(), and then finally ran recording.dump().

@marcoistasy note that se.BinaryRecordingExtractor.write_recording() + se.BinaryRecordingExtractor() is exactly equivalent to recording = numpy_recording.save(folder='your-folder') (the save function, in addition, also writes properties and probe info)

[marcoistasy]

@alejoe91 thanks for commenting on this! does that mean that I can 'lazily' loading the file from numpy_recording.save(folder='your-folder') as well?

[samuelgarcia]

Yes.

cached = numpy_recording.save(folder='your-folder')
# and then later
cached2 = load_extractor('your-folder')

cached and cached2 are totaly equivalent and are lazy.

[marcoistasy]

I'm running into another issue when running recording = load_extractor(INPUT_PATH):

ValueError                                Traceback (most recent call last)
~/d/n2b/n2b-visification/scripts/sort.py in <module>
     27 
     28 # LOAD RECORDINGEXTRACTOR
---> 29 reording = load_extractor(INPUT_PATH)

~/anaconda/envs/visification/lib/python3.7/site-packages/spikeinterface/core/base.py in load_extractor(file_or_folder_or_dict, base_folder)
    758         return BaseExtractor.from_dict(file_or_folder_or_dict, base_folder=base_folder)
    759     else:
--> 760         return BaseExtractor.load(file_or_folder_or_dict, base_folder=base_folder)
    761 
    762 

~/anaconda/envs/visification/lib/python3.7/site-packages/spikeinterface/core/base.py in load(file_path, base_folder)
    484                     file = f
    485             if file is None:
--> 486                 raise ValueError(f'This folder is not a cached folder {file_path}')
    487             extractor = BaseExtractor.load(file, base_folder=folder)
    488 

ValueError: This folder is not a cached folder /mnt/e/visification/processed/neural/TWH161_spike_interface_output

I generated the folder using recording.save(folder=OUTPUT_PATH) and the recording object itself via recording = se.NumpyRecording(traces_list=np_traces, sampling_frequency=SAMPLING_FREQUENCY, channel_ids=np_names)

The exact files in the folder are provenance.json and traces_cached_seg0.raw. However, provenance.json contains only a single line: {"warning": "the provenace is not dumpable!!!"}

[alejoe91]

@marcoistasy don't you have also a cached.json file in the folder?

[marcoistasy]

This is the image of the folder:

And this is the exact script used to generate it:

#%%

import numpy as np
import spikeinterface.extractors as se

# CONSTANTS

PATIENT_ID = 1
SAMPLING_FREQUENCY = 32000

INPUT_PATH = 'X{}.npz'.format(PATIENT_ID)  
OUTPUT_PATH =  'X{}_spike_interface_output'.format(PATIENT_ID)  # make sure this folder does not exists prior to running script

# LOAD NP DATA FROM FILE

print('Loading numpy data from {}'.format(INPUT_PATH))
np_data = np.load(INPUT_PATH)
np_traces = np_data['traces'].transpose()
np_names = np_data['names']
print('Finished loading numpy data.')

# CAST NP DATA TO SPIKEINTERFACE AS A RECORDINGEXTRACTOR OBJECT

print('Constructing RecordingExtractor from numpy data')
recording = se.NumpyRecording(traces_list=np_traces, sampling_frequency=SAMPLING_FREQUENCY, channel_ids=np_names)
print('Finished constructing RecordingExtractor')

# SAVE RECORDINGEXTRACTOR

print('Saving recording to {}'.format(OUTPUT_PATH))
recording.save(folder=OUTPUT_PATH)
print('Finished saving recording.')

[samuelgarcia]

Very strange, you should have a "cached.json" file in the folder.
Could you update to master to be sure that you have evrything.

You you also play with chunks with this:

rec_saved = recording.save(folder=OUTPUT_PATH, chunk_memory="10M", progress_bar=True, n_jobs=1)
print(rec_saved)

[marcoistasy]

Chunking did the trick!