spyking_circus error .

[Aryo-Zare]

I'm not sure if this error is totally related to the sorter (spyking_circus), or spike_interface.
In any case, I encountered this frequently. Sometimes this error would be resolved if I change (increase) the 'detect_threshold' from the sorter parameters.

But such tricks did not work in this case. Reason : unlike my previous successful trials which were on relatively short duration recordings (about 3 minutes), this file (which constitues my main data) consists of 30 minutes of recording.
Hence I suspect this may be due to the high volume of input data, perhaps making some memory problem ?

ss.run_sorter( sorter_name='spykingcircus' , recording=rd_l_pps , detect_sign=0 , detect_threshold=8 , template_width_ms=2 , filter=False , auto_merge=0.6 , verbose=True , output_folder=r'/home/azare/groups/PrimNeu/Aryo/analysis/sort/srt_l_2')

I had activated the verbose=True while sorting : the following is the error from spyking circus itself :

Smart Search of good isolated spikes for the clustering (1/3)...


  0%|                                            |[00:00<?, ?it/s]

===================================================================================

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=   PID 26456 RUNNING AT nib03.int.lin-magdeburg.de

=   EXIT CODE: 9

=   CLEANING UP REMAINING PROCESSES

=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================

YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)

This typically refers to a problem with your application.

Please see the FAQ page for debugging suggestions


----------------------------  Error  -----------------------------

| Step "clustering" failed for reason Command '['mpiexec', '-np', '72', 'spyking-circus-subtask', 'clustering', '/home/azare/groups/PrimNeu/Aryo/analysis/sort/srt_l_2/recording.npy', '72', '0', 'False', 'False']' returned non-zero exit status 9.!

------------------------------------------------------------------

spykingcircus run time 3847.61s

Note: the initial steps of the sorting runs without any error. I did not copy all of the report (before the error) for the sake of conciseness.

And the subsequent error from python :

Traceback (most recent call last):

  File "/tmp/ipykernel_9621/1917434363.py", line 1, in <module>
    ss.run_sorter( sorter_name='spykingcircus' , recording=rd_l_pps , detect_sign=0 , detect_threshold=8 , template_width_ms=2 ,  filter=False , auto_merge=0.6 , verbose=True , output_folder=r'/home/azare/groups/PrimNeu/Aryo/analysis/sort/srt_l_2')

  File "/home/azare/anaconda3/envs/env_11/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 142, in run_sorter
    return run_sorter_local(**common_kwargs)

  File "/home/azare/anaconda3/envs/env_11/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 164, in run_sorter_local
    sorting = SorterClass.get_result_from_folder(output_folder)

  File "/home/azare/anaconda3/envs/env_11/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 281, in get_result_from_folder
    sorting = cls._get_result_from_folder(output_folder)

  File "/home/azare/anaconda3/envs/env_11/lib/python3.9/site-packages/spikeinterface/sorters/spyking_circus/spyking_circus.py", line 169, in _get_result_from_folder
    sorting = SpykingCircusSortingExtractor(folder_path=Path(output_folder) / 'recording')

  File "/home/azare/anaconda3/envs/env_11/lib/python3.9/site-packages/spikeinterface/extractors/spykingcircusextractors.py", line 67, in __init__
    raise Exception(spykingcircus_folder, " is not a spyking circus folder")

Exception: (PosixPath('/home/azare/groups/PrimNeu/Aryo/analysis/sort/srt_l_2/recording'), ' is not a spyking circus folder')

[Aryo-Zare]

I just found a similar issue posted on the spyking-cirus page :

spyking-circus/spyking-circus#377

The error reported by the sorter is similar. But seemingly the sorting process had already advanced further than in my case.
Even though the user was implementing it via docker. The user was also using spike-interface.

Unfortunately that issue is still open.

[alejoe91]

@yger any idea?

[Aryo-Zare]

I found the source of the problem. memory overflow.

As I had tested for high & low volume file sizes, the program would crash at processing large file volume sizes.

I tested on our LAB workstation (with 64 GB of RAM) & it ran without problem. Hence the problem is probably the amount of RAM available on the server. I had done previous analyses on the server. Though servers usually have a lot of memory, due to various people connecting with them, the amount available for each person may be low. I'm enquiring more about this issue from our administrator.

I also found a page dealing with a similar error. This error has probably been raised by MPI (OPEN MPI or MPICH).
https://stackoverflow.com/questions/71827149/what-does-signal-9-mean-in-mpi-how-to-debug

A possible solution for this may be decreasing the chunk size in each parallel process. Though I'm not expert in this. Would it be possible that like other spike-interface functions that have options for chunk-size & number of CPUs, these parameters would also be modifiable here ? I have a problem with the number of CPUs used in sorter that I may post as a question in another issue. In circus2 (currently possibly experimental) there are arguments for n_jobs & chunk_duration, but in the current version I didn't find similar parameters.

And as a suggestion, I would say in the spyking-circus documentation we can add a description of this error for people who encounter it for the first time. Or a better option would be modifying the error report that would direct the user to checking their memory status.

[yger]

Hi all

Indeed, such a non informative crash is likely to be because of some memory error. However, SC should work well with 32Gb of RAM for large dataset, so I don't know what exactly went wrong here. In Spyking Circus, yes, you can also modify the size of the chunks. In [data], chunk_size = 30 (by default, and in s). This can be decrease if needed

[Aryo-Zare]

Thanks a lot @yger for your response.

I use Spyking-cicrus indirectly via spike-interface. I've never ran it directly. Spike-interface itself doesn't have a parameter for this (chunksize).
I guess the command you mentioned ( data path / -c chunk_size = 30 ) is when running spyking-circus standalone. How can I instruct it via spike-interface ?

I continue the disscusion in #895.

[Aryo-Zare]

I found the solution to this : explained at #895.