KeyError when running GUI in curation mode

[LizzyMcKay]

Hello,

I sometimes encounter a strange bug. I sort some tetrode data using Mountainsort 5, then create a sorting analyzer, compute extensions and visualize the output using the Qt-based GUI with curation=True. Everything works fine. This is the relevant part of the pipeline:

ms5_outdir = '/home/<user>/arbeit/spike_interface/ms5_output'
os.path.isdir(ms5_outdir) and shutil.rmtree(ms5_outdir)
sortings_ms5 = ss.run_sorter( sorter_name='mountainsort5', folder=ms5_outdir,
                             recording=rec_all_grpd,
                             scheme='2',
                             detect_threshold=7.8, detect_sign=1, 
                             detect_time_radius_msec=0.5, scheme2_phase1_detect_channel_radius=100,
                             whiten=True, filter=True, 
                             n_jobs=6
                             )

analyzer = si.create_sorting_analyzer(sorting=sortings_ms5, recording = rec_all_grpd, 
                                      format="binary_folder",
                                      folder='/home/<user>/arbeit/spike_interface/sort_analyzer', overwrite=True,
                                      **job_kwargs)

analyzer.compute("random_spikes", method="uniform", max_spikes_per_unit=500)
analyzer.compute("waveforms", **job_kwargs)
analyzer.compute("templates", **job_kwargs)
analyzer.compute("noise_levels")
analyzer.compute("unit_locations", method="monopolar_triangulation")
analyzer.compute("isi_histograms")
analyzer.compute("correlograms", window_ms=50, bin_ms=1.)
analyzer.compute("principal_components", n_components=3, mode='by_channel_global', whiten=True, **job_kwargs)
analyzer.compute("quality_metrics", metric_names=["snr", "firing_rate"])
analyzer.compute("template_similarity")
analyzer.compute("spike_amplitudes", **job_kwargs)
analyzer.compute("spike_amplitudes")

sorting_analyzer = si.load_sorting_analyzer("/home/<user>/arbeit/spike_interface/sort_analyzer")
# run_mainwindow(sorting_analyzer, mode="web", curation=True)

sw.plot_sorting_summary(sorting_analyzer, curation=True, backend="spikeinterface_gui") # in the gui, perform curation & then "export to json"

However if I decide to change some Mountainsort parameters and run part of my pipeline again (i.e. all of the steps shown above), launching spikeinterface-gui sometimes fails with the following error:

sw.plot_sorting_summary(sorting_analyzer, curation=True, backend="spikeinterface_gui") # in the gui, perform curation & then "export to json"
Traceback (most recent call last):

  Cell In[52], line 1
    sw.plot_sorting_summary(sorting_analyzer, curation=True, backend="spikeinterface_gui") # in the gui, perform curation & then "export to json"

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface/widgets/sorting_summary.py:128 in __init__
    BaseWidget.__init__(self, data_plot, backend=backend, **backend_kwargs)

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface/widgets/base.py:87 in __init__
    self.do_plot()

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface/widgets/base.py:108 in do_plot
    func(self.data_plot, **self.backend_kwargs)

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface/widgets/sorting_summary.py:238 in plot_spikeinterface_gui
    run_mainwindow(

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface_gui/main.py:120 in run_mainwindow
    win = QtMainWindow(controller, layout_preset=layout_preset, layout=layout)

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface_gui/backend_qt.py:154 in __init__
    view._refresh()

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface_gui/view_base.py:113 in _refresh
    self._qt_refresh(**kwargs)

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface_gui/curationview.py:125 in _qt_refresh
    color = self.get_unit_color(unit_id)

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface_gui/view_base.py:124 in get_unit_color
    color = self.controller.get_unit_color(unit_id)

  File ~/conda/envs/mntenv2/lib/python3.14/site-packages/spikeinterface_gui/controller.py:421 in get_unit_color
    return self.colors[unit_id]

KeyError: np.int64(12)

The KeyError can fail with different ints.

To resolve this, I have to restart the kernel and run the whole pipeline again, which sometimes includes lengthy pre-processing steps. I suspect that spikeinterface-gui is somehow looking for units which were present in the previous sorting but are no longer present in the new sorting? Why, if both sorting and analyzer are re-calculated?

[chrishalcrow]

Hi @LizzyMcKay , this is strange!

Just to make sure I understand: you run the pipeline once and inspect the results in the GUI. This is fine. You then re-run the pipeline using different parameters, and inspect the results in the GUI again. But this time you get the error.

How do you run your code - a Jupyter notebook? A script? And do you run the pipeline twice during the same python session? So if you use a notebook: do you run the first pipeline and inspect, then the second and inspect without restarting?

[LizzyMcKay]

Yes, that is correct. Now I see that it also happens when I apply curation to the analyzer, re-compute the extensions and then try to launch the gui with curation=True. It seems that the error is triggered when something causes the number of units in the sorter/analyzer to change. Interestingly, the error is never triggered when I run the gui with curaiton=False.

I run the pipeline inside Spyder IDE 5.5.0. I just execute the code, then inspect with gui and if I don't like the sorting,I execute the same code with changes to some parameters (e.g. different detect_threshold for mountainsort). That's a good idea, I'll try Jupyter notebook / different IDE.

[chrishalcrow]

Thanks! I'm sure we have a bug, I'll look into it. ..

For now, I'm pretty sure it will be ok if you only run the GUI once per kernel. One option would be to generate your analyzers like you're doing, but launch the GUI from a separate kernel. Some ways of doing this:

From your terminal, with your environment active, run sigui "/home/<user>/arbeit/spike_interface/sort_analyzer"

Write a python script just for launching the GUI, could be called launch_gui.py and look like:

import blah blah
sorting_analyzer = si.load_sorting_analyzer("/home/<user>/arbeit/spike_interface/sort_analyzer")
sw.plot_sorting_summary(sorting_analyzer, curation=True, backend="spikeinterface_gui") # in the gui, perform curation & then "export to json"

then run python launch_gui.py from the terminal. Or just stick that python in another notebook running in another kernel.

[LizzyMcKay]

Thanks, loading the sorting analyzer in another console works flawlessly. On a side note, I could not confirm the bug in Jupyter notebook, so it probably is something IDE specific. Maybe a conflict between Spikeinterface's and Spyder's version of numpy?