699847_2024-04-15 units incorrect len

[bjhardcastle]

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/npc_ephys/units.py", line 269, in make_units_table_from_spike_interface_ks25
    _ = future.result()
  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/opt/conda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/opt/conda/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/lib/python3.9/site-packages/npc_ephys/units.py", line 182, in _device_helper
    df_device_metrics["peak_channel"] = awc.peak_channels
  File "/opt/conda/lib/python3.9/site-packages/pandas/core/frame.py", line 4299, in __setitem__
    self._set_item(key, value)
  File "/opt/conda/lib/python3.9/site-packages/pandas/core/frame.py", line 4512, in _set_item
    value, refs = self._sanitize_column(value)
  File "/opt/conda/lib/python3.9/site-packages/pandas/core/frame.py", line 5253, in _sanitize_column
    com.require_length_match(value, self.index)
  File "/opt/conda/lib/python3.9/site-packages/pandas/core/common.py", line 573, in require_length_match
    raise ValueError(
ValueError: Length of values (750) does not match length of index (373)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/scripts/write_sessions_to_nwb.py", line 90, in <module>
    main()
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/scripts/write_sessions_to_nwb.py", line 86, in main
    write_sessions_to_cache(**kwargs)  # type: ignore[misc, arg-type]
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/scripts/write_sessions_to_nwb.py", line 75, in write_sessions_to_cache
    helper(
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/scripts/write_sessions_to_nwb.py", line 27, in helper
    npc_sessions.write_and_upload_session_nwb(
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/utils/cache.py", line 192, in write_and_upload_session_nwb
    path = session.write_nwb_zarr(path=path, metadata_only=metadata_only)
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/sessions.py", line 369, in write_nwb_zarr
    nwb = self.nwb if not metadata_only else self.metadata
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/sessions.py", line 332, in nwb
    analysis=self._analysis,
  File "/opt/conda/lib/python3.9/site-packages/npc_io/cached_property.py", line 78, in __get__
    val = self.func(instance)
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/sessions.py", line 750, in _analysis
    modules.append(self.all_spike_histograms)
  File "/opt/conda/lib/python3.9/site-packages/npc_io/cached_property.py", line 78, in __get__
    val = self.func(instance)
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/sessions.py", line 2632, in all_spike_histograms
    for probe in sorted(set(self.units.electrode_group_name[:])):
  File "/opt/conda/lib/python3.9/site-packages/npc_io/cached_property.py", line 78, in __get__
    val = self.func(instance)
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/sessions.py", line 1429, in units
    for column in self._units.columns:
  File "/opt/conda/lib/python3.9/site-packages/npc_io/cached_property.py", line 78, in __get__
    val = self.func(instance)
  File "/opt/conda/lib/python3.9/site-packages/npc_sessions/sessions.py", line 1368, in _units
    units=npc_ephys.make_units_table_from_spike_interface_ks25(
  File "/opt/conda/lib/python3.9/site-packages/npc_ephys/units.py", line 280, in make_units_table_from_spike_interface_ks25
    raise RuntimeError(
RuntimeError: Error fetching units for 699847_2024-04-15 - see original exception above/below

[arjunsridhar12345]

@bjhardcastle not sure if you noticed this already, but it looks like something went way out of whack. There's 2 folders for probe F, one with 373 units in the metrics and the other with 377 units in metrics. I think then when it makes the NWB it just gets everything for a probe, so it got both folders metrics (377 + 373 = 750).

Is this what you were saying about there being more in the NWB than metrics? At least the ones we saw yesterday had the same, but there might be more that are not like this.

[bjhardcastle]

Where has 2 folders? Thanks, Ben

…

________________________________ From: arjunsridhar12345 ***@***.***> Sent: Wednesday, May 22, 2024 9:32:36 AM To: AllenInstitute/npc_sessions ***@***.***> Cc: Ben Hardcastle ***@***.***>; Mention ***@***.***> Subject: Re: [AllenInstitute/npc_sessions] `699847_2024-04-15` units incorrect len (Issue #93) @bjhardcastle<https://github.com/bjhardcastle> not sure if you noticed this already, but it looks like something went way out of whack. There's 2 folders for probe F, one with 373 units in the metrics and the other with 377 units in metrics. I think then when it makes the NWB it just gets everything for a probe, so it got both folders metrics (377 + 373 = 750). Is this what you were saying about there being more in the NWB than metrics? At least the ones we saw yesterday had the same, but there might be more that are not like this. — Reply to this email directly, view it on GitHub<#93 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APD42FAEOPZWO6MWT4D4Z3LZDTCCJAVCNFSM6AAAAABG6UJ2G6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRVGI2DKMBRHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>