KS4 test matrix is failing on Kilosort 4.1.1 – 4.1.6 (NumPy 2.0 incompatibility)

[h-mayorquin]

cc @JoeZiminski @alejoe91

The Testing Kilosort4 workflow iterates over every published Kilosort release on PyPI and runs the SpikeInterface integration tests against each one. Kilosort versions 4.1.1 through 4.1.6 were released in such a way that they are incompatible with the latest NumPy versions (MouseLand/Kilosort#1026), and that was only fixed in Kilosort 4.1.7.

The consequence is that every PR now picks up six pre-existing red checks that have nothing to do with the PR's diff. This has been broken on main itself since the Sunday cron run on 2026-04-19; five consecutive scheduled runs have failed without anyone catching it because cron-workflow failures don't surface on the main repo UI. The breakage is visible on my PR #4588 where 4.1.1 – 4.1.6 are red despite the diff not touching any sorter code; the workflow run is at actions/runs/26117618646 and the AttributeError: module 'numpy' has no attribute 'in1d' trace is visible in the 4.1.1 job at job/76810948024.

What should we do with these releases going forward? Kilosort 4.1.1 – 4.1.6 are functional on NumPy < 2.0 but will always fail on any environment with NumPy ≥ 2.0; 4.1.7 onward works on either. Honestly, if the promise is that we keep working with those old versions we kind of need to put a ceiling on numpy which does not seem feasible... what do you think?

[JoeZiminski]

Hey @h-mayorquin thanks for catching this and the write up. So if I understand correctly the situations are:

1. users working with numpy < 2.0 and kilosort 4.1.1-4.16 can currently run these versions through spikeinterface, but they are untested. 4.1.1 was released in August 2025 and 4.1.7 in March 2026 so I guess there is quite a few people who might be running this combination.
2. everyone else running ks < 4.1.1 or >= 4.1.7 with numpy 2.0 or < 2.0

Some possible solutions in order of preference might be:

1. just copy and paste the workflow, adjusting it to run 4.1.1-4.1.6 only and install numpy < 2.0 (we could have to duplicate the workflow file and this. This is obviously hellish from a code perspective but at least we will have test coverage for all cases and as this isn't a often looked at part of the codebase, an explanatory readme should remedy most confusion. I guess eventually it can be deleted when numpy <2.0 goes out of support.
2. Test these locally once to check they work as-is and / or leave them untested going forward. We could add a warning if these versions are detected on the SI side, telling the user that they are untested and to upgrade to 4.1.7.
3. Raise an error if these versions are used to ensure the untested combination cannot be run by users, but I guess this would penalize people using the legitimate combination of numpy < 2.0 and these KS versions

Don't feel particularly strongly about any as its a difficult situation, but lean towards keeping full coverage as this might save us trouble down the line with users finding possible incompatibilities. What do you think?

Maybe we could get a bot to raise an issue if the cron job tests fail? Although, I think I tried this once on a repo and it was really annoying 😅 but would be nice to get notified about these somehow

[chrishalcrow]

Hello, it's actually a deprecation that happened at numpy=2.4: https://numpy.org/devdocs/release/2.4.0-notes.html#removed-numpy-in1d
You can see 4.1.1-4.1.7 passing tests here: https://github.com/SpikeInterface/spikeinterface/actions/runs/24307064388
which are using numpy=2.0.

[alejoe91]

Hello, it's actually a deprecation that happened at numpy=2.4: https://numpy.org/devdocs/release/2.4.0-notes.html#removed-numpy-in1d You can see 4.1.1-4.1.7 passing tests here: https://github.com/SpikeInterface/spikeinterface/actions/runs/24307064388 which are using numpy=2.0.

Thansk for looking into this guys! Then I think we could:

• pin numpy<2.4.0 in KS tests
• In KS wrapper, check numpy version and raise if >2.4.0.

What do you think?

[chrishalcrow]

We only need to raise if 4.1.1 <= KS <= 4.1.6 and numpy>=2.4. In the ci, I think it wouldn't be difficult to install different numpy's per ks version, we already do it with setuptools:

spikeinterface/.github/workflows/test_kilosort4.yml

Line 75 in af9d5aa

- name: Install legacy setuptools for KS4 < 4.0.19

[JoeZiminski]

Oh nice!! much better than copying the entire workflow 😅 if the test can be easily customized to cover the case then great!

[chrishalcrow]

It works! #4591
(Don't think I've ever one-shotted a gh-actions PR before!)