Skip to content

Add mean shift clustering - #369

Merged
josevalim merged 4 commits into
elixir-nx:mainfrom
RicardoSantos-99:add-mean-shift
Sep 2, 2026
Merged

Add mean shift clustering#369
josevalim merged 4 commits into
elixir-nx:mainfrom
RicardoSantos-99:add-mean-shift

Conversation

@RicardoSantos-99

@RicardoSantos-99 RicardoSantos-99 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Closes the Mean Shift item on #135.

Mean shift moves every seed towards the mean of the samples within one
bandwidth of it until the seed settles, then keeps the strongest of the seeds
that landed on the same mode. The number of clusters comes out of the data
rather than being given, which is what the bandwidth controls.

iex> x = Nx.tensor([[1, 1], [1.2, 1.1], [8, 8], [8.1, 8.2]])
iex> model = Scholar.Cluster.MeanShift.fit(x, bandwidth: 1.0)
iex> Scholar.Cluster.MeanShift.prune(model).labels
#Nx.Tensor<
  s32[4]
  [1, 1, 0, 0]
>

Shape of the implementation

Every seed moves at once rather than one at a time, so an iteration is a single
pairwise matrix instead of one per seed. That costs O(seeds * samples) of
space, the same shape DBSCAN already pays through radius_neighbors. A
k-d tree would not help here: the query mean shift needs is "every sample
within the bandwidth", whose result size is not known until it runs, and
Scholar.Neighbors.KDTree only answers k nearest anyway.

fit/2 keeps one row per seed so the shapes stay static, marking the centers
that lost with :infinity and reporting how many survived, and prune/1 drops
them. This follows Scholar.Cluster.AffinityPropagation, which has the same
problem of not knowing the cluster count until it has run. prune/1 raises
rather than building an empty tensor when nothing survived.

Validation

Checked against scikit-learn 1.6.1 on 131 datasets: 11 hand-written edge cases
and 120 random ones spanning 2 to 25 samples, 1 to 4 features, bandwidths from
0.3 to 6.0, and 1 to 21 resulting clusters. Labels, centers and cluster counts
match on all of them, on the default backend and under EXLA.

That includes the order scikit-learn puts the centers in, which sorts by how
many samples a center gathered and breaks ties on the coordinates. The edge
cases cover a bandwidth wide enough to hold everything, one narrower than the
closest pair, samples that are all the same point, a single sample, one
feature, samples sitting exactly at the bandwidth, and seeds that never reach a
sample, which scikit-learn drops and so does this.

max_iterations and cluster_all each have a test where the option changes
the answer, and there is a jit_apply test.

Left out on purpose

bin_seeding and estimate_bandwidth are not here. Binning produces a number
of seeds that is only known at runtime, which does not fit a static shape, so
the cheap way in is the :seeds option, the way Trimap accepts precomputed
triplets. Both are worth a follow-up issue.

I added HDBSCAN and OPTICS because were never
added to groups_for_modules, so ex_doc filed them outside the Models group.

Mean shift moves every seed towards the mean of the samples within one
bandwidth of it until the seed settles, then keeps the strongest of the seeds
that landed on the same mode. The number of clusters comes out of the data
rather than being given.

Every seed is moved at once rather than one at a time, so an iteration is a
single pairwise matrix instead of one per seed. That costs O(seeds * samples)
of space, the same shape DBSCAN already pays through radius_neighbors.

fit/2 keeps one row per seed so the shapes stay static, marking the centers
that lost with :infinity and reporting how many survived, and prune/1 drops
them. This follows Scholar.Cluster.AffinityPropagation, which has the same
problem of not knowing the cluster count until it has run.

Validated against scikit-learn 1.6.1 on 131 datasets: 11 hand-written edge
cases and 120 random ones spanning 2 to 25 samples, 1 to 4 features and 1 to 21
clusters. Labels, centers and cluster counts match on all of them, including
the order scikit-learn puts the centers in, which breaks ties on the
coordinates.
Both were left out of groups_for_modules when they landed, so ex_doc filed them
outside the Models group the rest of the clustering algorithms sit in.
The seeds carry the while accumulator, so their type has to survive a pass of
the loop. Samples of a wider type promoted the moved seeds through Nx.dot and
the do-block then failed to match what it was given, which raised a CompileError
for a f64 sample set with f32 seeds. Merge the two types up front.

Also documents that :max_iterations counts moves, where scikit-learn's max_iter
checks the limit after moving and so takes one step more than the number given.
Two independent divergences, both invisible until a fit is cut short by
:max_iterations, and both changing the centers rather than only a reported
number.

The cap was checked before moving the seeds rather than after. scikit-learn
tests its limit once the move is done, so a run capped at k takes k + 1 steps
and reports k, while this took k steps and reported k. Passing the same limit to
both gave centers one step less converged here, and :iterations agreed with
n_iter_ only when the run converged on its own.

The weight that decides which of two centers on the same mode survives was
counted around where a seed landed. scikit-learn counts the neighborhood that
produced the center, before the last move. The two agree at a fixed point, so
this only showed up when the run was truncated, and it picked a different
representative of the same mode.

Measured over 200 datasets against scikit-learn 1.6.1, spanning max_iter of 1,
2, 3, 4, 7 and 300, of which 56 are cut short by the limit. Labels, centers,
cluster counts and iteration counts went from 194, 177, 199 and 56 out of 200
to 200 out of 200 on all four.
@josevalim
josevalim merged commit 2cd74f4 into elixir-nx:main Sep 2, 2026
2 checks passed
@josevalim

Copy link
Copy Markdown
Contributor

💚 💙 💜 💛 ❤️

@RicardoSantos-99
RicardoSantos-99 deleted the add-mean-shift branch September 2, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants