coreoff

A point's core distance in HDBSCAN is the distance to its min_samples-th nearest neighbour, the value used to build the mutual-reachability graph during fit. PredictionData, which caches core distances so approximate_predict and membership_vector can place new points, queries the tree with k=min_samples and takes the last column. A tree query returns the point itself at position 0 and its j-th nearest neighbour at position j, so k=min_samples gives position min_samples-1, the (min_samples-1)-th nearest neighbour, one neighbour too close. Every cached core distance is under-estimated, and it disagrees both with the fit convention and with the prediction module's own core distance for a new point.

The code

hdbscan, hdbscan/prediction.py and hdbscan/hdbscan_.py, byte-identical on current main:

# PredictionData.__init__, prediction.py line 171: k=min_samples -> the (min_samples-1)-th neighbour
self.core_distances = self.tree.query(data, k=min_samples)[0][:, -1]

# the fit paths, hdbscan_.py line 255 / 296: k=min_samples+1 -> the min_samples-th neighbour
tree.query(X, k=min_samples + 1, ...)[0][:, -1]

# the new point's OWN core distance, prediction.py line 252: the min_samples-th neighbour
point_core_distances = neighbor_distances[min_samples] * np.ones(...)

The fit paths and the prediction module's own new-point core distance take the min_samples-th neighbour; only the stored training-neighbour core distances take the (min_samples-1)-th. The two core distances that meet in approximate_predict, the new point's and its training neighbours', are therefore computed with different neighbours, which is self-inconsistent, so this is a bug rather than a convention. Training points do not expose it because approximate_predict hits an in-dataset fast path, which is why it is silent.

What it does

Driving the real library (fp64):

stored core == query(k=min_samples)     : True
stored core == query(k=min_samples+1)   : False  (the fit value)
under-estimated: 1.00 of points, mean 0.0632
correcting the core: 2/800 label flips, membership |Δ|max 0.376, predict-prob |Δ|max 0.768, GLOSH |Δ|max 0.125

Every cached core distance equals the k=min_samples query and differs from the fit's k=min_samples+1 query, under-estimated for all points. Correcting the cached core distances to the fit convention on the same fitted clusterer, changing only that one query, shifts the approximate_predict membership probability of some new point by 0.77, the membership vectors by 0.38, and the GLOSH outlier scores by 0.13, and flips the predicted cluster label of two of the eight hundred new points. The base case isolates the fault: the tree, the condensed cluster tree, and the prediction machinery are the clusterer's own; only the k in the core-distance query changes.

Scope

The fault triggers on the documented mainstream prediction path, prediction_data=True followed by approximate_predict / membership_vector / approximate_predict_scores, on default parameters, silently: no exception, just under-estimated core distances that feed wrong mutual-reachability into every prediction output. It is deterministic and not floating-point round-off; the base case with the corrected query removes every delta. The one-line fix is k=min_samples + 1 at prediction.py:171, matching the fit paths and the new-point core distance.

Layout

• excerpt.py: the stored core-distance query, the fit-path query, and the new-point core distance quoted with the flags that name the fault (BSD-3).
• coredist.py: the core distance as the k-th nearest neighbour, so k=min_samples returns the (min_samples-1)-th neighbour and under-estimates the true min_samples-th.
• consequence.py: the real cached core distances against the two tree queries, and the approximate_predict outputs before and after correcting the core to the fit convention.
• test_coreoff.py: the model buggy core is one neighbour closer, the real stored core equals query(k=min_samples) and under-estimates the fit value for every point, and correcting it shifts the membership probabilities, the GLOSH scores, and the labels.

Reproduce

python coredist.py
python consequence.py
python test_coreoff.py

The queries are quoted from current main; the numbers are produced by the real clusterer. The fix is k=min_samples + 1, so the cached core distance is the min_samples-th nearest neighbour as the fit uses.