[SPARK-37959][ML] Fix the UT of checking norm in KMeans & BiKMeans

[zhengruifeng]

What changes were proposed in this pull request?

In KMeansSuite and BisectingKMeansSuite, there are some unused lines:

model1.clusterCenters.forall(Vectors.norm(_, 2) == 1.0 

For cosine distance, the norm of centering vector should be 1, so the norm checking is meaningful;

For euclidean distance, the norm checking is meaningless;

Why are the changes needed?

to enable norm checking for cosine distance, and diable it for euclidean distance

Does this PR introduce any user-facing change?

No

How was this patch tested?

updated testsuites

[zhengruifeng]

friendly ping @srowen @huaxingao

[srowen]

The assertion says that cluster centers are vectors with norm 1? yes I do not see a reason to expect that when the distance metric in use is Euclidean. I wonder how that ever worked? it will not be true in the general case. Is it just something that happens to work in the unit test?

Fine to make the other assertions about the norm include an error term, yes.

[zhengruifeng]

in each iteration, val newCenter = distanceMeasureInstance.centroid(sum, weightSum) is called to get the new center vector, and CosineDistanceMeasure will normalize the center. So the final vectors all have norm 1.

It seems just a special attribute in spark's impl, I'm not sure whether other impls normalize the centers too.

[srowen]

OK but not for Euclidean distance? then I wonder how that test ever passed, unless the data just happens to produce centers at distance 1 from the origin

[zhengruifeng]

Euclidean distance doesn't have this attribute.

then I wonder how that test ever passed

they were not assertions, just expressions.

[srowen]

Oh right, I'm not reading. OK yes delete them

[huaxingao]

Merged to master/3.2. Thanks!

[zhengruifeng]

thank you all!