-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-37959][ML] Fix the UT of checking norm in KMeans & BiKMeans #35247
Conversation
friendly ping @srowen @huaxingao |
The assertion says that cluster centers are vectors with norm 1? yes I do not see a reason to expect that when the distance metric in use is Euclidean. I wonder how that ever worked? it will not be true in the general case. Is it just something that happens to work in the unit test? Fine to make the other assertions about the norm include an error term, yes. |
in each iteration, It seems just a special attribute in spark's impl, I'm not sure whether other impls normalize the centers too. |
OK but not for Euclidean distance? then I wonder how that test ever passed, unless the data just happens to produce centers at distance 1 from the origin |
Euclidean distance doesn't have this attribute.
they were not assertions, just expressions. |
Oh right, I'm not reading. OK yes delete them |
### What changes were proposed in this pull request? In `KMeansSuite` and `BisectingKMeansSuite`, there are some unused lines: ``` model1.clusterCenters.forall(Vectors.norm(_, 2) == 1.0 ``` For cosine distance, the norm of centering vector should be 1, so the norm checking is meaningful; For euclidean distance, the norm checking is meaningless; ### Why are the changes needed? to enable norm checking for cosine distance, and diable it for euclidean distance ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? updated testsuites Closes #35247 from zhengruifeng/fix_kmeans_ut. Authored-by: Ruifeng Zheng <ruifengz@foxmail.com> Signed-off-by: huaxingao <huaxin.gao11@gmail.com> (cherry picked from commit 789fce8) Signed-off-by: huaxingao <huaxin.gao11@gmail.com>
Merged to master/3.2. Thanks! |
thank you all! |
### What changes were proposed in this pull request? In `KMeansSuite` and `BisectingKMeansSuite`, there are some unused lines: ``` model1.clusterCenters.forall(Vectors.norm(_, 2) == 1.0 ``` For cosine distance, the norm of centering vector should be 1, so the norm checking is meaningful; For euclidean distance, the norm checking is meaningless; ### Why are the changes needed? to enable norm checking for cosine distance, and diable it for euclidean distance ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? updated testsuites Closes apache#35247 from zhengruifeng/fix_kmeans_ut. Authored-by: Ruifeng Zheng <ruifengz@foxmail.com> Signed-off-by: huaxingao <huaxin.gao11@gmail.com> (cherry picked from commit 789fce8) Signed-off-by: huaxingao <huaxin.gao11@gmail.com>
### What changes were proposed in this pull request? In `KMeansSuite` and `BisectingKMeansSuite`, there are some unused lines: ``` model1.clusterCenters.forall(Vectors.norm(_, 2) == 1.0 ``` For cosine distance, the norm of centering vector should be 1, so the norm checking is meaningful; For euclidean distance, the norm checking is meaningless; ### Why are the changes needed? to enable norm checking for cosine distance, and diable it for euclidean distance ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? updated testsuites Closes apache#35247 from zhengruifeng/fix_kmeans_ut. Authored-by: Ruifeng Zheng <ruifengz@foxmail.com> Signed-off-by: huaxingao <huaxin.gao11@gmail.com> (cherry picked from commit 789fce8) Signed-off-by: huaxingao <huaxin.gao11@gmail.com>
### What changes were proposed in this pull request? In `KMeansSuite` and `BisectingKMeansSuite`, there are some unused lines: ``` model1.clusterCenters.forall(Vectors.norm(_, 2) == 1.0 ``` For cosine distance, the norm of centering vector should be 1, so the norm checking is meaningful; For euclidean distance, the norm checking is meaningless; ### Why are the changes needed? to enable norm checking for cosine distance, and diable it for euclidean distance ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? updated testsuites Closes apache#35247 from zhengruifeng/fix_kmeans_ut. Authored-by: Ruifeng Zheng <ruifengz@foxmail.com> Signed-off-by: huaxingao <huaxin.gao11@gmail.com> (cherry picked from commit 789fce8) Signed-off-by: huaxingao <huaxin.gao11@gmail.com> (cherry picked from commit 5cf8108) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
In
KMeansSuite
andBisectingKMeansSuite
, there are some unused lines:For cosine distance, the norm of centering vector should be 1, so the norm checking is meaningful;
For euclidean distance, the norm checking is meaningless;
Why are the changes needed?
to enable norm checking for cosine distance, and diable it for euclidean distance
Does this PR introduce any user-facing change?
No
How was this patch tested?
updated testsuites