[SPARK-39446][MLLIB] Add relevance score for nDCG evaluation #36843

uchiiii · 2022-06-11T10:00:28Z

What changes were proposed in this pull request?

To add relevance score to evaluate nDCG in the function ndcgAt.
To extend the interface of constructor of RankingMetrics class.

Why are the changes needed?

The precise definition of nDCG is here on Wikipedia, where relevance score is used. Currently, the implementation of nDCG on spark (MLlib) treats this as binary (0 or 1). This PR is to extend the ndcgAt function to be able to treat relevance score.

Does this PR introduce any user-facing change?

I extended the interface of RankingMetrics class for ndcgAt function, so now it accepts RDD[(Array[T], Array[T], Array[Double])] or RDD[(Array[T], Array[T])]) as the constructor arguments while the RDD[(Array[T], Array[T])]) was only accepted.

How was this patch tested?

One test was added in mllib/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala

AmplabJenkins · 2022-06-11T11:47:17Z

Can one of the admins verify this patch?

srowen · 2022-06-13T23:14:44Z

mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala

-class RankingMetrics[T: ClassTag](predictionAndLabels: RDD[(Array[T], Array[T])])
-  extends Logging with Serializable {
+class RankingMetrics[T: ClassTag](
+    predictionAndLabels: RDD[(Array[T], Array[T], Array[(T, Double)])])


Hm, why does the last need to be (T, Double) pairs? Wouldn't this be an attribute of the ground truth? Looking this up via Map seems clunky later. The problem is not changing the binary signature, but, at least, how about a third array of Double only, that is parallel to the second array?

Thank you for your review.
You definitely have a point. I changed from (T, Double) to Double.

mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala

srowen · 2022-06-14T14:38:35Z

mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala

-          if (i < pred.length && labSet.contains(pred(i))) {
-            dcg += gain
+    predictionAndLabels
+      .map {


Can you revert these changes that made one line into 3? the original was better IMHO

mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala

srowen · 2022-06-14T14:42:25Z

mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala

+                  maxDcg += gain
+                }
+              } else {
+                if (i < pred.length) {


Does this not need labSet.contains(pred(i))?
I would imagine the only difference is the computation of gain, between these two cases

We manage this by relMap.getOrElse(pred(i), 0.0) below.

I would imagine the only difference is the computation of gain, between these two cases

Basically yes. So we could write in the way you suggest without getOrElse.

srowen · 2022-06-15T01:10:05Z

mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala

-      countRelevantItemRatio(pred, lab, k, k)
-    }.mean()
+    predictionAndLabels.map { case (pred, lab, _) =>
+          countRelevantItemRatio(pred, lab, k, k)


Unindent - just match how the code was

srowen · 2022-06-15T12:56:44Z

mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala

@@ -35,8 +35,15 @@ import org.apache.spark.rdd.RDD
 * @param predictionAndLabels an RDD of (predicted ranking, ground truth set) pairs.
 */
 @Since("1.2.0")


Oh, we need to update the @Since tags. I think you can somehow add a @Since annotation to the default constructor args? as it's since 3.4.0. I'm not sure exactly where it goes. The old constructor can remain since 1.2.0. If that doesn't work, maybe we can leave this constructor and add the new one as a new def this(...) since 3.4.0?

I made changes and updated the docs.

I think that we should refer to BinaryClassificationMetrics and MulticlassMetrics, in which RDD[_ <: Product] was used as the input.

since end users now mainly use the .ml, is there any plan to expose this function to .ml?

Sorry in advance for my poor understanding. I have some questions.

What is .ml ? Do you mean org.apache.spark.ml ?

RDD[_ <: Product] is used to makeMulticlassMetric class available to .ml ?

For the first one - yeah this change is in the 'older' .mllib package. I don't think there is an equivalent for it in the DataFrame-based .ml packages, so, maybe we can ignore that here. But if nDCG is supported in the .ml package somewhere and I forgot it, would be good to add it there too.

The declaration suggested here might actually work for both input types without a separate constructor. Try it maybe? if it works, yes, that is simpler, and lets this API support even more inputs

Thank you for your explanation!

Could you review this?
#36920

…thon ### What changes were proposed in this pull request? - Updated `RankingMetrics` for Java and Python - Modified the interface for Java and Python - Added test for Java ### Why are the changes needed? - To expose the change in #36843 to Java and Python. - To update the document for Java and Python. ### Does this PR introduce _any_ user-facing change? - Java users can use a JavaRDD of (predicted ranking, ground truth set, relevance value of ground truth set) for `RankingMetrics` ### How was this patch tested? - Added test for Java Closes #37019 from uchiiii/modify_ranking_metrics_for_java_and_python. Authored-by: uchiiii <uchikun17@gmail.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>

…cgAk ### What changes were proposed in this pull request? This PR fixes the condition to raise the following warning in MLLib's RankingMetrics ndcgAk function: "# of ground truth set and # of relevance value set should be equal, check input data" The logic for raising warnings is faulty at the moment: it raises a warning if the `rel` input is empty and `lab.size` and `rel.size` are not equal. The logic should be to raise a warning if `rel` input is **not empty** and `lab.size` and `rel.size` are not equal. This warning was added in the following PR: #36843 ### Why are the changes needed? With the current logic, RankingMetrics will: - raise incorrect warning when a user is using it in the "binary" mode (i.e. no relevance values in the input) - not raise warning (that could be necessary) when the user is using it in the "non-binary" model (i.e. with relevance values in the input) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No change made to the test suite for RankingMetrics: https://github.com/uchiiii/spark/blob/a172172329cc78b50f716924f2a344517deb71fc/mllib/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala Closes #42207 from guilhem-depop/patch-1. Authored-by: Guilhem Vuillier <101632595+guilhem-depop@users.noreply.github.com> Signed-off-by: Sean Owen <srowen@gmail.com>

…cgAk ### What changes were proposed in this pull request? This PR fixes the condition to raise the following warning in MLLib's RankingMetrics ndcgAk function: "# of ground truth set and # of relevance value set should be equal, check input data" The logic for raising warnings is faulty at the moment: it raises a warning if the `rel` input is empty and `lab.size` and `rel.size` are not equal. The logic should be to raise a warning if `rel` input is **not empty** and `lab.size` and `rel.size` are not equal. This warning was added in the following PR: #36843 ### Why are the changes needed? With the current logic, RankingMetrics will: - raise incorrect warning when a user is using it in the "binary" mode (i.e. no relevance values in the input) - not raise warning (that could be necessary) when the user is using it in the "non-binary" model (i.e. with relevance values in the input) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No change made to the test suite for RankingMetrics: https://github.com/uchiiii/spark/blob/a172172329cc78b50f716924f2a344517deb71fc/mllib/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala Closes #42207 from guilhem-depop/patch-1. Authored-by: Guilhem Vuillier <101632595+guilhem-depop@users.noreply.github.com> Signed-off-by: Sean Owen <srowen@gmail.com> (cherry picked from commit 72af2c0) Signed-off-by: Sean Owen <srowen@gmail.com>

…cgAk ### What changes were proposed in this pull request? This PR fixes the condition to raise the following warning in MLLib's RankingMetrics ndcgAk function: "# of ground truth set and # of relevance value set should be equal, check input data" The logic for raising warnings is faulty at the moment: it raises a warning if the `rel` input is empty and `lab.size` and `rel.size` are not equal. The logic should be to raise a warning if `rel` input is **not empty** and `lab.size` and `rel.size` are not equal. This warning was added in the following PR: apache#36843 ### Why are the changes needed? With the current logic, RankingMetrics will: - raise incorrect warning when a user is using it in the "binary" mode (i.e. no relevance values in the input) - not raise warning (that could be necessary) when the user is using it in the "non-binary" model (i.e. with relevance values in the input) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No change made to the test suite for RankingMetrics: https://github.com/uchiiii/spark/blob/a172172329cc78b50f716924f2a344517deb71fc/mllib/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala Closes apache#42207 from guilhem-depop/patch-1. Authored-by: Guilhem Vuillier <101632595+guilhem-depop@users.noreply.github.com> Signed-off-by: Sean Owen <srowen@gmail.com> (cherry picked from commit 72af2c0) Signed-off-by: Sean Owen <srowen@gmail.com>

…cgAk ### What changes were proposed in this pull request? This PR fixes the condition to raise the following warning in MLLib's RankingMetrics ndcgAk function: "# of ground truth set and # of relevance value set should be equal, check input data" The logic for raising warnings is faulty at the moment: it raises a warning if the `rel` input is empty and `lab.size` and `rel.size` are not equal. The logic should be to raise a warning if `rel` input is **not empty** and `lab.size` and `rel.size` are not equal. This warning was added in the following PR: apache#36843 ### Why are the changes needed? With the current logic, RankingMetrics will: - raise incorrect warning when a user is using it in the "binary" mode (i.e. no relevance values in the input) - not raise warning (that could be necessary) when the user is using it in the "non-binary" model (i.e. with relevance values in the input) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No change made to the test suite for RankingMetrics: https://github.com/uchiiii/spark/blob/a172172329cc78b50f716924f2a344517deb71fc/mllib/src/test/scala/org/apache/spark/mllib/evaluation/RankingMetricsSuite.scala Closes apache#42207 from guilhem-depop/patch-1. Authored-by: Guilhem Vuillier <101632595+guilhem-depop@users.noreply.github.com> Signed-off-by: Sean Owen <srowen@gmail.com>

github-actions bot added the MLLIB label Jun 11, 2022

uchiiii changed the title ~~[WIP] Add relevance score for nDCG evaluation~~ [WIP][MLLIB] Add relevance score for nDCG evaluation Jun 11, 2022

uchiiii changed the title ~~[WIP][MLLIB] Add relevance score for nDCG evaluation~~ [WIP][SPARK-39446][MLLIB] Add relevance score for nDCG evaluation Jun 11, 2022

uchiiii added 2 commits June 14, 2022 00:49

Add relevance to nDCG evaluation

631f91a

Format by ./dev/scalafmt

74ac0a5

uchiiii force-pushed the add_relevance_to_ndcg branch from 3289d65 to 74ac0a5 Compare June 13, 2022 15:50

uchiiii changed the title ~~[WIP][SPARK-39446][MLLIB] Add relevance score for nDCG evaluation~~ [SPARK-39446][MLLIB] Add relevance score for nDCG evaluation Jun 13, 2022

srowen reviewed Jun 13, 2022

View reviewed changes

uchiiii added 2 commits June 14, 2022 22:24

Remove the third input from (Int, Double) to Double

c78a4e2

Modify conditions

de7b922

srowen requested changes Jun 14, 2022

View reviewed changes

uchiiii added 2 commits June 15, 2022 09:13

Revert formating map code

075aa42

Change infix style to dot style

2ef33d5

srowen reviewed Jun 15, 2022

View reviewed changes

uchiiii added 2 commits June 15, 2022 10:23

Revert formating map code (remove indent)

3902ec8

Revert formating map code

95733af

srowen reviewed Jun 15, 2022

View reviewed changes

uchiiii added 2 commits June 15, 2022 23:16

Add since tags

6b3dc8b

Update docs

d653135

srowen approved these changes Jun 15, 2022

View reviewed changes

Update docs

a172172

srowen closed this in 8120899 Jun 16, 2022

uchiiii mentioned this pull request Jun 28, 2022

[SPARK-39446][MLLIB][FOLLOWUP] Modify ranking metrics for java and python #37019

Closed

guilhem-depop mentioned this pull request Jul 28, 2023

[SPARK-44585][MLLIB] Fix warning condition in MLLib RankingMetrics ndcgAk #42207

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-39446][MLLIB] Add relevance score for nDCG evaluation #36843

[SPARK-39446][MLLIB] Add relevance score for nDCG evaluation #36843

uchiiii commented Jun 11, 2022 •

edited

AmplabJenkins commented Jun 11, 2022

srowen Jun 13, 2022

uchiiii Jun 14, 2022 •

edited

srowen Jun 14, 2022

srowen Jun 14, 2022

uchiiii Jun 15, 2022 •

edited

srowen Jun 15, 2022

uchiiii Jun 15, 2022

srowen Jun 15, 2022

uchiiii Jun 15, 2022

zhengruifeng Jun 18, 2022

zhengruifeng Jun 18, 2022

uchiiii Jun 19, 2022

srowen Jun 19, 2022

uchiiii Jun 20, 2022

[SPARK-39446][MLLIB] Add relevance score for nDCG evaluation #36843

[SPARK-39446][MLLIB] Add relevance score for nDCG evaluation #36843

Conversation

uchiiii commented Jun 11, 2022 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

AmplabJenkins commented Jun 11, 2022

Choose a reason for hiding this comment

uchiiii Jun 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uchiiii Jun 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uchiiii commented Jun 11, 2022 •

edited

uchiiii Jun 14, 2022 •

edited

uchiiii Jun 15, 2022 •

edited