[SPARK-15938]Adding "support" property to MLlib Association Rule #13656
Conversation
| */ | ||
| @Since("2.1.0") | ||
| def support: Double = freqUnion.toDouble | ||
|
|
There was a problem hiding this comment.
This is intentionally typed as Double. In the future, it could be fraction value ( < 1.0).
There was a problem hiding this comment.
I don't think the meaning of this should ever be overloaded. Support is a count.
There was a problem hiding this comment.
Two major considerations:
- In most definition and text books,
supportis a fraction value in [0.0, 1.0]. It's possible for us to align with it in the future. - Current implementation of Association rule actually allows both
freqUnion: Double,and
freqAntecedent: Double
to be fraction value [0.0, 1.0] although they are both counts now. I don't want to destroy the flexibility and break API in the future.
There was a problem hiding this comment.
Ah right, we use support as a fraction. Well, then best to be consistent and return it as a fraction of the data set size. I can't imagine having a method sometimes return a value with one type of semantics and sometimes another. Just make two methods.
freqUnion however appears to be a count only, and is even explicitly called a 'frequency'.
There was a problem hiding this comment.
I suppose freqUnion is made as a Double on purpose for the same reason. (flexibility for the future)
Making support a fraction now requires that we must keep the dataset size info in FPGrowthModel and AssociationRule. Yet that would introduce API change in the constructor. I thought we should avoid breaking API between 2.0 and 2.1.
There was a problem hiding this comment.
Dunno, that seems like a mistake to me. It should be a Long if it's a count, and should expose alternative factory methods to accept input of different types if needed. Overloading one argument seems like a hack and I'd prefer not to extend it (or fix it).
See SPARK-15930 which concerns adding the input size just for this reason, I assume. We haven't released 2.0, and so could in theory still put in a change to the constructor. I agree, we might however have to deprecate the existing one, add a new one, and still deal with calls to the old constructor, which would mean it's not possible to compute values that are a fraction of the whole data set. This in turn may argue for clearly separating inputs/outputs that are counts vs percentages.
There was a problem hiding this comment.
I find it hard to just deprecating the old constructor and still keep support as a fraction if no dataset size is passed in.
There was a problem hiding this comment.
Yes, it's not possible to implement in that case. There's an argument for just adding the parameter and removing the old constructor for 2.0 in order to support this without the convolutions. I'd love to get a thumbs up from @jkbradley or @mengxr though.
There was a problem hiding this comment.
support should be a fraction to be consistent with the semantic of minSupport in FPGrowth and PrefixSpan. There should be a compatible way to add support. Rule is not a case class and its constructor is package private. So this should be easy to add. Another approach is to add total number of records in the model, so people can calculate the support easily.
|
Thanks for the review @srowen |
|
Test build #60475 has finished for PR 13656 at commit
|
|
Hm, I suppose the problem is that you're returning a count here though. See also SPARK-15930 which is related, and concerns tracking the total size of the input. |
|
Yes, I've linked the two issues and provided some illustration about the fraction/count choice in the description. |
|
@srowen I'm also working on the ml.fpm, in which it's easier to include more information in the model and rules. I would suggest:
|
|
Test build #61130 has finished for PR 13656 at commit
|
|
I made a quick change to demo what's it like if we pass the data size along FPGrowthModel and AssociationRules. |
|
Test build #61131 has finished for PR 13656 at commit
|
|
I'm not sure if this is something that would still be considered since we aren't doing new development for MLlib anymore. It might make more sense to work on https://issues.apache.org/jira/browse/SPARK-14503 and then implement this after. |
|
Close this and add the support to ml.fpm. #17280 |
What changes were proposed in this pull request?
jira: https://issues.apache.org/jira/browse/SPARK-15938
Supportis an indication of how frequently the item-set appears in the database. Besides confidence, "Support" is another critical property for Association rule.References:
https://en.wikipedia.org/wiki/Association_rule_learning
http://www.philippe-fournier-viger.com/spmf/index.php?link=documentation.php#allassociationrules
https://www-users.cs.umn.edu/~kumar/dmbook/ch6.pdf
Supportcan be either the count of appearances or the fraction within the dataset. I choose to use the count as:In the next steps, we could add constraint like minSupport as in other libraries. FPGrowthModel should also contains the size of the dataset.
How was this patch tested?
existing ut.