[SPARK-17704][ML][MLlib] ChiSqSelector performance improvement.

[srowen]

What changes were proposed in this pull request?

Partial revert of #15277 to instead sort and store input to model rather than require sorted input

How was this patch tested?

Existing tests.

[yanboliang]

@srowen I think this PR may fail MiMa tests, since it makes binary incompatible change. The major disagreement between this and #15277 is whether to keep selectedFeatures sorted. I think both is OK and these two PRs have no difference comparing computation cost and others. So I still not prefer to make this change(due to binary incompatible), since we don't have strong requirements to make it. I'm still open to hear others' thoughts. Thanks!

[yanboliang]

We need sort cost in any case, and put it in fit/training or model has no difference. So I think if we want to introduce this binary incompatible change, there should be strong requirements or users complain. Thanks!

[SparkQA]

Test build #66103 has finished for PR 15299 at commit c0c0174.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

What is binary incompatible here?

[yanboliang]

Oh, isSorted is left and it's not introduce binary incompatible right now. Thanks for your remind. I'm neutral for this change. Thanks!

[srowen]

Yeah I left it deprecated because I really can't figure why it was left protected. If you're ok with this then yes I'd like to restore these parts from the original change. I think it makes slightly more sense API wise

[yanboliang]

@srowen It looks strange to left it protected, and deprecating it looks ok to me except someone tells me any reason. BTW, please update Python API docs to reflect that selectedFeatures is not necessary to be sorted. Thanks.

[mpjlu]

hi @srowen , is @transient needed for val selectedFeatures or val filterIndices, one of them?
is it good to define filterIndices lazy?

[srowen]

I don't think it can be transient, because if serialized, this data must be part of the model. It is just an array of ints. I don't think lazy makes sense because it is just holding on to an unsorted array to later sort it. But this is an array of, in any reasonable case, less than a thousand elements. quicksorting it is trivial.

Done, I added the selectedFeatures change.

[SparkQA]

Test build #66107 has finished for PR 15299 at commit 02a2b40.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

Hey folks - what exactly is the issue we're concerned about here for binary compat? SPARK-17017 is for 2.1, not branch-2.0.

Is MiMa really an issue since 2.1 is not released yet?

[yanboliang]

@MLnick This change will not break binary compatibility currently. It marks isSorted as deprecated and will break binary compatibility when we delete that method. This should be not a big issue then, since a deprecated method may not be used. What's your opinion about this change? Thanks.

[MLnick]

Ah sorry ok I was a bit confused - I thought the isSorted method had only been added after 2.0, but it has been around since the beginning. It is indeed public (well protected) so is a MiMa issue if removed.

[MLnick]

Yeah, I also fail to see why this needs to be exposed. +1 on deprecation.

[MLnick]

I wonder if we should say "since the model requires sorted indices, selectedFeatures will be sorted" or something - just to make it clear the model does have this requirement, but takes care of that itself?

[srowen]

I don't mind that, though, my original theory behind this little change was that the sorting is wholly an implementation detail that callers don't need to promise or be promised about these features. It's very small, but, do we need to even promise these are sorted here?

[srowen]

Merged to master