[SPARK-13097][ML] Binarizer allowing Double AND Vector input types

[seddonm1]

This enhancement extends the existing SparkML Binarizer [SPARK-5891] to allow Vector in addition to the existing Double input column type.

A use case for this enhancement is for when a user wants to Binarize many similar feature columns at once using the same threshold value (for example a binary threshold applied to many pixels in an image).

This contribution is my original work and I license the work to the project under the project's open source license.

@viirya @mengxr

[sethah]

The Jira in the PR title is for adding a Binarizer and was already resolved. Is there a Jira for extending Binarizer to Vector types? Typically a Jira is created to discuss the merits and use cases and then a PR can be made against that Jira ticket.

[seddonm1]

Thanks @sethah. I was unsure if this was sufficiently different from the original code where they mentioned 'We need to discuss whether we should process multiple columns or a vector column.'

I have raised SPARK-13097 and renamed this issue.

[mengxr]

ok to test

[mengxr]

test this please

[mengxr]

Please keep logging on, which uses DEBUG level.

[mengxr]

Some minor comments. Overall it looks good to me.

[seddonm1]

Thanks @mengxr.

I have updated the pull request with your suggestions. I believe that we still need to use the .compressed even after moving the if (value > td) { so that we can 'Return a vector in either dense or sparse format, whichever uses less storage.' which will occur if all values in a dense vector are greater than the threshold.

Vectors.sparse(data.size, indices.result(), values.result()).compressed

[mengxr]

LGTM pending Jenkins

[mengxr]

test this please

[SparkQA]

Test build #51209 has finished for PR 10976 at commit ea07001.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

test this please

[SparkQA]

Test build #51255 has finished for PR 10976 at commit 6a6988c.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51261 has finished for PR 10976 at commit 8a8dc67.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

nit: we can use val here like:

val outCol: StructField = inputType match {
  case DoubleType =>
    BinaryAttribute.defaultAttr.withName(outputColName).toStructField()
  case _: VectorUDT =>
    new StructField(outputColName, new VectorUDT, true)
  case other =>
    throw new IllegalArgumentException(s"Data type $other is not supported.")
}

[viirya]

LGTM, besides a minor comment.

[SparkQA]

Test build #51278 has finished for PR 10976 at commit 9750238.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

Merged into master. Thanks!