Skip to content

[SPARK-14159][ML] Fixed bug in StringIndexer + related issue in RFormula #11965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jkbradley wants to merge 2 commits into from
Closed

[SPARK-14159][ML] Fixed bug in StringIndexer + related issue in RFormula #11965

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5d0a0b3
Fixed bug in StringIndexer, and fixed problem caused by that fix in R…
jkbradley Mar 25, 2016
36bf730
Added StringIndexer unit test which failed before fix
jkbradley Mar 25, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 6 additions & 9 deletions mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ class RFormula(override val uid: String)
encoderStages += new StringIndexer()
.setInputCol(term)
.setOutputCol(indexCol)
prefixesToRewrite(indexCol + "_") = term + "_"
Copy link
Member Author

@jkbradley jkbradley Mar 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the bug fix, this now needs to be added. RFormula tests for factor-numeric and factor-factor interactions fail otherwise because of prefixes like "stridx_ie94k9ef90127_"

(term, indexCol)
case _ =>
(term, term)
Expand Down Expand Up @@ -229,7 +230,7 @@ class RFormulaModel private[feature](
override def copy(extra: ParamMap): RFormulaModel = copyValues(
new RFormulaModel(uid, resolvedFormula, pipelineModel))

override def toString: String = s"RFormulaModel(${resolvedFormula}) (uid=$uid)"
override def toString: String = s"RFormulaModel($resolvedFormula) (uid=$uid)"

private def transformLabel(dataset: DataFrame): DataFrame = {
val labelName = resolvedFormula.label
Expand Down Expand Up @@ -400,14 +401,10 @@ private class VectorAttributeRewriter(
val group = AttributeGroup.fromStructField(dataset.schema(vectorCol))
val attrs = group.attributes.get.map { attr =>
if (attr.name.isDefined) {
val name = attr.name.get
val replacement = prefixesToRewrite.filter { case (k, _) => name.startsWith(k) }
if (replacement.nonEmpty) {
val (k, v) = replacement.headOption.get
attr.withName(v + name.stripPrefix(k))
} else {
attr
val name = prefixesToRewrite.foldLeft(attr.name.get) { case (curName, (from, to)) =>
Copy link
Member Author

@jkbradley jkbradley Mar 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change was needed because of multiple prefixes being added (one for StringIndexer, and one for Interaction)

curName.replace(from, to)
}
attr.withName(name)
} else {
attr
}
Expand All @@ -416,7 +413,7 @@ private class VectorAttributeRewriter(
}
val otherCols = dataset.columns.filter(_ != vectorCol).map(dataset.col)
val rewrittenCol = dataset.col(vectorCol).as(vectorCol, metadata)
dataset.select((otherCols :+ rewrittenCol): _*)
dataset.select(otherCols :+ rewrittenCol : _*)
}

override def transformSchema(schema: StructType): StructType = {
Expand Down
7 changes: 3 additions & 4 deletions mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,15 +161,14 @@ class StringIndexerModel (
}

val metadata = NominalAttribute.defaultAttr
.withName($(inputCol)).withValues(labels).toMetadata()
.withName($(outputCol)).withValues(labels).toMetadata()
Copy link
Member Author

@jkbradley jkbradley Mar 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the bug

// If we are skipping invalid records, filter them out.
val filteredDataset = (getHandleInvalid) match {
case "skip" => {
val filteredDataset = getHandleInvalid match {
case "skip" =>
val filterer = udf { label: String =>
labelToIndex.contains(label)
}
dataset.where(filterer(dataset($(inputCol))))
}
case _ => dataset
}
filteredDataset.select(col("*"),
Expand Down
13 changes: 13 additions & 0 deletions mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,4 +210,17 @@ class StringIndexerSuite
.setLabels(Array("a", "b", "c"))
testDefaultReadWrite(t)
}

test("StringIndexer metadata") {
val data = sc.parallelize(Seq((0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")), 2)
val df = sqlContext.createDataFrame(data).toDF("id", "label")
val indexer = new StringIndexer()
.setInputCol("label")
.setOutputCol("labelIndex")
.fit(df)
val transformed = indexer.transform(df)
val attrs =
NominalAttribute.decodeStructField(transformed.schema("labelIndex"), preserveName = true)
assert(attrs.name.nonEmpty && attrs.name.get === "labelIndex")
}
}