[SPARK-7198] [MLLIB] VectorAssembler should output ML attributes #6452
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #33642 has finished for PR 6452 at commit
|
| // Schema transformation. | ||
| val schema = dataset.schema | ||
| lazy val first = dataset.first() | ||
| val attrs = $(inputCols).flatMap { c => |
There was a problem hiding this comment.
This does not remove "index" from attributes. Should it remove or fix that field? (It's sort of odd that attributes have an index field.)
There was a problem hiding this comment.
The indices gets reset in AttributeGroup:
Let's re-visit the attribute API in 1.5:)
|
LGTM other than those 2 items. Side comment: Where should we document that a column of Vectors needs to have Vectors of the same length? This is a general rule, but some users may not realize it. |
|
I don't have a good place in mind. Maybe in the home page of |
|
Test build #33675 has finished for PR 6452 at commit
|
|
LGTM pending tests |
|
Oh I guess tests are done |
|
I'll go ahead and merge this into master and branch-1.4 |
asfgit
pushed a commit
that referenced
this pull request
May 28, 2015
`VectorAssembler` should carry over ML attributes. For unknown attributes, we assume numeric values. This PR handles the following cases: 1. DoubleType with ML attribute: carry over 2. DoubleType without ML attribute: numeric value 3. Scalar type: numeric value 4. VectorType with all ML attributes: carry over and update names 5. VectorType with number of ML attributes: assume all numeric 6. VectorType without ML attributes: check the first row and get the number of attributes jkbradley Author: Xiangrui Meng <meng@databricks.com> Closes #6452 from mengxr/SPARK-7198 and squashes the following commits: a9d2469 [Xiangrui Meng] add space facdb1f [Xiangrui Meng] VectorAssembler should output ML attributes (cherry picked from commit 7859ab6) Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
jeanlyn
pushed a commit
to jeanlyn/spark
that referenced
this pull request
Jun 12, 2015
`VectorAssembler` should carry over ML attributes. For unknown attributes, we assume numeric values. This PR handles the following cases: 1. DoubleType with ML attribute: carry over 2. DoubleType without ML attribute: numeric value 3. Scalar type: numeric value 4. VectorType with all ML attributes: carry over and update names 5. VectorType with number of ML attributes: assume all numeric 6. VectorType without ML attributes: check the first row and get the number of attributes jkbradley Author: Xiangrui Meng <meng@databricks.com> Closes apache#6452 from mengxr/SPARK-7198 and squashes the following commits: a9d2469 [Xiangrui Meng] add space facdb1f [Xiangrui Meng] VectorAssembler should output ML attributes
nemccarthy
pushed a commit
to nemccarthy/spark
that referenced
this pull request
Jun 19, 2015
`VectorAssembler` should carry over ML attributes. For unknown attributes, we assume numeric values. This PR handles the following cases: 1. DoubleType with ML attribute: carry over 2. DoubleType without ML attribute: numeric value 3. Scalar type: numeric value 4. VectorType with all ML attributes: carry over and update names 5. VectorType with number of ML attributes: assume all numeric 6. VectorType without ML attributes: check the first row and get the number of attributes jkbradley Author: Xiangrui Meng <meng@databricks.com> Closes apache#6452 from mengxr/SPARK-7198 and squashes the following commits: a9d2469 [Xiangrui Meng] add space facdb1f [Xiangrui Meng] VectorAssembler should output ML attributes
yogeshg
reviewed
Mar 19, 2018
| } else { | ||
| Some(attr.withName(c)) | ||
| } | ||
| case _: NumericType | BooleanType => |
There was a problem hiding this comment.
I had a quick question regarding this, why do we not consider a 3.2 case where we have Scalar type with ML attributes? is it because there's no such thing?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
VectorAssemblershould carry over ML attributes. For unknown attributes, we assume numeric values. This PR handles the following cases:@jkbradley