[FLINK-2157] [ml] Create evaluation framework for ML library #1849

thvasilo · 2016-04-04T09:38:18Z

Using this PR instead of #871 due to rebase issues.

thvasilo · 2016-04-04T09:40:26Z

@mbalassi @tillrohrmann

Closed the previous PR and opened this one for the evaluation framework, as I had some issues with rebasing.

rawkintrevo · 2016-04-20T19:14:27Z

Are there going to be useage docs on this?

rawkintrevo · 2016-04-20T22:16:03Z

Also two quick issues.

pipelines

val scaler = MinMaxScaler()
val pipeline = scaler.chainPredictor(mlr)
val evaluationDS = survivalLV.map(x => (x.vector, x.label))

pipeline.fit(survivalLV)
scorer.evaluate(evaluationDS, pipeline).collect().head

When using this with a ChainedPredictor as the predictor I get the following error:
error: could not find implicit value for parameter evaluateOperation: org.apache.flink.ml.pipeline.EvaluateDataSetOperation[org.apache.flink.ml.pipeline.ChainedPredictor[org.apache.flink.ml.preprocessing.MinMaxScaler,org.apache.flink.ml.regression.MultipleLinearRegression],(org.apache.flink.ml.math.Vector, Double),Double]

MinMaxScaler()
Merging for me broke the following code:

val scaler = MinMaxScaler()
val scaledSurvivalLV = scaler.transform(survivalLV)

With the following error (omiting part of the stack trace)
Caused by: java.lang.NoSuchMethodError: breeze.linalg.Vector$.scalarOf()Lbreeze/linalg/support/ScalarOf;
at org.apache.flink.ml.preprocessing.MinMaxScaler$$anonfun$3.apply(MinMaxScaler.scala:156)
at org.apache.flink.ml.preprocessing.MinMaxScaler$$anonfun$3.apply(MinMaxScaler.scala:154)
at org.apache.flink.api.scala.DataSet$$anon$7.reduce(DataSet.scala:584)
at org.apache.flink.runtime.operators.chaining.ChainedAllReduceDriver.collect(ChainedAllReduceDriver.java:93)
at org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:97)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:345)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
at java.lang.Thread.run(Thread.java:745)

I'm looking for a work around. Just saying I found a regression. Other than that, looks/works AWESOME well done.

thvasilo · 2016-04-21T07:59:00Z

Hello Trevor,

Thanks for taking the time to look at this, I'll investigate these issues
today hopefully.

Sent from a mobile device. May contain autocorrect errors.
On Apr 21, 2016 12:16 AM, "Trevor Grant" notifications@github.com wrote:

Also two quick issues.

pipelines

val scaler = MinMaxScaler()val pipeline = scaler.chainPredictor(mlr)val evaluationDS = survivalLV.map(x => (x.vector, x.label))

pipeline.fit(survivalLV)
scorer.evaluate(evaluationDS, pipeline).collect().head

When using this with a ChainedPredictor as the predictor I get the
following error:
error: could not find implicit value for parameter evaluateOperation:
org.apache.flink.ml.pipeline.EvaluateDataSetOperation[org.apache.flink.ml.pipeline.ChainedPredictor[org.apache.flink.ml.preprocessing.MinMaxScaler,org.apache.flink.ml.regression.MultipleLinearRegression],(org.apache.flink.ml.math.Vector,
Double),Double]

MinMaxScaler()
Merging for me broke the following code:

val scaler = MinMaxScaler()val scaledSurvivalLV = scaler.transform(survivalLV)

With the following error (omiting part of the stack trace)
Caused by: java.lang.NoSuchMethodError:
breeze.linalg.Vector$.scalarOf()Lbreeze/linalg/support/ScalarOf;
at
org.apache.flink.ml.preprocessing.MinMaxScaler$$anonfun$3.apply(MinMaxScaler.scala:156)
at
org.apache.flink.ml.preprocessing.MinMaxScaler$$anonfun$3.apply(MinMaxScaler.scala:154)
at org.apache.flink.api.scala.DataSet$$anon$7.reduce(DataSet.scala:584)
at
org.apache.flink.runtime.operators.chaining.ChainedAllReduceDriver.collect(ChainedAllReduceDriver.java:93)
at
org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:78)
at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:97)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:345)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
at java.lang.Thread.run(Thread.java:745)

I'm looking for a work around. Just saying I found a regression. Other
than that, looks/works AWESOME well done.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#1849 (comment)

rawkintrevo · 2016-04-21T13:18:58Z

np, also RE: my comment on the docs- I think I can lend a hand there (I was actually testing functionality to make sure I understood how it worked). Let me know if I can be of assistance.

Also, I did some more hacking this morning...

import org.apache.flink.api.scala._

import org.apache.flink.ml.preprocessing.StandardScaler
val scaler = StandardScaler()//MinMaxScaler()

import org.apache.flink.ml.evaluation.{RegressionScores, Scorer}
val loss = RegressionScores.squaredLoss
val scorer = new Scorer(loss)

import org.apache.flink.ml.regression.MultipleLinearRegression
val mlr = MultipleLinearRegression()
                            .setIterations(10)
                            .setConvergenceThreshold(0.001)

val pipeline = scaler.chainPredictor(mlr)
val evaluationDS = survivalLV.map(x => (x.vector, x.label))

pipeline.fit(survivalLV)
//pipeline.evaluate(survivalLV).collect()
scorer.evaluate(evaluationDS, pipeline).collect().head

This throws the breeze.linalg... error. So I'm not sure exactly what is different, but it would seem the breeze.linalg is close to the heart of the problem(?) E.g. it is trying to use the pipeline, but still gets gigged in the scaler.

thvasilo · 2016-04-21T14:27:01Z

~~Well breeze was recently bumped to 0.12 #1876, maybe that has something to do with it, but let's see.~~

~~Any chance you can try with the prev. Breeze version?~~

Irrelevant as Breeze version still 0.11 in this branch.

thvasilo · 2016-04-21T15:38:32Z

I did some testing and I think the problem has to do with the types that each scaler expects.

StandardScaler has fit and transform operations for DataSets of type Vector, LabeledVector, and (T :< Vector, Double) while MinMaxScaler does not provide one for (T :< Vector, Double). If you add the operations the code runs fine (at least re. you first comment).

So this is a bug unrelated to this PR I think. The question becomes if we want to support all three of these types. My recommendation would be to have support for Vector and LabeledVector only, and remove all operations that work on (Vector, Double) tuples. I will file a JIRA for that.

There is an argument to be whether some pre-processing steps are supervised (e.g. PCA vs. LDA) but in the strict definition of a transformer we shouldn't care about the label, only the features, so that operation can implemented at the Transformer level.

rawkintrevo · 2016-04-22T12:41:35Z

The transformer needs to scale the label too... I might not be correctly understanding your last paragraph / what you are proposing.

I agree with paragraph 1-3.

gaborhermann · 2016-10-04T10:54:23Z

Hi all,

What is the status of this PR?
It would be relevant for us, because we might like to use the evaluation framework proposed here. See FLINK-4713 for details.

Can I do anything to help resolving the issues you've been discussing here?

thvasilo · 2016-10-04T14:01:21Z

@gaborhermann In terms of missing features, documentation is definitely missing, as @rawkintrevo mentioned.

For the issues mentioned in the JIRA issue you linked I've replied on the dev list thread you started, all valid points re. adjusting this to handle recommendations.

skonto · 2017-01-17T12:13:41Z

Hey @thvasilo is this under development? From what I see many other tasks depend on it right?

skonto · 2017-01-17T12:46:34Z

flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/evaluation/Score.scala

+ *
+ * @tparam PredictionType output type
+ */
+trait Score[PredictionType] {


What is the benefit of having the scores independent of the models. For example each model could implement it's own score function within its implementation class. I may miss something here...

The goal is to reduce code duplication, many models can share the same evaluation infrastructure.

thvasilo · 2017-01-17T13:05:45Z

Hello @skonto this PR will probably be subsumed by #2838, you can check out the latest development there.

skonto · 2017-01-20T12:28:02Z

@thvasilo thnx I will have a look

gaborhermann · 2017-01-20T13:36:33Z

Hi @skonto, I did not have time lately to finish up #2838, but I could clean it up next week. Although I believe this PR could be merged separately from mine. (Evaluating ranking recommendations is a bit more complicated.) As @thvasilo mentioned, the documentation is missing in his PR, but most of the work is already in place here. I could easily rebase my PR on top of this, if you don't modify much in the structure of classes. @thvasilo what do you think?

thvasilo · 2017-01-20T13:50:07Z

Hello @gaborhermann. Personally I prefer to have PRs be as specific as possible, so I would recommend we try to get this merged before #2838, and then rebase that on master.

Given the committer load however this could take a while.

skonto · 2017-01-20T14:06:21Z

Hi guys, my intention was to review #2838 but my feeling is that it overlaps with this one. @thvasilo we can push this one first as you said so I will have a look at it and comment on it. The benefit is unblocking other tasks in this area which rely on the framework.

gaborhermann · 2017-01-20T14:45:09Z

Great. I agree this PR should be merged before #2838.
@skonto thanks for taking up the review :) This is indeed a bit blocking.
Hopefully I can improve upon #2838 next week, so by the time you get there, the PR could be ready (i.e. not in a WIP state).

zentol · 2019-02-28T22:58:10Z

Closing since flink-ml is effectively frozen.

[FLINK-2157] [ml] Create evaluation framework for ML library

8ce81d5

thvasilo mentioned this pull request Apr 4, 2016

[FLINK-2157] [ml] Create evaluation framework for ML library #871

Closed

gaborhermann mentioned this pull request Nov 21, 2016

[FLINK-4712] [FLINK-4713] [ml] Ranking recommendation & evaluation (WIP) #2838

Closed

skonto reviewed Jan 17, 2017

View reviewed changes

zentol closed this Feb 28, 2019

rmetzger added the component=Library/MachineLearning label Mar 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-2157] [ml] Create evaluation framework for ML library #1849

[FLINK-2157] [ml] Create evaluation framework for ML library #1849

thvasilo commented Apr 4, 2016

thvasilo commented Apr 4, 2016

rawkintrevo commented Apr 20, 2016

rawkintrevo commented Apr 20, 2016

thvasilo commented Apr 21, 2016

rawkintrevo commented Apr 21, 2016 •

edited

thvasilo commented Apr 21, 2016 •

edited

thvasilo commented Apr 21, 2016 •

edited

rawkintrevo commented Apr 22, 2016

gaborhermann commented Oct 4, 2016

thvasilo commented Oct 4, 2016

skonto commented Jan 17, 2017

skonto Jan 17, 2017

thvasilo Jan 17, 2017

thvasilo commented Jan 17, 2017

skonto commented Jan 20, 2017

gaborhermann commented Jan 20, 2017

thvasilo commented Jan 20, 2017

skonto commented Jan 20, 2017 •

edited

gaborhermann commented Jan 20, 2017

zentol commented Feb 28, 2019

[FLINK-2157] [ml] Create evaluation framework for ML library #1849

[FLINK-2157] [ml] Create evaluation framework for ML library #1849

Conversation

thvasilo commented Apr 4, 2016

thvasilo commented Apr 4, 2016

rawkintrevo commented Apr 20, 2016

rawkintrevo commented Apr 20, 2016

thvasilo commented Apr 21, 2016

rawkintrevo commented Apr 21, 2016 • edited

thvasilo commented Apr 21, 2016 • edited

thvasilo commented Apr 21, 2016 • edited

rawkintrevo commented Apr 22, 2016

gaborhermann commented Oct 4, 2016

thvasilo commented Oct 4, 2016

skonto commented Jan 17, 2017

skonto Jan 17, 2017

Choose a reason for hiding this comment

thvasilo Jan 17, 2017

Choose a reason for hiding this comment

thvasilo commented Jan 17, 2017

skonto commented Jan 20, 2017

gaborhermann commented Jan 20, 2017

thvasilo commented Jan 20, 2017

skonto commented Jan 20, 2017 • edited

gaborhermann commented Jan 20, 2017

zentol commented Feb 28, 2019

rawkintrevo commented Apr 21, 2016 •

edited

thvasilo commented Apr 21, 2016 •

edited

thvasilo commented Apr 21, 2016 •

edited

skonto commented Jan 20, 2017 •

edited