[FLINK-1723] [ml] [WIP] Add cross validation for model evaluation #891
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…instead of functions. Not too happy with the extra biolerplate of Score as classes will probably revert, and have objects like RegressionsScores, ClassificationScores that contain the definitions of the relevant scores.
All predictors must now implement a calculateScore function. We are for now assuming that predictors are supervised learning algorithms, once unsupervised learning algorithms are added this will need to be reworked. Also added an evaluate dataset operation to ALS, to allow for scoring of the algorithm. Default performance measure for ALS is RMSE.
| .setStepsize(10.0) | ||
| .setIterations(100) | ||
|
|
||
| println() |
There was a problem hiding this comment.
It prints a line between the consecutive test runs, I just have it there so I can more easily see what is happening. The tests don't really do anything yet, just print results.
|
No news? |
|
Closing since flink-ml is effectively frozen. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cross validation (CV) [1] is a standard tool to estimate the test error for a model. As such it is a crucial tool for every machine learning library.
This builds upon the ongoing work on the evaluation framework for FlinkML.
As such, the current version supports calculating the score of Predictors only, however the end goal is to be able to have CV for Estimators as well to cover the unsupervised learning case.
We are using some code from the Apache Spark project, mostly simple routines for probabilistic sampling of datasets and generation of KFold CV data.
More and better tests need to be added to the implementation, and the current sampling approaches probably will not work if used within an iteration.