Add Decoupled CV to master (#240) #271
Conversation
Codecov Report
@@ Coverage Diff @@
## master #271 +/- ##
==========================================
+ Coverage 82.34% 82.58% +0.24%
==========================================
Files 136 138 +2
Lines 4995 5059 +64
==========================================
+ Hits 4113 4178 +65
+ Misses 882 881 -1
Continue to review full report at Codecov.
|
|
I think the first 2 commits (the one different to Decoupled CV) are already on master and not needed in this PR. I don't think they appear in the diff so maybe it's not a big deal. I also think that we could use this opportunity to remove the co-authoring of the decoupled CV? @kegl used 2 usernames. |
yes they don't appear in the changes. |
* passing fold to workflow.test_submission * passing fold indices to Prediction init instead of indexing in util.submission.run_submission_on_cv_fold * getting rid of passing valid_indexes to scores * getting rid of valid_indexes in score_type.score_function * flake * functional data flow in doc * training script in doc * docstring minor * Add test protocol to README * Update README.rst
Thanks for checking @albertcthomas ! I removed it and the 2 extra commits. |
rampwf/score_types/README.md
Outdated
|
|
||
| It is called in `ramp_test_submission` and at the [ramp-board][rboard] frontend when evaluating user submissions. Most of the time it uses the default implementation in `BaseScoreType` that optionally selects the cv fold and checks if the number of rows are identical in `y_true` and `y_pred`, then calls `__call__(self, y_true, y_pred)`. `ClassifierBaseScoreType` does the same except it calls `__call__` with `y_true_label_index` and `y_pred_label_index`. When __call__ raises a non implemented error, `score_function` has to be overridden. | ||
|
|
||
| It is called in `ramp_test_submission` and at the [ramp-board][rboard] frontend when evaluating user submissions. Most of the time it uses the default implementation in `BaseScoreType` that checks if the number of rows are identical in `y_true` and `y_pred`, then calls `__call__(self, y_true, y_pred)`. `ClassifierBaseScoreType` does the same except it calls `__call__` with `y_true_label_index` and `y_pred_label_index`. When __call__ raises a non implemented error, `score_function` has to be overridden. |
There was a problem hiding this comment.
we should maybe use ramp-test instead of ramp_test_submission
| trained_workflow = problem.workflow.train_submission( | ||
| 'submissions/starting_kit', X_train, y_train) | ||
| y_pred_test = problem.workflow.test_submission( | ||
| trained_workflow, X_test) |
There was a problem hiding this comment.
It's nice to have this documentation! I think few users are aware that it's possible.
There was a problem hiding this comment.
OK, I have changed slightly the procedure for testing with ramp-board. Now one can go to ramp-board github actions and copy paste a pip installable URL of ramp-workflow to run CI with,
It's still a bit manual and it requires write permissions to ramp-board to trigger CI, but at least it doesn't require to push commits to ramp-board on a separate branch and keep that branch in sync with master as the previous procedure did.
Code wise this LGTM. I can also confirm this passes ramp-board CI on master
This is a backport of #243 to master.
I'll mention paris-saclay-cds/ramp-board#515 in the test protocol documentation which should allow running ramp-board CI on arbitrary ramp-workflow PRs using manual triggers