Improve resampling/evaluation; add measures API (incl. LossFunctions) #206
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
evaluate!
to report per-observation measures when available (for later use by Bayesian optimisers, for example)The new traits are defined at the top of /src/measures.jl
Improvements to resampling:
evaluate!
method now reports per-observation measures when availableevaluate!
for use by measures that support weightsevaluate!
, in place of aResamplingStrategy
objectResamplingStrategies
now straightforwardevaluate
(no exclamation mark) directly on model + data without first constructing a machine, if desiredSubstantial updates to the documentation to cover the changes. See, in particular new sections "Evaluating model performance" and "Measures".
Commits before rebase:
add evaluate! for user-specified resampling; needs integration with tuning
refactor cv to be special case of general evaluate!
integrate evaluation changes with tuning
update docs
add support for weighted measures in evaluuate!; tuning needs tests
update cheatsheet
add show method for Measure objects
Fix weights in resampler/tuning and add tests
Implement an MLJ interface for LossFunctions pkg
reports_each_sample->reports_each_observation per_sample->per_observation
fix LossFunctions MarginLoss's to be probabilistic
Add pretty printing to evaluation!
update docs
update docs
add measure/model compatibility checks
make resampling interface more customizable
fix @test_logs
update docs
update docs
update doc string
improve measure checks
update docs