Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve resampling/evaluation; add measures API (incl. LossFunctions) #206

Merged
merged 2 commits into from
Aug 14, 2019

Conversation

ablaom
Copy link
Member

@ablaom ablaom commented Aug 13, 2019

  • Introduction of traits for measures (loss functions, etc):
    • allowing user to use loss functions from LossFunctions.jl,
    • enabling improved measure checks and error message reporting with measures
    • allowing evaluate! to report per-observation measures when available (for later use by Bayesian optimisers, for example)
    • allowing support for sample-weighted measures playing nicely with rest of API

The new traits are defined at the top of /src/measures.jl

  • Improvements to resampling:

    • evaluate! method now reports per-observation measures when available
    • sample weights can be passed to evaluate! for use by measures that support weights
    • user can pass a list of train/evaluation pairs of row indices directly to evaluate!, in place of a ResamplingStrategy object
    • implementing new ResamplingStrategies now straightforward
    • one can call evaluate (no exclamation mark) directly on model + data without first constructing a machine, if desired
  • Substantial updates to the documentation to cover the changes. See, in particular new sections "Evaluating model performance" and "Measures".

Commits before rebase:

add evaluate! for user-specified resampling; needs integration with tuning

refactor cv to be special case of general evaluate!

integrate evaluation changes with tuning

update docs

add support for weighted measures in evaluuate!; tuning needs tests

update cheatsheet

add show method for Measure objects

Fix weights in resampler/tuning and add tests

Implement an MLJ interface for LossFunctions pkg

reports_each_sample->reports_each_observation per_sample->per_observation

fix LossFunctions MarginLoss's to be probabilistic

Add pretty printing to evaluation!

update docs

update docs

add measure/model compatibility checks

make resampling interface more customizable

fix @test_logs

update docs

update docs

update doc string

improve measure checks

update docs

add evaluate! for user-specified resampling; needs integration with tuning

refactor cv to be special case of general evaluate!

integrate evaluation changes with tuning

update docs

add support for weighted measures in evaluuate!; tuning needs tests

update cheatsheet

add show method for Measure objects

Fix weights in resampler/tuning and add tests

Implement an MLJ interface for LossFunctions pkg

reports_each_sample->reports_each_observation per_sample->per_observation

fix LossFunctions MarginLoss's to be probabilistic

Add pretty printing to evaluation!

update docs

update docs

add measure/model compatibility checks

make resampling interface more customizable

fix @test_logs

update docs

update docs

update doc string

improve measure checks

update docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant