Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved API Objectives #445

Merged
merged 79 commits into from Apr 13, 2020
Merged

Improved API Objectives #445

merged 79 commits into from Apr 13, 2020

Conversation

angela97lin
Copy link
Contributor

@angela97lin angela97lin commented Mar 4, 2020

Closes #346

angela97lin added 4 commits Mar 2, 2020
…emove objectives from pipeline classes (#405)

* creating new binary / multiclass variants of pipelines, duplicating code for now

* moving common fxns back to pipeline base

* more cleanup, making predict_proba standard regardless of binary/multiclass

* renaming other_objectives to objectives

* updating score's objective parameter to calculate all objectives, not just additional

* removing self.objective for scoring

* removing objectives from pipeline initialization, adding objective as predict param

* changelog

* adding classification pipeline subclass, cleaning up via PR comments

* more cleanup for docstrings

* putting tests in subfolders and adding few more tests

* more cleanup, add new ObjectiveNotFound exception

* remove duplicates

* add optional param for predict in binary pipelines
* removing ROC from auto_base, need to clean up and readd functionality

* circleci

* creating (maybe temporary) plot metrics class

* moving ROC / cm into plotmetrics class and cleanup

* fixing copy and paste error

* cleanup

* adding abc

* fixing abc import

* updating plot data getter for binary

* making predict objective optional for binary for plotting

* cleaning up test
@codecov
Copy link

codecov bot commented Mar 4, 2020

Codecov Report

Merging #445 into master will not change coverage by %.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #445   +/-   ##
=======================================
  Coverage   98.95%   98.95%           
=======================================
  Files         132      132           
  Lines        4504     4504           
=======================================
  Hits         4457     4457           
  Misses         47       47           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a88e69c...a88e69c. Read the comment docs.

@angela97lin angela97lin changed the base branch from master to 373_objective_classes Mar 11, 2020
@angela97lin angela97lin changed the base branch from 373_objective_classes to master Mar 11, 2020
@dsherry
Copy link
Collaborator

dsherry commented Apr 2, 2020

@angela97lin what's the status of this? You think we can get it merged by the end of the week?

Update: oops, commented in the wrong place, fixing

angela97lin and others added 5 commits Apr 3, 2020
* creating new binary / multiclass variants of pipelines, duplicating code for now

* moving common fxns back to pipeline base

* more moving around fxns in pipelines classes

* capping xgboost

* fixing typo

* more cleanup, making predict_proba standard regardless of binary/multiclass

* renaming other_objectives to objectives

* updating score's objective parameter to calculate all objectives, not just additional

* removing self.objective for scoring

* removing objectives from pipeline initialization, adding objective as predict param

* remove xgboost cap from branch

* changelog

* capping xgboost on local branch since tests timing out

* cleaning up

* more cleanup

* reverting requirements file

* adding classification pipeline subclass, cleaning up via PR comments

* more cleanup for docstrings

* more cleanup of changelog and comments

* putting tests in subfolders and adding few more tests

* Update dependencies  (#412)

* Update latest dependencies

* Hide features with zero importance in plot by default (#413)

* adding functionality and test

* changelog and adding boolean param

* Update dependencies check: package whitelist (#417)

* Add a whitelist for update_deps check

* Remove from expected

* Update deps

* Changelog

* adding skeleton for subclasses

* fixing test and linting

* updating change from master

* fixing fixture

* cherry picked wip remove ROC and confusion matrix

* fixing merge

* fixing merge

* cleaning up

* make test use static attribute instead of instance'

* deleting needs_fitting

* updating code to use new objective classes, still broken

* updating threshold, still need to clean up tests

* comment out for now

* more cleanup

* cleaning up

* more cleanup

* still more cleanup

* fixing plot unsuccessful merging

* more cleanup but still some things to work out

* cleaning up using multiclass objectives for binary classification problems

* fixing typo with recall and cleanup

* cleaning up

* adding default

* some more cleaning up

* removing irrelevant test

* forgot to add attribute, breaking things again

* cleanup and change objective of test

* removing objective from predict

* more cleanup :d

* remove unused attribute

* cleaning up via comments

* more comments

* changelog

* order of decorators changed

* fixing copy and paste err

* update random state for binary class pipelines

* updating objective

* typo

* fixing?

* fixing imports

* fixing tests

* adding objective as parameter for predict, removing for fit

* cleaning up test

* more fixing test

* minor linting, need more to go

* more cleanup

* forgot to fix test

* more merging :x

* starting to add tuning logic to automl

* changelog

* cleaning up

* change conditional for objective split

* cleaning up docstrings

* forgot to use classificationobjective class...

* add additional cond

* adding tests

* cleanup

* removing decision function for multiclass

* updating via comments

* removing classification_objective file

* add test + more updates

* use cls instead for pep8 standards

* updating can_optimize to property

* update score

* fix tests

* minor cleanup from comments

* updating predict behavior

* add separate objective check

* fixing some merge conflicts cont

* add fraud test

* patching

* remove old test

* updating for now

* add another test

* add more tests

* adding test structure, still need to fix

* adding test

* fix iloc

* fix tests

* fix import

* fix test?

* removing can_optimize_threshold

* linting

* update docs a little

* remove accuracy

* add more doc fixes

* move binary and multi pipelines in api ref

* revert components notebook

* updating from comments

* oops, fix none set

* update docstring

* update api ref?

* addressing comments

* revert and update

* update docstring

* updating docstrings and lint

* updating unnecessary call to constructor

* pushing empty commit to refresh

Co-authored-by: Jeremy Shih <jeremyliweishih@gmail.com>
Co-authored-by: Dylan Sherry <sharshofski@gmail.com>
@dsherry dsherry mentioned this pull request Apr 9, 2020
…#605)

* doc changes

* remove increase timeout

* fixing changelog which had been reverted to master by accident

* fixing changelog

* fix changelog
angela97lin and others added 6 commits Apr 10, 2020
…default disabled (#606)

* In automl, only optimize binary classification thresholds if boolean param is set. Default false.

* Update unit test

* Docs update

* Add unit test coverage of binary classification objective

* Add one more unit test
* removing plot

* changelog and docs

* fix changelog pr num

* removing plotting entirely

* adding tests for codecov

* fix rtd warnings
@angela97lin angela97lin changed the title [WIP] Improved API Objectives Improved API Objectives Apr 12, 2020
@angela97lin angela97lin requested a review from dsherry Apr 13, 2020
Copy link
Collaborator

@dsherry dsherry left a comment

PRs which were merged to this branch:

  • #405 binary/multiclass pipelines
  • #422 have ROC and confusion matrix not be objectives
  • #429 have score return a single dict
  • #504 bin/multiclass objectives
  • #605 fix docs
  • #606 (Dylan) disable threshold tuning by default
  • #615 don't precompute ROC and confusion matrix in automl
  • Merge commits with conflict resolutions.
  • I see some direct commits too.

RTD build looks good.

This is a lot of work! Very excited to get this onto master :)

We filed #610 to track putting ROC/confusion support back in place. There's a number of other tickets related to this. Looks good. I'll do some organizing shortly.

@angela97lin angela97lin merged commit f1cd43a into master Apr 13, 2020
2 checks passed
@angela97lin angela97lin deleted the improved_objectives branch Apr 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Objective API
2 participants