Add custom component docs #993

jeremyliweishih · 2020-07-29T14:42:28Z

Fixes #964.
View here

Add bit on hyperparameters
Add class reference component_graph to pipelines page and talk about custom components
Add info on automl about allowed_pipelines

codecov · 2020-07-29T14:51:26Z

Codecov Report

Merging #993 into main will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #993   +/-   ##
=======================================
  Coverage   99.86%   99.86%           
=======================================
  Files         181      181           
  Lines        9584     9584           
=======================================
  Hits         9571     9571           
  Misses         13       13

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a92cf5...a00a7ae. Read the comment docs.

…964_cc

…_964_cc

angela97lin

Looking good! Left some feedback for rewording, mostly non-blocking and stylistic :d

docs/source/user_guide.ipynb

docs/source/user_guide/automl.ipynb

docs/source/user_guide/components.ipynb

angela97lin · 2020-07-30T15:15:50Z

docs/source/user_guide/components.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "However, if your component does not require a `component_obj`, you will need to override all methods you intend on using. This can be seen in [DropNullColumns](../generated/evalml.pipelines.components.DropNullColumns.ipynb)."


Yeah, I feel like this could be emphasized more / the idea that you could provide component_obj to provide or not.

I wrote it this way because I wanted to emphasize that EvalML's implementation works off of component_obj you need to override everything. I'll try to make it more clear in the text above.

docs/source/user_guide/components.ipynb

angela97lin · 2020-07-30T15:21:40Z

docs/source/user_guide/components.ipynb

+    "\n",
+    "#### Estimators\n",
+    "- target expected as int from 0 to n-1\n",
+    "- `predict_proba` column order must match the class integers and column names are set to integer values\n",


Hmmm, I'm not quiteeee sure what this is supposed to say 🤔

predict_proba column order for custom estimators must match the order of classes...?

What do you mean by column names are set to integer values :o

@angela97lin I took these assumptions from here. I tried to follow the PR and from my understanding it was just that targets must be of type int from 0 to n-1, column order(?) from predict_proba must match the class integers (I guess predict_proba must return columns like [0, 1, 2]), and column names in predict_proba must be of type int as well. I'll reword what I have but let me know if my understanding is correct.

I think the link is broken :thonk: I'm guessing this was my str target PR but let me know and I'll take a look? :))

Sorry its this

This reverts commit f111cfc.

eccabay · 2020-07-30T20:04:54Z

docs/source/user_guide/components.ipynb

+   "metadata": {},
+   "source": [
+    "### Implementation\n",
+    "EvalML components by default work off of a `component_object` that provides the implementation for methods such as `fit()`, `predict()`, and `transform()`. This can be seen in [ComponentBase](../generated/evalml.pipelines.components.ComponentBase.ipynb) where the base class calls the `fit()` method of the `component_obj`. This applies to both transformers and estimators. You can provide this `component_obj` in the `__init__()` method of your new component."


This could be confusing for newer users - it may be worth it to clarify here that these component_objects are third party classes that implement the necessary methods, since that's not implied here.

It might also be helpful to move the section about implementing a component without a component object up here, to make it clear that it's possible to implement a new component without a third party tool.

Good point on mentioning the fact that the component_obj is a third-party class, and that we provide that interface to make it easy to use third-party code.

I agree with @eccabay's suggestion to shuffle the section order a bit. I have a suggestion, will leave in another comment.

dsherry · 2020-07-30T20:21:59Z

docs/source/user_guide/components.ipynb

+    "    pass\n",
+    "\n",
+    "class NewEstimator(Estimator):\n",
+    "    pass"


I suggest you delete this example. People won't want to define empty components. Also won't this error? If it doesn't today it should eventually haha

dsherry

@jeremyliweishih nice going on this! This is going to make it a lot easier for people to understand how to do custom definitions. This has been a big pain point recently. I expect this'll save us some bug reports. Also I love that you've added the API ref / user guide links!

I think all the content is here, and now its just a question of reorganizing and deleting for the highest impact. What do you think of this (I'm skipping a bunch of details which you already have writing for, but the top-level organization is what I'm really getting at):

Defining Custom Components

Custom Transformers
- Transformer example, with init/fit/transform
- Talk about required fields, name. Required methods, init fit transform. Mention fit_transform as an optional optimization.
Custom Estimators
- Estimator example, with init/fit/predict
- Same as estimator, except predict instead of transform
- Mention classification, if estimator supports that it needs a predict_proba
Components Wrapping Third-party Objects
- Example of a transformer or estimator which uses component_obj. Just need to define init, unless third-party naming doesn't match our expectations.
Hyperparameter Ranges for AutoML
- Transformers and estimators can have hyperparameter ranges, not required by default. Example of how to set that for an estimator.
Expectations for Custom Classification Components
- Target format, predict_proba for estimators

With the code examples, let's try to include fewer, but where the example could be pasted into a terminal and used without errors. Hopefully we can add code which raises an error when you try to define a component incorrectly, but until then we need our examples to be spotless.

docs/source/user_guide/components.ipynb

dsherry · 2020-07-31T05:21:32Z

docs/source/user_guide/components.ipynb

+   "metadata": {},
+   "source": [
+    "### Implementation\n",
+    "EvalML components by default work off of a `component_object` that provides the implementation for methods such as `fit()`, `predict()`, and `transform()`. This can be seen in [ComponentBase](../generated/evalml.pipelines.components.ComponentBase.ipynb) where the base class calls the `fit()` method of the `component_obj`. This applies to both transformers and estimators. You can provide this `component_obj` in the `__init__()` method of your new component."


Good point on mentioning the fact that the component_obj is a third-party class, and that we provide that interface to make it easy to use third-party code.

I agree with @eccabay's suggestion to shuffle the section order a bit. I have a suggestion, will leave in another comment.

docs/source/user_guide/components.ipynb

docs/source/user_guide/pipelines.ipynb

docs/source/user_guide/automl.ipynb

…964_cc

jeremyliweishih · 2020-07-31T15:54:27Z

@dsherry FYI latest ver hasn't build on RTD yet - we're hitting the concurrency limit with all the jobs 😄

dsherry

@jeremyliweishih great!!! Thanks for updating this. I'm glad we're getting this in!

dsherry · 2020-07-31T19:24:33Z

docs/source/user_guide/components.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Components Wrapping Third-party Objects\n",


Third-Party

…964_cc

stale

jeremyliweishih added 3 commits July 29, 2020 10:41

Add first draft

2c7c373

add RL

f048079

Remove output

0d890d9

jeremyliweishih added 18 commits July 29, 2020 10:56

Move to one notebook

d388f55

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

e083d18

…964_cc

Remove cc from user_guide

ac13f03

remove faulty link

5b11071

fix links

8f2f38f

Fix more links'

31067fc

fix links for real

7d5d211

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

d121d03

…964_cc

fix links again..

38c8c21

mMerge branch 'main' of https://github.com/FeatureLabs/evalml into js…

72135a0

…_964_cc

Add hyperparameter ranges

0c05ae1

Add custom components in pipelines

97e0780

Add bit on automl

2effd3a

fix name

2e1630a

fix docs

bec26c4

Move header

92af156

Change header to Components'

8fc85da

fix link

378e0d2

jeremyliweishih marked this pull request as ready for review July 29, 2020 21:43

auto-assign bot assigned jeremyliweishih Jul 29, 2020

jeremyliweishih requested review from dsherry, freddyaboulton, angela97lin and eccabay and removed request for dsherry July 29, 2020 22:01

angela97lin previously requested changes Jul 30, 2020

View reviewed changes

jeremyliweishih added 4 commits July 30, 2020 14:07

revert python ver

f111cfc

Revert "revert python ver"

5ca15df

This reverts commit f111cfc.

revert python ver

38d199c

Address Angela's comments'

172b930

jeremyliweishih requested a review from angela97lin July 30, 2020 19:11

eccabay reviewed Jul 30, 2020

View reviewed changes

dsherry reviewed Jul 30, 2020

View reviewed changes

dsherry suggested changes Jul 31, 2020

View reviewed changes

jeremyliweishih added 8 commits July 31, 2020 10:03

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

d851884

…964_cc

Little changes

fdc22fe

Reorg

4ba0ec6

Change headers

66a8e9d

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

f357dd1

…964_cc

Change layout a little

98d8c3a

shift

fec1f16

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

49ab82d

…964_cc

jeremyliweishih requested review from eccabay and dsherry July 31, 2020 15:51

dsherry approved these changes Jul 31, 2020

View reviewed changes

jeremyliweishih added 3 commits July 31, 2020 15:26

Merge branch 'main' of https://github.com/FeatureLabs/evalml into js_…

cacfaf9

…964_cc

Capitilization

113b4fc

Merge branch 'main' into js_964_cc

a00a7ae

jeremyliweishih merged commit d6cc78f into main Jul 31, 2020

angela97lin mentioned this pull request Jul 31, 2020

Release v0.12.0 #1007

Merged

freddyaboulton deleted the js_964_cc branch May 13, 2022 14:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add custom component docs #993

Add custom component docs #993

jeremyliweishih commented Jul 29, 2020 •

edited

Loading

codecov bot commented Jul 29, 2020 •

edited

Loading

angela97lin left a comment

angela97lin Jul 30, 2020

jeremyliweishih Jul 30, 2020

angela97lin Jul 30, 2020

jeremyliweishih Jul 30, 2020

angela97lin Jul 30, 2020

jeremyliweishih Jul 30, 2020

eccabay Jul 30, 2020

dsherry Jul 31, 2020

dsherry Jul 30, 2020

dsherry left a comment

dsherry Jul 31, 2020

jeremyliweishih commented Jul 31, 2020

dsherry left a comment

dsherry Jul 31, 2020

Add custom component docs #993

Add custom component docs #993

Conversation

jeremyliweishih commented Jul 29, 2020 • edited Loading

codecov bot commented Jul 29, 2020 • edited Loading

Codecov Report

angela97lin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyliweishih commented Jul 31, 2020

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyliweishih commented Jul 29, 2020 •

edited

Loading

codecov bot commented Jul 29, 2020 •

edited

Loading