Update and overhaul documentation #937

dsherry · 2020-07-16T05:20:43Z

Fix #861, also: fix #713 fix #630

View the docs changes here.

Changes:

Adds new sphinx style from pandas
Updates docs outline: add Install and Start guides as top-level items, add Tutorials section to grow over time, User Guide to hold examples of our key APIs and features, Development to discuss how to contribute.
Update user guide content: added a Model Understanding section. Reworked the automl/objectives/components/pipelines sections
Rename changelog to release notes and update circleci job (will cause breaking changes)

What's missing:

Clean up the home page, make sure we're on board with the wording
Update / clean up the "Start" section to make it as high-impact as possible
Many more tutorials! Examples of using featuretools.
Clean up API ref style under new template. Release names aren't bolded anymore.
There's plenty of automl features, data check features, etc. which I didn't have time to fit in this PR, but which we should add back in the appropriate place. Perhaps the User Guide section needs two toctree levels instead of just being one flat list.
I like the cards and icons on the pandas homepage. We should add something similar for a good first impression.
Consider making the toctrees hidden on most if not all pages.

codecov · 2020-07-16T05:21:14Z

Codecov Report

Merging #937 into main will increase coverage by 0.20%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #937      +/-   ##
==========================================
+ Coverage   99.65%   99.86%   +0.20%     
==========================================
  Files         170      170              
  Lines        8593     8593              
==========================================
+ Hits         8563     8581      +18     
+ Misses         30       12      -18

Impacted Files	Coverage Δ
.../automl_tests/test_automl_search_classification.py	`100.00% <0.00%> (+0.45%)`	⬆️
evalml/automl/automl_search.py	`99.26% <0.00%> (+0.48%)`	⬆️
evalml/tests/component_tests/test_components.py	`100.00% <0.00%> (+0.61%)`	⬆️
evalml/tests/pipeline_tests/test_pipelines.py	`100.00% <0.00%> (+0.92%)`	⬆️
...ests/automl_tests/test_automl_search_regression.py	`100.00% <0.00%> (+1.07%)`	⬆️
evalml/utils/gen_utils.py	`100.00% <0.00%> (+2.63%)`	⬆️
evalml/tests/component_tests/test_utils.py	`100.00% <0.00%> (+3.57%)`	⬆️
evalml/tests/utils_tests/test_dependencies.py	`100.00% <0.00%> (+12.50%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07d5bea...dd9e5d4. Read the comment docs.

dsherry · 2020-07-16T05:30:57Z

docs/source/install.ipynb

@@ -9,7 +9,7 @@
    "EvalML is available for Python 3.6+. It can be installed by running the following command:\n",
    "\n",
    "```bash\n",
-    "    pip install evaml --extra-index-url https://install.featurelabs.com/<license>/\n",
+    "pip install evaml\n",


side note: has our release documents/process been updated to reflect pushing to pip?

also should be evalml!

Not yet! Good point. I'd like to do that once we've done our first pip release. I'll file something for that now.

angela97lin · 2020-07-16T14:51:49Z

Looks so much better!! Excited to look through this but at a quick glance, I noticed the pagenation links are a little funky?

Minor, and assuming this happens because it's the last link in the previous section but a little confusing 🤔

ctduffy · 2020-07-16T15:28:31Z

I have noticed that at the bottom of a few of the pages it seems like there is an extra code cell, as seen here. Looks super clean overall though!!

ctduffy · 2020-07-16T15:31:04Z

Also, this is a more general comment not specifically related to this PR, but when we begin open sourcing the repo, are we considering adding a "Thanks to the following people for contributing to this release: ..." to the changelog similar to the featuretools changelog?

dsherry · 2020-07-16T15:42:12Z

@ctduffy that's a great idea! Yes, once we open up the repo to outside contributors, adding that sort of info to the release notes would be good. GitHub shows who's contributed and when so that information will already be available in another form at least.

jeremyliweishih

Overall looking awesome and I'm super excited for this! I left comments below that can be are either typo/grammatical errors or style comments. Feel free to ignore the style comments based on your own judgement.

There are a couple more issues:

favicon: we need to update the favicon to the new evalml logo (or at least the old version still shows up for me). If we need to fix this would be great to include in this PR.
inheritance graph: since we change the templating of our API reference, the inheritance graph looks a little short now. We should put up a new issue fixing this.

This is a lot of comments at once so let me know if you want to hop on a call to go through them!

jeremyliweishih · 2020-07-16T14:43:32Z

docs/source/install.ipynb

@@ -9,7 +9,7 @@
    "EvalML is available for Python 3.6+. It can be installed by running the following command:\n",
    "\n",
    "```bash\n",
-    "    pip install evaml --extra-index-url https://install.featurelabs.com/<license>/\n",
+    "pip install evaml\n",


also should be evalml!

docs/source/user_guide/automl.ipynb

jeremyliweishih · 2020-07-16T14:59:06Z

docs/source/user_guide/automl.ipynb

+    "\n",
+    "There are [a variety](https://en.wikipedia.org/wiki/Machine_learning#Approaches) of ML problem types. Supervised learning describes the case where the collected data contains an output quantity to be modeled and a set of inputs with which to train the model. EvalML focuses on training models for the supervised learning case.\n",
+    "\n",
+    "EvalML supports three common supervised ML problem types. The first is regression, where the target quantity to model is a continuous numeric value. Next are binary and multiclass classification, where the target quantity to model consists of two or more discrete values or categories. The choice of which supervised ML problem type is most appropriate depends on domain expertise and on how the model will be evaluated and used.\n",


Although implied I think it might be nice to explain binary = 2 classes and multiclass = 2+.

Not sure if this is helpful but I'm more used to reading supervised ML problem types as defined between regression and classification (then classification being defined between multiclass and binary). I like how it says 3 problem types as it coincides with evalml problem types though. The former might be easier to understand for a ML newbie but I dont think this comment is important to consider if the target audience is ML engineers or data scientists etc..

jeremyliweishih · 2020-07-16T15:31:53Z

docs/source/user_guide/components.ipynb

+    "\n",
+    "Estimator classes each define a `model_family` attribute indicating what type of model is used.\n",
+    "\n",
+    "Here's an example of using the [`LogisticRegressionClassifier`](../generated/evalml.pipelines.components.LogisticRegressionClassifier.ipynb) estimator to fit and predict on a simple dataset:"


This link is broken on the actual website.

Thanks. I think I fixed it. Needed to be ref to rst I think?

docs/source/user_guide/pipelines.ipynb

jeremyliweishih · 2020-07-16T15:40:30Z

docs/source/user_guide/model_understanding.ipynb

   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "## Get Pipeline\n",
-    "We can get the object of any pipeline via their `id` as well:"
+    "## Feature Importance\n",


We can add how we calculate such importances (maybe in a new issue).

jeremyliweishih · 2020-07-16T15:41:11Z

docs/source/user_guide/model_understanding.ipynb

@@ -137,7 +103,7 @@
   "source": [
    "## Precision-Recall Curve\n",


This particular dataset makes the curves not very useful. We could put up an issue to use another dataset for the below curves.

kmax12 · 2020-07-16T15:46:03Z

a page on how to get help would be good to add. something like: https://docs.featuretools.com/en/stable/help.html. not blocking though

ctduffy · 2020-07-16T16:31:22Z

docs/source/start.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "First, we load in the features and outcomes we want to use to train our model."


I think it might be helpful here to describe what form this might take-like specify if this should be two dataframes, two arrays or something else. I think this is a good addition, as it might not be specifically mentioned elsewhere in the documentation as X and y together.
Also might be useful to change the wording from outcomes to target? Not sure exactly, but I feel like target is more generally used to refer to the y value in ML?

dsherry · 2020-07-16T16:32:54Z

Latest changes now up on readthedocs

docs/source/start.ipynb

freddyaboulton

@dsherry The new theme looks great! These changes look good to me but the links to classes, such as LogisticRegressionClassifier are still broken for me (even after you changed the link to point to the .rst file). I think it'd be nice to fix that before merging.

I also left some comments that can be addressed in other issues. One thing I have in mind is making sure all classes wrapped in "``" should link to their source. My instinct is to expect a link and go to the source when a new class or concept is introduced.

freddyaboulton · 2020-07-16T16:19:37Z

docs/source/index.ipynb

-    "[What is EvalML](self)\n",
-    "\n",
-    "[Installation](install)"
+    "[Start](start)"


Not blocking: I prefer the old "Getting Started" title

Got it. Let's circle back on this. I liked having both install and start as top-level items, but could be persuaded otherwise

docs/source/start.ipynb

freddyaboulton · 2020-07-16T16:31:30Z

docs/source/user_guide/components.ipynb

+    "\n",
+    "Estimator classes each define a `model_family` attribute indicating what type of model is used.\n",
+    "\n",
+    "Here's an example of using the [`LogisticRegressionClassifier`](../generated/evalml.pipelines.components.LogisticRegressionClassifier.rst) estimator to fit and predict on a simple dataset:"


This link is still broken for me. Also, we should make all the classes we put in "``" links? Might be a little confusing that some class names are broken and some aren't.

🤦 dang, IDK why its broken. Will try one more push but probably going to punt

maybe needs to be html

freddyaboulton · 2020-07-16T16:32:32Z

docs/source/user_guide/data_checks.ipynb

@@ -20,7 +20,7 @@
   "source": [
    "## Missing Data\n",
    "\n",
-    "Missing data or rows with `NaN` values provide many challenges for machine learning pipelines. In the worst case, many algorithms simply will not run with missing data! EvalML pipelines contain imputation [components](../pipelines/components.ipynb) to ensure that doesn't happen. Imputation works by approximating missing values with existing values. However, if a column contains a high number of missing values, a large percentage of the column would be approximated by a small percentage. This could potentially create a column without useful information for machine learning pipelines. By using the `HighlyNullDataCheck()` data check, EvalML will alert you to this potential problem by returning the columns that pass the missing values threshold."
+    "Missing data or rows with `NaN` values provide many challenges for machine learning pipelines. In the worst case, many algorithms simply will not run with missing data! EvalML pipelines contain imputation [components](../user_guide/components.ipynb) to ensure that doesn't happen. Imputation works by approximating missing values with existing values. However, if a column contains a high number of missing values, a large percentage of the column would be approximated by a small percentage. This could potentially create a column without useful information for machine learning pipelines. By using the `HighlyNullDataCheck()` data check, EvalML will alert you to this potential problem by returning the columns that pass the missing values threshold."


Not-blocking: Make HighlyNullDataCheck a link to the source?

docs/source/user_guide/pipelines.ipynb

freddyaboulton · 2020-07-16T16:41:07Z

docs/source/user_guide/model_understanding.ipynb

   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "## Feature Importance\n",
+    "## Permutation Importance\n",


Not-blocking: I think it'd be nice to add a few sentences interpreting the plots for users who are not familiar with these techniques.

kmax12 · 2020-07-16T16:51:00Z

docs/source/changelog.rst

@@ -1,7 +1,7 @@
 .. _changelog:


let's change the file name and page name to release notes, so the url updates. currently https://evalml.featurelabs.com/en/ds_update_theme_pandas/changelog.html

docs/source/development.ipynb

jeremyliweishih

🚢 lets goo

angela97lin · 2020-07-16T17:33:19Z

docs/source/install.ipynb

@@ -23,7 +23,7 @@
    "EvalML includes several dependencies in `requirements.txt` by default: `xgboost` and `catboost` support pipelines built around those modeling libraries, and `plotly` and `ipywidgets` support plotting functionality in automl searches. These dependencies are recommended but are not required in order to install and use EvalML. To install these additional dependencies run `pip install -r requirements.txt`.\n",
    "\n",
    "### Core Dependencies\n",
-    "If you wish to install EvalML with only the core required dependencies, include `--no-dependencies` in your EvalML pip install command, and then install all core dependencies with `pip install  -r core-requirements.txt`."
+    "If you wish to install EvalML with only the core required dependencies, include `--no-dependencies` in your EvalML pip install command, and then install all core dependencies with `pip install -r core-requirements.txt`."


Not necessarily related to your change, but I'm curious as to how a user actually installs evalml. Do they not just call pip install evalml and call it a day? Just thinking that with other pip packages, I don't call the pip install -r core-requirements.txt or similar commands.

@angela97lin and I just chatted about this. And I made some improvements to the install page as a result. Long-term, its unclear if we need to document our core-requirements.txt vs requirements.txt on a user-facing install page. Maybe we just keep that context for developers / contributors.

angela97lin

This is some really awesome stuff!! Thank you for doing this and adding in so much more documentation 👏 All my comments are minor / not blocking :)

docs/source/user_guide/automl.ipynb

docs/source/user_guide/components.ipynb

docs/source/user_guide/objectives.ipynb

…ormat changed. Add svg logo in top left.

… in nav bar

dsherry · 2020-07-16T20:07:16Z

Will merge when tests are green!

dsherry commented Jul 16, 2020

View reviewed changes

dsherry marked this pull request as ready for review July 16, 2020 14:29

auto-assign bot assigned dsherry Jul 16, 2020

dsherry requested review from kmax12, jeremyliweishih, angela97lin, ctduffy, eccabay and freddyaboulton July 16, 2020 14:29

jeremyliweishih requested changes Jul 16, 2020

View reviewed changes

dsherry force-pushed the ds_update_theme_pandas branch from 9e16318 to 8e6a427 Compare July 16, 2020 16:13

ctduffy reviewed Jul 16, 2020

View reviewed changes

dsherry requested a review from jeremyliweishih July 16, 2020 16:32

ctduffy reviewed Jul 16, 2020

View reviewed changes

docs/source/start.ipynb Outdated Show resolved Hide resolved

freddyaboulton approved these changes Jul 16, 2020

View reviewed changes

kmax12 approved these changes Jul 16, 2020

View reviewed changes

jeremyliweishih approved these changes Jul 16, 2020

View reviewed changes

angela97lin reviewed Jul 16, 2020

View reviewed changes

angela97lin approved these changes Jul 16, 2020

View reviewed changes

dsherry added 4 commits July 16, 2020 15:37

Use pydata sphinx theme. Add placeholder sub-headings since nav bar f…

f4432c0

…ormat changed. Add svg logo in top left.

Add new index

cf40da4

Update install

b43d454

Update start

f8a86ff

dsherry added 22 commits July 16, 2020 15:37

Update links

1a776a8

Clean up old notebooks

f47b759

Updated some links

2301de2

Remove unnecessary component

74e6a54

Update a couple links

dd68219

Use RF to avoid docs warning

635c725

Update top-left nav icon

63715b0

Rename dev resources to Development. Move changelog to Release Notes,…

5fa53cd

… in nav bar

Retitled changelog as release notes

6ad3e18

Second try on renaming to development

da5c9dd

Changelog

9a25121

Lint

ca1bfef

Try different logo

8bc45a5

Update main logo to svg

8e1b0b5

Update favicon

b767ef5

Delete empty cell

885f905

Grammar, rephrasing, fix bullets and links

bce820b

Add clarification to install instructions about core dependencies

a33710b

Debug why index title isn't smaller

bd422ef

Misc comments

c04ff89

Move FAQ to user guide, delete Development page

4b68526

Comments

66efc5b

dsherry force-pushed the ds_update_theme_pandas branch from a4564e6 to 66efc5b Compare July 16, 2020 19:37

dsherry added 3 commits July 16, 2020 15:41

Move changelog to release notes

618a5ef

Update faq links

1817fb3

Update changelog check to say release notes

dd9e5d4

dsherry merged commit 947d814 into main Jul 16, 2020

dsherry deleted the ds_update_theme_pandas branch July 16, 2020 20:14

dsherry mentioned this pull request Jul 16, 2020

Release v0.11.2 #939

Merged

		@@ -137,7 +103,7 @@
		"source": [
		"## Precision-Recall Curve\n",

Update and overhaul documentation #937

Update and overhaul documentation #937

Conversation

dsherry commented Jul 16, 2020 • edited Loading

codecov bot commented Jul 16, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angela97lin commented Jul 16, 2020

ctduffy commented Jul 16, 2020

ctduffy commented Jul 16, 2020

dsherry commented Jul 16, 2020

jeremyliweishih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmax12 commented Jul 16, 2020

Choose a reason for hiding this comment

dsherry commented Jul 16, 2020

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyliweishih left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angela97lin left a comment

Choose a reason for hiding this comment

dsherry commented Jul 16, 2020

dsherry commented Jul 16, 2020 •

edited

Loading

codecov bot commented Jul 16, 2020 •

edited

Loading

jeremyliweishih left a comment •

edited

Loading