Speed up docs build #2430

freddyaboulton · 2021-06-23T14:21:47Z

Pull Request Description

Following the process described in this document

I think this is about a 45% reduction in total run time, so it's about twice as fast now!

It's hard to look at the internals of sphinx process but I think there are two main buckets:

Running our notebooks and converting them to html
Creating our api ref from our source code by importing our modules

The sphinx build time is about 8 minutes total. It takes about 5 minutes to run all of the notebooks. So the other 3 minutes are sphinx creating our api reference according to our templates.

I scaled the notebook computations as much as possible while still being reasonable, so that 5 minutes looks hard to beat (feel free to make suggestions!). I'm not exactly sure how to speed up the api reference beyond setting collapse_navigation to True. It's not clear how we can profile that to see where the bottleneck is (if it exists). One thing that comes to mind is using the -j parameter to build the api ref in parallel, but RTD does not let us control that as far as I know (once again, correct me if I'm wrong!).

So I think this 8 minute build time is about as low as we can go if we still want to have dynamically generated docs, which I think we do!

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

codecov · 2021-06-23T14:25:12Z

Codecov Report

Merging #2430 (bebedd6) into main (150a1b8) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##            main   #2430   +/-   ##
=====================================
  Coverage   99.7%   99.7%           
=====================================
  Files        283     283           
  Lines      25478   25478           
=====================================
  Hits       25378   25378           
  Misses       100     100

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 150a1b8...bebedd6. Read the comment docs.

freddyaboulton · 2021-06-23T20:12:48Z

docs-requirements.txt

 Sphinx>=2.0.1,<4.0.0
 nbconvert>=5.5.0
 nbsphinx>=0.8.5
+ipython-autotime


This is just to time the notebook cells. I will delete before merge.

freddyaboulton · 2021-06-23T20:13:48Z

docs/source/conf.py

 html_theme_options = {
    "github_url": "https://github.com/alteryx/evalml",
    "twitter_url": "https://twitter.com/AlteryxOSS",
+    "collapse_navigation": True,


This turns out to be pretty important. It shaves about 3-4 minutes from build time without any change to the docs that I can notice 🤷

https://sphinx-rtd-theme.readthedocs.io/en/stable/configuring.html#confval-collapse_navigation

🤯 amazing.

I wonder if the navigation depth also helped

freddyaboulton · 2021-06-23T20:14:26Z

docs/source/demos/lead_scoring.ipynb

   "metadata": {},
   "source": [
-    "Because the lead scoring labels are binary, we will use `AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary')`. When we call `.search()`, the search for the best pipeline will begin. "
+    "Because the lead scoring labels are binary, we will use set the problem type to \"binary\". When we call `.search()`, the search for the best pipeline will begin. "


It was bugging me that AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary') is not how we actually create automl in the next cell 😂

freddyaboulton · 2021-06-23T20:15:27Z

docs/source/demos/lead_scoring.ipynb

    "                      problem_type='binary',\n",
    "                      objective=lead_scoring_objective,\n",
    "                      additional_objectives=['auc'],\n",
+    "                      allowed_model_families=[\"catboost\", \"random_forest\", \"linear_model\"],\n",


Seems like the only way to not fail the lead_score >= auc score assertion is to train catboost on the entire data. I think this is the cleanest way to do that while also cutting out some pipelines to not spend about a minute on this search.

freddyaboulton · 2021-06-23T21:16:56Z

docs/source/conf.py

    p.mkdir(parents=True, exist_ok=True)
    shutil.copy("disable-warnings.py", "/home/docs/.ipython/profile_default/startup/")
    shutil.copy("set-headers.py", "/home/docs/.ipython/profile_default/startup")
+    shutil.copy("ipython_config.py", "/home/docs/.ipython/profile_default")


Will also delete before merge.

chukarsten

I love this @freddyaboulton , thanks for making such an impactful change so quickly.

chukarsten · 2021-06-23T22:13:40Z

docs/source/user_guide/model_understanding.ipynb

   "source": [
    "from evalml.model_understanding.graphs import partial_dependence\n",
-    "partial_dependence(pipeline_binary, X, features='mean radius')"
+    "partial_dependence(pipeline_binary, X_holdout, features='mean radius', grid_resolution=5)"


classic grid resolution 👍

dsherry

👏 incredible

dsherry · 2021-06-23T22:23:56Z

docs/source/conf.py

 html_theme_options = {
    "github_url": "https://github.com/alteryx/evalml",
    "twitter_url": "https://twitter.com/AlteryxOSS",
+    "collapse_navigation": True,


🤯 amazing.

I wonder if the navigation depth also helped

dsherry · 2021-06-23T22:32:34Z

docs/source/demos/cost_benefit_matrix.ipynb

    "from evalml import AutoMLSearch\n",
-    "automl = AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary', objective='log loss binary')\n",
+    "automl = AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary', objective='log loss binary',\n",
+    "                      max_iterations=5)\n",


Seems fine. We'll probably have to revisit this when the new automl algo comes in because max_iterations may no longer be supported at that point.

Agreed! Hopefully there's a "super fast" mode hehe

dsherry · 2021-06-23T22:33:07Z

docs/source/demos/fraud.ipynb

   "outputs": [],
   "source": [
-    "X, y = evalml.demos.load_fraud(n_rows=5000)"
+    "X, y = evalml.demos.load_fraud(n_rows=1000)"


Does the fraud demo still read OK with this change? I seem to remember us changing this because the numbers in the demo were screwy when the data size was too small. But I could be wrong.

@freddyaboulton
Looks like for the fraud demo - both pipelines score evenly on fraud. AUC is actually higher on the fraud optimized pipeline as well. Not sure if this is a change in behavior but could make this section a little confusing.

I think the fraud demo is currently broken which is why @bchen1116 is reworking the fraud objective? It's possible we'll have to increase the size of the data once the fraud objective "works" again but when we get there, we can do what we're doing now for lead scoring and limit the model families to keep the computation fast.

So I think this is the simplest way to not spend a lot of time in this notebook while we fix it. Let me know what you think!

@freddyaboulton works for me! thanks for explaining 😄

Ohhh nvm, lower scores are better for fraud! I'll limit the model families now!

Ohh, that screen shot was from stable, not latest. Looks like it is broken on latest/main hehe

https://evalml.alteryx.com/en/latest/demos/fraud.html

dsherry · 2021-06-23T22:35:02Z

docs/source/demos/text_input.ipynb

    "\n",
    "input_data = urlopen('https://featurelabs-static.s3.amazonaws.com/spam_text_messages_modified.csv')\n",
-    "data = pd.read_csv(input_data)\n",
+    "data = pd.read_csv(input_data)[:750]\n",


Nice. Dang how big was this dataset?

Ah, 3k rows. Not too bad. Truncating seems good to me tho

Yea 3k isn't too bad but the problem is that the text featurizer takes up a lot of the run time. Perhaps when we add component caching, we can go back to the full 3k rows!

dsherry · 2021-06-23T22:35:17Z

docs/source/ipython_config.py

@@ -0,0 +1,2 @@
+
+c.InteractiveShellApp.extensions = ['autotime']


I'm guessing this will also be deleted before merge?

dsherry · 2021-06-23T22:35:46Z

docs/source/user_guide/automl.ipynb

   "outputs": [],
   "source": [
    "X, y = evalml.demos.load_breast_cancer()\n",
+    "X, y = X[:250], y[:250]\n",


Is this necessary? You already set n_rows at the top

This is a different dataset from fraud! Let me time if I can use the full breast cancer data.

Yea I think shrinking the model families and doing one less batch are sufficient so I'll use the full dataset!

dsherry · 2021-06-23T22:36:24Z

docs/source/user_guide/model_understanding.ipynb

+    "X_train, X_holdout, y_train, y_holdout = evalml.preprocessing.split_data(X, y, problem_type='binary',\n",
+    "                                                                         test_size=0.2, random_seed=0)\n",
+    "\n",
+    "\n",


Niiiice good call

jeremyliweishih

great work @freddyaboulton! I'll double check the docs itself but this looks good to me.

jeremyliweishih · 2021-06-24T01:46:37Z

docs/source/demos/lead_scoring.ipynb

    "                      problem_type='binary',\n",
    "                      objective=lead_scoring_objective,\n",
    "                      additional_objectives=['auc'],\n",
+    "                      allowed_model_families=[\"catboost\", \"random_forest\", \"linear_model\"],\n",


freddyaboulton added 2 commits June 23, 2021 10:20

Profile notebook cells

f2c944b

Copy profile

2f97c82

freddyaboulton added 4 commits June 23, 2021 14:25

Speed up docs

fddbafc

Merge branch 'main' into speed-up-docs-build

6c2a3a6

Release notes

a52fda8

Removing unneccesary lines

589fe3b

freddyaboulton commented Jun 23, 2021

View reviewed changes

freddyaboulton added 2 commits June 23, 2021 16:21

Delete Extra cell

f33710b

Merge branch 'main' into speed-up-docs-build

e27a5a1

freddyaboulton marked this pull request as ready for review June 23, 2021 20:47

auto-assign bot assigned freddyaboulton Jun 23, 2021

freddyaboulton changed the title ~~WIP: Speed up docs build~~ Speed up docs build Jun 23, 2021

freddyaboulton requested review from ParthivNaresh, angela97lin, bchen1116, christopherbunn, chukarsten, dsherry and jeremyliweishih June 23, 2021 21:02

freddyaboulton commented Jun 23, 2021

View reviewed changes

chukarsten approved these changes Jun 23, 2021

View reviewed changes

dsherry approved these changes Jun 23, 2021

View reviewed changes

jeremyliweishih approved these changes Jun 24, 2021

View reviewed changes

chukarsten and others added 3 commits June 24, 2021 10:18

Merge branch 'main' into speed-up-docs-build

fa315e3

Deleting autotime and limiting model families for fraud

2b17816

Merge branch 'main' into speed-up-docs-build

bebedd6

freddyaboulton merged commit 6f3b6e9 into main Jun 24, 2021

dsherry mentioned this pull request Jul 2, 2021

Release v0.28.0 #2467

Merged

freddyaboulton deleted the speed-up-docs-build branch May 13, 2022 14:50

		@@ -0,0 +1,2 @@

		c.InteractiveShellApp.extensions = ['autotime']

Speed up docs build #2430

Speed up docs build #2430

Uh oh!

Conversation

freddyaboulton commented Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Uh oh!

codecov bot commented Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton Jun 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dsherry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremyliweishih left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

freddyaboulton commented Jun 23, 2021 •

edited

Loading

codecov bot commented Jun 23, 2021 •

edited

Loading

freddyaboulton Jun 23, 2021 •

edited

Loading

freddyaboulton Jun 23, 2021 •

edited

Loading