Test Doc Builds

[jeff-hernandez]

Closes #159 by adding integration tests for doc builds.

[codecov]

Codecov Report

Merging #165 into main will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main      #165   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           28        28           
  Lines         1254      1254           
=========================================
  Hits          1254      1254           

[jeff-hernandez]

The docs take ~12 minutes to build on CircleCI. Locally, it takes about a minute. Not sure if this is a problem.

[rwedge]

I think this looks good, only comment is if you want warnings to be treated as errors by the docs build, there's a flag you need to set

[jeff-hernandez]

Hmm, after updating the doc builds to treat warnings as errors, a context deadline exceeded problem occurs on CircleCI. I don't get any errors when building the docs locally.

Solved by increasing the no output timeout (f4e4a1d).

[jeff-hernandez]

Found this on CircleCI Support:

In CircleCI, a command will be killed if a certain period of time has passed with no output. By default, this is 10 minutes. This is designed to prevent errors in builds from hanging using a large number of credits unintentionally.

[rwedge]

I reran a previous commit on CircleCi and also got a timeout, so the warnings as errors may not be the issue. It seems that a lot of time is being spent building the page on the fan degradation example

[jeff-hernandez]

On Read the Docs, the latest build with the notebooks took ~6 minutes. On CircleCI, it took ~20 minutes.

[jeff-hernandez]

I'm thinking about using a smaller sample of data in the notebooks to speed up the doc builds on CircleCI .

[rwedge]

What if we tried using a time-based timeout for the EvalML search instead of number of pipelines?

[jeff-hernandez]

@rwedge sounds good, will try the time-based search.

[jeff-hernandez]

I didn't notice a difference in the doc builds after pinning the EvalML version (a14f2bd) or using a time-based search (50dfe51).

[jeff-hernandez]

The latest build on CircleCI with an additional notebook took ~37 minutes.

[jeff-hernandez]

The latest build using smaller samples of data took ~28 minutes on CircleCI.

[jeff-hernandez]

After increasing the resource class of the executor, the latest build took ~11 minutes.

[jeff-hernandez]

I tried running the notebooks before the doc build at runtime. The build took ~7 minutes. Then after minor changes in the makefile (c9cf833, 62a4c20), the build took ~14 minutes. Not sure why this is.

[jeff-hernandez]

After limiting the search by model type, the build took a minute!

[rwedge]

Looks good