[MXNET-744] Docs build tools update

[aaronmarkham]

Description

While trying figure out how to debug the Sphinx website theme for #11916, I found the build tools frustratingly slow and nearly impossible to use for problems related to a full site build with the versions dropdown. So, I rewrote parts of it so you can do amazing and practical things like:

⭐ Decide what documentation sets to build on a per version basis!
⭐ Speed up the full version site build by 2.5x (was 43 minutes, now 17 minutes)!
⭐ Make incremental front-end site updates and build 14.3x faster (was 43 minutes, now 3 minutes)!
⭐ Optimize Sphinx configurations and pass these updates to the older versions of the site!
⭐ Not want to rage quit every time the docs build fails 40 minutes in!
⭐ Actually run the R docs build!

I'm pretty stoked. I hope you are too.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)

Preview

I'm actively running tests here, but you can see the current output (probably): http://34.201.8.176/

Changes

💥 Each version build happens in its own folder. This lets the MXNet build be cached if there's no update to the library. Big win for time savings here. Tradeoff is it takes up more space.
💥 When you build Sphinx docs with make html you can pass a new param, BUILD_VER={tag}, like BUILD_VER=1.2.0, and it'll load the settings according to that version
💥 Settings per version? Yes, in a new file at /docs/settings.ini we now have configs for each version and each document set. I could have also used the Makefile, but this seemed to be better for handling all of the version support variations. You can configure the build to build what you want, and more importantly, turn it off for a version where it isn't possible... like for Clojure and any previous version tag. Only the most recent version or master or your fork might have it.
💥 What's this about your fork? Yes, you can now build docs with your own fork and output different branches of your fork to represent the different version on the website output. ✨ Cool, right? ✨ Gawd, why didn't I do this earlier!?

And I tweaked the css a bit so the h1-h4 tags pop a little more. The mxnet.css file gets pulled from the artifacts folder now, but we can alter this design down the line.

Usage

Building Docs with Sphinx

From the docs folder:
You can just use make html and it'll load the defaults which will build everything:

[document_sets_default]
clojure_docs = 1
doxygen_docs = 1
r_docs = 1
scala_docs = 1

Or you can use a specific version and it'll load the settings for it:

make html USE_OPENMP=1 BUILD_VER=1.0.0

From settings.ini:

[document_sets_1.0.0]
clojure_docs = 0
doxygen_docs = 1
r_docs = 0
scala_docs = 0

Building the Full Site with Your Fork

The previous example is a precursor to the more complicated process of building the full site and its many versions.
The scripts used by CI and that can be used yourself for building dev version of the site are in build_version_doc and are still build_all_version.sh and update_all_version.sh. Now they take optional params for your fork.

These scripts call make docs which in turn calls make html from the docs directory. So, they inherit what I just described above using the settings file. The version tags get passed down from the shell script into the docs Makefile invocation, which then is read in the Sphinx plugin mxdoc.py, which reads the settings file, which is what then calls all of the other docs generation tools. I had to bubble up the version info all the way to this plugin.

Note: I would certainly like to see this less complicated, but this is the current architecture, so... until we can possible refactor the whole thing with CMAKE, this is what we have to contend with...**

Building Each Version and Optionally Using Your Fork

Build the content of the 1.2.0 branch in the main project repo to the 1.2.1 folder:

./build_all_version.sh "1.2.0" "1.2.1"

Using the main project repo, map the 1.2.0 branch to output to a 1.2.1 directory; others as is:

./build_all_version.sh "1.2.0;1.1.0;master" "1.2.1;1.1.0;master"

Using a custom branch and fork of the repo, map the branch to master, map 1.2.0 branch to 1.2.1 and leave 1.1.0 in 1.1.0:

./build_all_version.sh "sphinx_error_reduction;1.2.0;1.1.0" "master;1.2.1;1.1.0" https://github.com/aaronmarkham/incubator-mxnet.git

Updating Each Version and Optionally Using Your Fork

Assuming you build 1.2.1, 1.1.0, 1.0.0, and master, you need to inject the versions dropdown to reflect these options for each page on the site:

./update_all_version.sh "1.2.1;1.1.0;1.0.0;master" master http://mxnet.incubator.apache.org/

It doesn't use your fork at this point, but it will pull the latest project README.md from your current branch and use that for the root of the output site.

Comments

Building R Docs

I'm not adding the R deps and making CI changes here. That's just too much in one PR. However, in a followup PR, or if you want test the pipeline with the R docs build turned on you need the following in addition to the R deps already in the docs setup scripts in CI or the one in build_version_doc:

sudo apt-get install \
    texinfo \
    texlive \
    texlive-fonts-extra \

You would also want to change settings.ini and enable the build of the R docs (per version you want).

Building Scala Docs

The current Scala config is incompatible with version 1.1.0 or earlier. You'll get an error about the namespace change. I'm disabling the build for this in those earlier versions in the provided settings.ini. Please discuss with me if this is a bad choice and if we need to use a custom config for those earlier versions. The function in mxdoc.py for scaladocs could use the new version param, and make some conditional statements for how it builds...

[aaronmarkham]

@marcoabreu @ankkhedia @kpmurali @szha - Please review.
@vandanavk / @cclauss - I made this compatible with the R docs changes in PR #11970 - this can stack or supersede as needed.

[aaronmarkham]

@marcoabreu this does impact CI. The scripts I modified should work seamlessly with the current jobs and reduce the website build times by half.
We should chat about regularly running docs builds for PRs. The new modes would allow us to configure it to run specific sets, so if you update a Python file you don't run scaladocs. Or those are run separately...

[marcoabreu]

We are already running docs as part of the PR stage, right? We could move the doc generation into the build stage and then run all flavours. WDYT?

[aaronmarkham]

@marcoabreu I don't know much about the PR checking logic. Can you point me to it?
It seems like we could scan a PR and if it matches *.scala then you run make scalapkg then scaladocs. (In this PR's features, you'd update the settings file accordingly. I could extend things to use env vars or the settings file, so then Jenkins could set the env var, and that would override the default settings.)
And do a similar match for each kind of doc, rather than run all docs for every update.
Conversely, if I make a .md, .js, or .html change, we don't run the full suite of CI on that PR. If docs changes are a separate pipeline, wouldn't this be possible?

[marcoabreu]

https://github.com/apache/incubator-mxnet/blob/master/Jenkinsfile#L1088
https://github.com/apache/incubator-mxnet/blob/master/ci/docker/runtime_functions.sh#L785

I think your method is something we will get to in the future, but it requires quite some work until we are there.

[aaronmarkham]

So assuming this passes CI, this PR is good to go. It passed a test here:
http://jenkins.mxnet-ci.amazon-ml.com/job/test-website-build/15/

[lebeg]

LGTM

[aaronmarkham]

That switching to separate Python 2 and 3 in CI is breaking this. I'm working on a patch.

[haojin2]

This commit is causing problems on CI: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12188/1/pipeline
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12192/1/pipeline
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12184/2/pipeline
@aaronmarkham Can you take a look?

[aaronmarkham]

It was actually the switch to Python 3 in CI that overlapped on this PR. Docs has been building in Python 2 prior to last week. Anyways, I believe I fixed it. Here's my test run:
http://jenkins.mxnet-ci.amazon-ml.com/job/test-website-build/22/console

See #12195. It needs a review.

[nswamy]

aaronmarkham understand CI changing to Py3 caused your code to break, however we cannot have CI blocked. Thanks for acting swiftly on this. I'll review #12195 and merge once it passes CI