Docs build all versions refactor

[aaronmarkham]

Description

I updated the build_all_versions.sh logic so that it works locally. It can repair the docs site when the version builds are broken. You can now build specific tags, and update them as a collection or one at a time. You can pick a specific tag to be the default.

There's a dockerfile that can be used with the CI build script and general script (build_site_tag) that can be called to generate any tags and update the site content.

You can now update AddVersion.py in master and have its changes override the site content, so it can patch any issues without being locked into code from a tag. Just run update_all_version.sh and it will regen the site with the fixes.

Checklist

Essentials

• Changes are complete (i.e. I finished coding on this PR)
• Code is well-documented - should follow-up with another PR to better describe the docs build system, but at this point my view is still limited.
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• I created a setup script (setup.sh) to handle all of the deps that were missing from the documentation - for an ubuntu setup.
• I split out the docker setup to another script (setup_docker.sh) since the directions for this work, but they only help you create a local version of master and not the production style "versioned" site.
• I split the logic in build_all_version.sh which is build and then update (injecting dom elements like the version dropdown) to be just build, and then created another script (update_all_version.sh) to handle the updates. This way you can build everything, then test how the injection stuff works. Otherwise, for every change you're forced to rebuild each version, which is crazy slow.
  • The logic also had you check out from master and then use the scripts within master to do the updating. This makes it hard to test changes you want to make, so I moved this logic to using your current modified local version, and then update the temp build directories.
  • The files mixed up the tag list naming convention, so I made sure it is always tag_list.txt and not tag.txt sometimes.
  • I set USE_OPENMP=1 - because the console output during build kept saying it would be better/faster with it on, so why is it off?

Usage

Use an Ubuntu image. I'm using Deep Learning AMI v5.

./setup_docs_ubuntu.sh

Or use Docker as described below.
The scripts can be run stand-alone or in conjunction.

build_all_version.sh

Takes one argument:

• tag list - space delimited list of Github tags; Example: "1.1.0 1.0.0 master"
  Example Usage:
  ./build_all_version.sh "1.1.0 1.0.0 0.12.1 0.12.0 0.11.0 master"

update_all_version.sh

Takes three arguments:

• tag list - space delimited list of Github tags; Example: "1.1.0 1.0.0 master"
• default tag - which version should the site default to; Example: 1.0.0
• root URL - for the versions dropdown to change to production or dev server; Example: http://mxnet.incubator.apache.org/

Example Usage:
./update_all_version.sh "1.1.0 1.0.0 0.12.1 0.12.0 0.11.0 master" 1.1.0 http://mxnet.incubator.apache.org/

build_site_tag.sh

This one is useful for Docker. When you run the image you can call this script as a command a pass the tags, default tag, and root url.
Takes the same three arguments that update_all_version.sh takes.
It will execute build_all_version first, then execute update_all_version next.

Example Usage:
./build_site_tag.sh "1.1.0 master" 1.0.0 http://mxnet.incubator.apache.org/
Then run a web server on the outputted VersionedWeb folder.

Docker Usage

The dockerfile will build all of the docs when you create the docker image. You can also run the scripts listed above to regenerate any tag or collection of tags.

Build like:
sudo docker build -t aaronmarkham:mxnet-docs-base .

Run like:
sudo docker run -it aaronmarkham:mxnet-docs-base

Comments

Please help me test this out and let me know what suggestions you have.
The docker/Jenkins integration needs some testing but I think the scripts should be plug n play or pretty close.

[marcoabreu]

Docker is already installed on Ci, thus there's no need to have this script. I don't think that a script to do basic docker setup should be part of MXNet.

[aaronmarkham]

Ok, I'm removing this.

[marcoabreu]

Please move this code into a Dockerfile instead of a shell script. See https://github.com/apache/incubator-mxnet/blob/master/docs/Dockerfile for reference

[aaronmarkham]

This is taking a really long time to test, so that's why an update hasn't landed yet! I had the benefit of some deps handled by the DL AMI, but in a vanilla container I had a lot of troubleshooting to do. That being said, I'm hoping I'm near the end of testing the scripts inside a container for every tag we need.

I'm quite tempted to modify the scripts though so that they take in the tag as an input and we do a RUN for each, so we can use the docker cache for when the build steps are completed fine.

[marcoabreu]

It would be good if the tag(s) can be passed as argument. E.g. "docker run Dockerfile_doc -e tags=1.1.0,1.0.0"

[marcoabreu]

Btw, please remove this file as it is no longer needed

[aaronmarkham]

I'd like to keep it around... or maybe move the info to documentation since it is really helpful for installing outside of docker.

[marcoabreu]

Feel free to move the instructions into a README.md.

[marcoabreu]

Is this file deprecated? Please remove

[marcoabreu]

TODO: remove

[marcoabreu]

Please add a comment that update_all_versions.sh should be run afterwards. Maybe even print it?

[aaronmarkham]

Added.

[marcoabreu]

Have this as environment variable that gets passed through docker?