build docs and push to branch in CI

[mattip]

This creates a job to build docs and push them. The basic idea is to reuse the install part of the unittest smoke_test jobs, and then do

pushd doc
pip install -r requriements.txt
build html
popd

Then a separate step commits the built documentation, checks out the gh-pages branch, copies the rendered documentation into the correct directory, commits it, and pushes the result to the branch.

Things I don't understand about the config.yml, I hope I did it right
- how do the unittest_linux_cpu jobs know not to run if the build job fails? There is no requires clause

• git push hopefully will JustWork when a PR is merged, although it will fail when done on a fork (like when people submit a PR). Should I limit the job to only run on nightly to avoid the problem?

[mattip]

ping @brianjo

[mattip]

hmm, I wanted to use the built wheel which should be in ./dist but instead seem to be building the project. @seemethere any ideas what I am doing wrong?

[mthrok]

hmm, I wanted to use the built wheel which should be in ./dist but instead seem to be building the project. @seemethere any ideas what I am doing wrong?

The script used in the preparation steps (.circleci/unittest/linux/scripts/install.sh and .circleci/unittest/linux/scripts/setup_env.sh) are intended for installing torchaudio from source.

If the binary package is preferable, you can set up things like smoke_test do, which depends on the binary build process.

audio/.circleci/config.yml.in

Lines 275 to 292 in b7c17f8

	smoke_test_linux_conda:
	<<: *smoke_test_common
	steps:
	- attach_workspace:
	at: ~/workspace
	- designate_upload_channel
	- run:
	name: install binaries
	command: |
	set -x
	source /usr/local/etc/profile.d/conda.sh && conda activate python${PYTHON_VERSION}
	conda install -v -y -c pytorch-${UPLOAD_CHANNEL} pytorch
	conda install -v -y -c file://$HOME/workspace/conda-bld torchaudio
	- run:
	name: smoke test
	command: |
	source /usr/local/etc/profile.d/conda.sh && conda activate python${PYTHON_VERSION}
	python -c "import torchaudio"

[mthrok]

• how do the unittest_linux_cpu jobs know not to run if the build job fails? There is no requires clause

unittest workflow and binary workflow are completely independent. This is because binary workflow can fail due to external reasons like network issue, but most often we still want to run unit tests for the PR we are working on.

• git push hopefully will JustWork when a PR is merged, although it will fail when done on a fork (like when people submit a PR). Should I limit the job to only run on nightly to avoid the problem?

@seemethere Does the CI job executor have a push privilege?

[mthrok]

Looks good to me, although I might be missing some detail. Let's see how things work out on nightly build.

[seemethere]

Looks mostly good to me, just one thing

[mattip]

@seemethere build_docs is failing, it is not "continuing" from where the binary_linux_wheel_3.8 is finishing. What do I need to preserve to not have to rebuild from 0?

[mattip]

OK, CI passes. There are warnings when building the docs, but that is not new: brianjo/audio#1.

[mthrok]

Looks good. Let's see how the build process goes. Thanks!

[mthrok]

Sorry for making comment after the merge, but this means that release documentation is only pushed on release tag, right?
How about changing this to release/* branch so that even after the release, we can fix the documentation by pushing doc change to release/* branch?
@seemethere @brianjo @mattip

[mattip]

The way I see it is like this:

• gh-pages/master is built every night and documents the latest and greatest HEAD of the development
• gh-pages/0.8 and all the numbered directories document the latest release; what end users would download to their machines if they install from conda or pip and specified a version.
• stable points to the latest numbered directory, what users would get if they conda or pip install and do not specify a version.

Bugs in the release would indeed be committed to the release/* branch via PRs, these would trigger the build_docs build and you could examine the artifacts*. But the new documentation would only become "official" when the release is tagged and the fix is available to the general public as a major/minor/patch release, so a tag event is when the contents of the numbered directory should change.

You could fix bugs in the documentation that are discovered after release by building documentation locally and pushing the built documentation to the correct directory (using the steps in .circleci/build_docs/commit_docs.sh or calling the script to do it for you).

* I should improve the CI process and make the artifacts available for viewing after the build_docs step.