Docs build failures are difficult to identify

[mruberry]

Warnings in the docs build are now a build failure in CI. These failures, however, are difficult for developers to relate to their PRs. See, for example, #50046 (comment).

The warning:

Jan 05 15:57:22 docstring of torch.linalg.qr:29: WARNING: Explicit markup ends without a blank line; unexpected unindent.

appears in a lengthy printout that can include other warnings like these:

Jan 05 15:57:23 /opt/conda/lib/python3.6/site-packages/torch/jit/_trace.py:804: TracerWarning: Trace had nondeterministic nodes. Did you forget call .eval() on your model? Nodes:
Jan 05 15:57:23 	%13 : Float(4, strides=[1], requires_grad=0, device=cpu) = aten::rand(%8, %9, %10, %11, %12) # <doctest default[0]>:2:0
Jan 05 15:57:23 This may cause errors in trace checking. To disable trace checking, pass check_trace=False to torch.jit.trace()
Jan 05 15:57:23   _module_class,
Jan 05 15:57:23 /opt/conda/lib/python3.6/site-packages/torch/jit/_trace.py:804: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Jan 05 15:57:23 With rtol=1e-05 and atol=1e-05, found 4 element(s) (out of 12) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 0.9027273058891296 (0.904914915561676 vs. 0.0021876096725463867), which occurred at index (0, 3).
Jan 05 15:57:23   _module_class,
Jan 05 15:57:23 /opt/conda/lib/python3.6/site-packages/torch/jit/_trace.py:804: TracerWarning: Trace had nondeterministic nodes. Did you forget call .eval() on your model? Nodes:
Jan 05 15:57:23 	%14 : Float(1, 4, strides=[4, 1], requires_grad=0, device=cpu) = aten::rand(%9, %10, %11, %12, %13) # <doctest default[0]>:2:0
Jan 05 15:57:23 This may cause errors in trace checking. To disable trace checking, pass check_trace=False to torch.jit.trace()
Jan 05 15:57:23   _module_class,
Jan 05 15:57:23 /opt/conda/lib/python3.6/site-packages/torch/jit/_trace.py:804: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Jan 05 15:57:23 With rtol=1e-05 and atol=1e-05, found 4 element(s) (out of 8) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 0.7007782459259033 (0.7614061832427979 vs. 0.06062793731689453), which occurred at index (0, 3).
Jan 05 15:57:23   _module_class,
Jan 05 15:57:23 /opt/conda/lib/python3.6/site-packages/torch/jit/_recursive.py:197: UserWarning: 'mods' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
Jan 05 15:57:23   " but it is a non-constant {}. Consider removing it.".format(name, hint))
Jan 05 15:57:24 Makefile:38: recipe for target 'doctest' failed

It would be nice to more clearly describe to developers why the doc build(s) failed on their PR.

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @jlin27 @mruberry @rgommers @mattip @ranman

[mattip]

The easiest thing to do would be to improve the error message at the end of the build step, either in the Makefile or in the circleCI script that calls make html to state "if this step fails, look back in the log for WARNING: (in capital letters)". A more advanced solution would be to do something like

set -o pipefail
make html | tee /tmp/doc_build.log
if [ $? -ne 0 ]; then 
    echo "doc build error detected:"; 
    grep WARNING /tmp/doc_build.log;
fi

I wonder if there is a way to write a test to make sure it works as advertised

[mruberry]

Telling people to search for "WARNING" would be a major improvement. Replicating the WARNINGs with some context at the bottom of the build output would be even better. We could also consider creating an artifact and telling people to search for WARNING and/or look at the artifact to understand the failures.

[rgommers]

There is an artifact already, adding to the message that the pytorch_python_doc_build CI job takes you straight to the rendered html that you can inspect for issues may make sense.