Add rstcheck to CI #3624

stsewd · 2018-02-16T18:20:09Z

This is an initial step to have more consistent docs. Closes #3622

~~Depends on #3498 (since some errors are solved there)~~

stsewd · 2018-02-16T18:21:28Z

tox.ini

 changedir = {toxinidir}/docs
 commands =
+    rstcheck `find **/*.rst | xargs`


I'm not sure if this is the right place, but looks logical to me, what I don't like is to have to install all the deps from lint.txt. Maybe adding a docs.txt?

Also the --recursive option from rstcheck was added reciently rstcheck/rstcheck@6102bec, so when that version gets released I will update the command :)

It's already released, 3.2.

I also noted that you made a PR there! Hehe! Nice job! :)

stsewd · 2018-02-16T18:24:53Z

After seen the results from travis I will add some rules to a .rstcheck.cfg file and see what problems we can solve.

stsewd · 2018-02-16T19:38:11Z

Looks like currently the tool isn't very mature, we can't specify what errors to ignore (there are some very specific cases that are false positives). Anyway, I think this could be a good improvement to merge in the future :). I'm going to see if I can collaborate on rstcheck.

ericholscher · 2018-02-18T02:02:42Z

Big +1 on this, but generally adding linting requires a lot of fixing before merging.

stsewd · 2018-02-18T03:31:57Z

@ericholscher yeah, probably we may want to do separate PRs for fixing some problems, and keep this branch up to date with master.

humitos · 2018-02-20T16:42:31Z

Seems they are not that many:

api/bookmarks.rst:5: (INFO/1) Duplicate implicit target name: "".
api/builds.rst:5: (INFO/1) Duplicate implicit target name: "".
api/core.rst:5: (INFO/1) Duplicate implicit target name: "".
api/doc_builder.rst:5: (INFO/1) Duplicate implicit target name: "".
api/projects.rst:5: (INFO/1) Duplicate implicit target name: "".
api/vcs_support.rst:5: (INFO/1) Duplicate implicit target name: "".
business/index.rst:2: (INFO/1) Duplicate explicit target name: "readthedocs.com".
custom_installs/local_rtd_vm.rst:2: (INFO/1) Enumerated list start value not ordinal-1: "2" (ordinal 2)
dmca/index.rst:4: (INFO/1) Hyperlink target "our process" is not referenced.

I think we can fix them in this PR also, right?

stsewd · 2018-02-20T16:46:57Z

@humitos the first errors target name: "" are false positives, I'm already collaborating to rstcheck so we can used in a more configurable way :)

humitos · 2018-02-20T16:48:00Z

tox.ini

 changedir = {toxinidir}/docs
 commands =
+    bash -c 'rstcheck `find **/*.rst | xargs`'


I think we should create a [testenv:doclint] with the rstcheck linting process instead of making it part of the docs building

humitos · 2018-02-21T17:56:56Z

tox.ini

+commands =
+    sphinx-build -b html -d {envtmpdir}/doctrees .  {envtmpdir}/html
+
+[testenv:docs-lint]
 deps = -r{toxinidir}/requirements/lint.txt


Here. You just need rstcheck instead all the linting packages, I'd say

(or maybe creating another file docs-lint.txt?)

We need also sphinx for more integration https://github.com/myint/rstcheck#sphinx

Nice! So, I'd say that's better to have a separate reqs file.

stsewd · 2018-02-21T18:26:21Z

requirements/docs-lint.txt

@@ -0,0 +1,2 @@
+-r pip.txt


Not sure if we need all the packages from pip.txt, but just to follow similar convention, and we need Sphinx installed.

No we don't need them. That's the idea of using a different file.

Add here just the packages that are needed. As far as I know, sphinx and rstcheck (also, do not pin them)

humitos · 2018-02-21T18:40:36Z

.travis.yml

@@ -7,6 +7,8 @@ matrix:
  include:
    - python: 2.7
      env: TOXENV=docs
+    - python: 2.7


stsewd · 2018-03-17T18:42:28Z

The 3.3 version of rstcheck was released! https://github.com/myint/rstcheck#id1

Soon I'll be updating this PR :)

stsewd · 2018-03-17T20:53:37Z

Done, I made sure that all the links were shown correctly and that the deletes were not used in other parts of the docs.

stsewd · 2018-03-17T20:55:22Z

Also, sorry for so many commits, you can squashed into one if you want

stsewd · 2018-03-17T21:02:06Z

I missed one file, now the linter for docs should pass.

stsewd · 2018-03-17T21:17:35Z

docs/.rstcheck.cfg

+[rstcheck]
+ignore_directives=automodule,http:get
+ignore_roles=djangosetting,setting
+ignore_messages=(Duplicate implicit target name: ".*")|(Hyperlink target ".*" is not referenced)


Those errors are from some hyperlinks that are referenced globally. The duplicates are really false positives.

By ignoring this now that are false positivies, we won't detect the ones that are true when we updated the docs :(

Yeah, I going to try to solve others and see if I can get a better regex here

There is a better way to expose global hiperlinks? Maybe something like this https://stackoverflow.com/a/19543591/5689214? And for the duplicates target name the most are from subheaders with the same name.

Why not use the :ref: directive for linking across files?

http://www.sphinx-doc.org/es/stable/markup/inline.html#role-ref

rstcheck doesn't search for global refs and are marked as unused. But using ref is fine.

And also I change others place to use the doc directive https://docs.readthedocs.io/en/latest/docs.html#content.

humitos

I think this is a really good improve to have automatically triggered in our PRs.

Regarding the ignore_messages, I'd say that it's good enough for now. Having this and ignore just 2 messages it too much better than ignoring everything as we were doing before this PR.

Optimization: is it possible to trigger this docs-lint env _only when there are changes in the docs/ directory?

humitos · 2018-03-19T14:50:52Z

docs/dmca/index.rst

@@ -4,32 +4,36 @@ DMCA Takedown Policy
 These are the guidelines that Read the Docs follows when handling DMCA takedown
 requests and takedown counter requests. If you are a copyright holder wishing to
 submit a takedown request, or an author that has been notified of a takedown
-request, please familiarize yourself with `our process <Takedown Process_>`_.
+request, please familiarize yourself with `our process`__.


Why we don't leave the extended name here? Isn't it clearer to understand where this will go?

(the same for the other changes in this file)

rstcheck didn't recognize those and marked it as errors. And I didn't find that syntax on the quickref http://docutils.sourceforge.net/docs/user/rst/quickref.html#internal-hyperlink-targets.

`our process <Takedown Process_>`_

links to the "Takedown Process" title (L20 in dcma/index.rst)

If you remove the title from the link, like this

`our process`__

where that link will follow you? If you build the docs locally, you will find that link doesn't work.

The link is referenced just below https://github.com/rtfd/readthedocs.org/pull/3624/files#diff-1df69b2724a0d4a6195621dcef6dec3cR11

I see the link just fine on my browser

(see the link on the bottom)

My mouse is on our process but it doesn't appear on the screeenshot 😁

humitos · 2018-03-19T14:54:09Z

docs/.rstcheck.cfg

+[rstcheck]
+ignore_directives=automodule,http:get
+ignore_roles=djangosetting,setting
+ignore_messages=(Duplicate implicit target name: ".*")|(Hyperlink target ".*" is not referenced)


By ignoring this now that are false positivies, we won't detect the ones that are true when we updated the docs :(

stsewd · 2018-03-19T19:10:16Z

Optimization: is it possible to trigger this docs-lint env _only when there are changes in the docs/ directory?

I'm not sure if that would be possible, maybe a setting on travis? Since the linting is very quick, the heavy stuff comes from travis and installing the deps p:

humitos · 2018-03-23T14:56:33Z

docs/automatic-redirects.rst

@@ -33,7 +33,7 @@ Supported Top-Level Redirects
 .. note:: These "implicit" redirects are supported for legacy reasons.
          We will not be adding support for any more magic redirects.
          If you want additional redirects,
-          they should live at a prefix like :ref:`page-redirect`
+          they should live at a prefix like `Redirecting to a Page`_


Why did you change these ones?

Using ref is advised over standard reStructuredText links to sections (like Section title_) because it works across files, when section headings are changed, and for all builders that support cross-references.

from http://www.sphinx-doc.org/es/stable/markup/inline.html#role-ref

mmm, well I don't like the idea of exposing a lot of links as global refs and more when they aren't used on the global scope, but I will revert those if you consider.

I wonder if we can make a Sphinx extension that automatically ads each section heading as a global ref, prefixed by pagename or similar? That way you wouldn't need to declare all the refs, but still be able to reference them. That's definitely for another ticket/PR though.

http://www.sphinx-doc.org/en/stable/markup/inline.html#role-numref looks interesting, via sphinx-doc/sphinx#2963

Hmm, looks like those are still for explicit section refs, not implicitly creating them, ignore me :)

Found it: http://www.sphinx-doc.org/en/stable/ext/autosectionlabel.html -- it's already built into Sphinx :)

humitos · 2018-03-23T14:58:46Z

docs/contribute.rst

@@ -1,5 +1,3 @@
-.. _contributing-to-read-the-docs:


Be careful when removing these ones. This one in particular, is used at https://github.com/rtfd/readthedocs.org/pull/3624/files#diff-c7f13470cacdf0634e185d4313d02a12R55 (L57 in design.rst) so that link is probably dead after removing this.

Ups, I just missed that one. I usually do a grep to search those references.

humitos · 2018-03-23T15:01:02Z

docs/localization.rst

@@ -1,5 +1,3 @@
-.. _`Localization of Documentation`:


This one is used in faq.rst

Yeah, and was changed to use the doc directive as rtd recommends

https://github.com/rtfd/readthedocs.org/pull/3624/files#diff-c6f0a0722c6a2f86277535d7bcec7f8cR167

humitos

This looks good and very close to merge, from my point of view.

I'm marking this as "Request changes" because there are some internal links that will be broken with these modifications.

We will need to be sure that each link that we modify still works before merging. You can check that by building the documentation by yourself in your local instance:

cd docs
make html
firefox _build/htmls/index.html

stsewd · 2018-03-23T16:36:21Z

@humitos done, just one link was broken, the others were properly modified. By the way I use sphinx-autobuild for building the docs on dev.

stsewd · 2018-05-23T05:32:27Z

I just rebased and solved others conflicts from upstream, this should be ready to go, also I checked that all links work properly.

stsewd · 2018-05-23T05:34:07Z

Also, I found this target on the docs' Makefile make livehtml

ericholscher · 2018-05-24T14:30:08Z

@stsewd yea, that lets you auto-rebuild the docs if you have sphinx-autobuild installed.

ericholscher

Great changes. We should write a blog post about this, so that other folks in the community might be able to benefit from the knowledge here.

/cc @davidfischer, can we add that to our editorial calendar, and I can help write it up w/ Santos' review perhaps.

ericholscher · 2018-05-24T14:31:59Z

docs/automatic-redirects.rst

@@ -33,7 +33,7 @@ Supported Top-Level Redirects
 .. note:: These "implicit" redirects are supported for legacy reasons.
          We will not be adding support for any more magic redirects.
          If you want additional redirects,
-          they should live at a prefix like :ref:`page-redirect`
+          they should live at a prefix like `Redirecting to a Page`_


I wonder if we can make a Sphinx extension that automatically ads each section heading as a global ref, prefixed by pagename or similar? That way you wouldn't need to declare all the refs, but still be able to reference them. That's definitely for another ticket/PR though.

humitos · 2018-05-24T16:40:39Z

Goooood work! :)

stsewd commented Feb 16, 2018

View reviewed changes

stsewd mentioned this pull request Feb 17, 2018

Add rstcheck for linting rst files on travis #3622

Closed

humitos reviewed Feb 20, 2018

View reviewed changes

stsewd force-pushed the add-rstcheck-to-ci branch from 1886d28 to 98cd3ac Compare February 21, 2018 17:17

humitos reviewed Feb 21, 2018

View reviewed changes

stsewd commented Feb 21, 2018

View reviewed changes

humitos reviewed Feb 21, 2018

View reviewed changes

.travis.yml Outdated

@@ -7,6 +7,8 @@ matrix:

include:

- python: 2.7

env: TOXENV=docs

- python: 2.7

Copy link

Member

humitos Feb 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Py3? :)

RichardLitt added the PR: work in progress Pull request is not ready for full review label Feb 22, 2018

stsewd force-pushed the add-rstcheck-to-ci branch from 5b70dc9 to 68d9cd1 Compare February 24, 2018 23:20

stsewd mentioned this pull request Feb 28, 2018

Fixes internal reference #3695

Closed

stsewd force-pushed the add-rstcheck-to-ci branch from 68d9cd1 to efa5e7b Compare March 17, 2018 19:15

stsewd commented Mar 17, 2018

View reviewed changes

stsewd mentioned this pull request Mar 18, 2018

Add travis-ci style pull request builder #1340

Closed

humitos approved these changes Mar 19, 2018

View reviewed changes

humitos reviewed Mar 23, 2018

View reviewed changes

humitos requested changes Mar 23, 2018

View reviewed changes

stsewd added 21 commits May 22, 2018 23:30

Lint docs with python 3.6

2ee1311

Check files with recursive option

318d3fa

Ignore false positive error

3c35393

Remove unused global ref

ed91b6d

Remove unused links

87e92e5

Remove unused link

b1affec

Change links to rst

5ed816b

Remove unused link

d153df3

Remove unused links

6d4a28a

Fix internal links

7aafded

Fix internal links

f5b6189

Remove unused links

5a67cfa

Remove unused link

65914a3

Fix internal links

8683ab1

Fix internal link

ea76877

Ignore false positives

ffa1179

Fix lists

994be93

Use doc directive

eac2029

Fix reference

646c2f8

Fix python dictionary on docs

0e25f29

Add description to tox env

d6292bc

stsewd force-pushed the add-rstcheck-to-ci branch from fbceb5f to d6292bc Compare May 23, 2018 04:33

Remove trailing comma

51f7c7c

ericholscher approved these changes May 24, 2018

View reviewed changes

ericholscher merged commit ea0c473 into readthedocs:master May 24, 2018

stsewd deleted the add-rstcheck-to-ci branch May 24, 2018 14:51

Add rstcheck to CI #3624

Add rstcheck to CI #3624

Conversation

stsewd commented Feb 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Feb 16, 2018 • edited Loading

stsewd commented Feb 16, 2018

ericholscher commented Feb 18, 2018

stsewd commented Feb 18, 2018

humitos commented Feb 20, 2018

stsewd commented Feb 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Mar 17, 2018

stsewd commented Mar 17, 2018

stsewd commented Mar 17, 2018

stsewd commented Mar 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd Mar 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Mar 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

stsewd commented Mar 23, 2018 • edited Loading

stsewd commented May 23, 2018

stsewd commented May 23, 2018 • edited Loading

ericholscher commented May 24, 2018

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented May 24, 2018

stsewd commented Feb 16, 2018 •

edited

Loading

stsewd commented Feb 16, 2018 •

edited

Loading

stsewd Mar 23, 2018 •

edited

Loading

stsewd commented Mar 23, 2018 •

edited

Loading

stsewd commented May 23, 2018 •

edited

Loading