New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing reStructuredText markup in docstrings #1221

Closed
peterjc opened this Issue May 8, 2017 · 9 comments

Comments

Projects
None yet
2 participants
@peterjc
Member

peterjc commented May 8, 2017

This is a tracking issue for fixing the reStructuredText (RST) markup violations in our docstrings.

@peterjc

This comment has been minimized.

Show comment
Hide comment
@peterjc

peterjc May 8, 2017

Member

Background

One year ago in May 2016 we switched from using epydoc to reStructuredText as the default markup language in the docstrings as rendered into help pages using epydoc:

biopython/biopython.github.io@06548be

While this runs, the output is in many cases less than perfect - and will hamper long term goals to replace using epydoc with something else like Sphinx, see #906.

Member

peterjc commented May 8, 2017

Background

One year ago in May 2016 we switched from using epydoc to reStructuredText as the default markup language in the docstrings as rendered into help pages using epydoc:

biopython/biopython.github.io@06548be

While this runs, the output is in many cases less than perfect - and will hamper long term goals to replace using epydoc with something else like Sphinx, see #906.

@peterjc

This comment has been minimized.

Show comment
Hide comment
@peterjc

peterjc May 8, 2017

Member

Linting / Validating

Right now there is no off the shelf tool that I am aware of for reStructuredText (RST) validation of docstrings, despite RST being the proposed as the preferred markup language for docstrings in PEP287 https://www.python.org/dev/peps/pep-0287/

I have written a proof of principle pull-request to pydocstring which can be used for this, calling docutils to validate the RST: PyCQA/pydocstyle#254

The bad news is this identifies hundreds of problematic docstrings in Biopython.

Member

peterjc commented May 8, 2017

Linting / Validating

Right now there is no off the shelf tool that I am aware of for reStructuredText (RST) validation of docstrings, despite RST being the proposed as the preferred markup language for docstrings in PEP287 https://www.python.org/dev/peps/pep-0287/

I have written a proof of principle pull-request to pydocstring which can be used for this, calling docutils to validate the RST: PyCQA/pydocstyle#254

The bad news is this identifies hundreds of problematic docstrings in Biopython.

@souravsingh

This comment has been minimized.

Show comment
Hide comment
@souravsingh

souravsingh May 15, 2017

Contributor

I have run the doc8 on Doc/doc.rst as a preliminary check, Here is the output-

$ doc8 doc.rst
doc.rst:504: D004 Found literal carriage return
doc.rst:505: D004 Found literal carriage return
doc.rst:506: D004 Found literal carriage return
doc.rst:507: D004 Found literal carriage return
doc.rst:508: D004 Found literal carriage return
doc.rst:293: D001 Line too long
doc.rst:329: D001 Line too long
========
Total files scanned = 1
Total files ignored = 0
Total accumulated errors = 1036
Detailed error counts:
    - doc8.checks.CheckCarriageReturn = 508
    - doc8.checks.CheckIndentationNoTab = 0
    - doc8.checks.CheckMaxLineLength = 2
    - doc8.checks.CheckNewlineEndOfFile = 0
    - doc8.checks.CheckTrailingWhitespace = 508
    - doc8.checks.CheckValidity = 18
Contributor

souravsingh commented May 15, 2017

I have run the doc8 on Doc/doc.rst as a preliminary check, Here is the output-

$ doc8 doc.rst
doc.rst:504: D004 Found literal carriage return
doc.rst:505: D004 Found literal carriage return
doc.rst:506: D004 Found literal carriage return
doc.rst:507: D004 Found literal carriage return
doc.rst:508: D004 Found literal carriage return
doc.rst:293: D001 Line too long
doc.rst:329: D001 Line too long
========
Total files scanned = 1
Total files ignored = 0
Total accumulated errors = 1036
Detailed error counts:
    - doc8.checks.CheckCarriageReturn = 508
    - doc8.checks.CheckIndentationNoTab = 0
    - doc8.checks.CheckMaxLineLength = 2
    - doc8.checks.CheckNewlineEndOfFile = 0
    - doc8.checks.CheckTrailingWhitespace = 508
    - doc8.checks.CheckValidity = 18
@peterjc

This comment has been minimized.

Show comment
Hide comment
@peterjc

peterjc May 15, 2017

Member

@souravsingh this issue is about the docstrings inside Python files, not actual RST files. Please continue any notes about Doc/doc.rst that on new issue #1232.

Member

peterjc commented May 15, 2017

@souravsingh this issue is about the docstrings inside Python files, not actual RST files. Please continue any notes about Doc/doc.rst that on new issue #1232.

@peterjc

This comment has been minimized.

Show comment
Hide comment
@peterjc

peterjc Jun 8, 2017

Member

I wrote a proof of principle pull-request extending to pydocstring calling docutils to validate the docstrings as RST: PyCQA/pydocstyle#254

The pydocstyle team recommended implementing RST docstring validation as a separate flake8 plugin, which I have started at https://github.com/peterjc/flake8-rst-docstrings

Member

peterjc commented Jun 8, 2017

I wrote a proof of principle pull-request extending to pydocstring calling docutils to validate the docstrings as RST: PyCQA/pydocstyle#254

The pydocstyle team recommended implementing RST docstring validation as a separate flake8 plugin, which I have started at https://github.com/peterjc/flake8-rst-docstrings

@peterjc

This comment has been minimized.

Show comment
Hide comment
@peterjc

peterjc Jun 16, 2017

Member

With the proviso that https://github.com/peterjc/flake8-rst-docstrings is still a work in progress, I have used it to silence a lot of RST validation errors it reported from docutils - see commits leading up to and including 6ae8fa1 - mostly it wanting more blank lines in certain places.

Right now my flake8 plugin only reports some "information" level messages Possible title underline, too short for the title. for Bio/, and to my eye these look like false positives.

I am inclined to make an initial PyPI release of flake8-rst-docstrings and use it on the Biopython TravisCI style checks...

Member

peterjc commented Jun 16, 2017

With the proviso that https://github.com/peterjc/flake8-rst-docstrings is still a work in progress, I have used it to silence a lot of RST validation errors it reported from docutils - see commits leading up to and including 6ae8fa1 - mostly it wanting more blank lines in certain places.

Right now my flake8 plugin only reports some "information" level messages Possible title underline, too short for the title. for Bio/, and to my eye these look like false positives.

I am inclined to make an initial PyPI release of flake8-rst-docstrings and use it on the Biopython TravisCI style checks...

peterjc added a commit to peterjc/biopython that referenced this issue Jun 19, 2017

TravisCI/Tox: Use flake8-rst-docstrings
This will install and use my new flake8 plugin to validate
the Python code's docstring markup as reStructuredText (RST),
using docutils internally.

https://github.com/peterjc/flake8-rst-docstrings/blob/master/README.rst
https://pypi.python.org/pypi/flake8-rst-docstrings

Note I am currently assuming flake8-rst-docstrings v0.0.3 which
does not show the INFO level messages from docutils.

This and recent commits closes #1221 on fixing invalid RST in
our docstrings.
@peterjc

This comment has been minimized.

Show comment
Hide comment
@peterjc

peterjc Jun 19, 2017

Member

My flake8 plugin is now on PyPI, https://pypi.python.org/pypi/flake8-rst-docstrings

Currently it does not report the "INFO" level messages from docutils, which in the cases I checked all seemed to be false positives, meaning it is happy with the Biopython docstrings.

Member

peterjc commented Jun 19, 2017

My flake8 plugin is now on PyPI, https://pypi.python.org/pypi/flake8-rst-docstrings

Currently it does not report the "INFO" level messages from docutils, which in the cases I checked all seemed to be false positives, meaning it is happy with the Biopython docstrings.

@peterjc

This comment has been minimized.

Show comment
Hide comment
@peterjc

peterjc Jun 19, 2017

Member

I'm trying enabling this on our TravisCI/Tox style check - interestingly this failed on TravisCI under Python 2.7,

flake8 Bio/
Traceback (most recent call last):
  File "/home/travis/build/peterjc/biopython/.tox/style/bin/flake8", line 11, in <module>
    sys.exit(main())
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/main/cli.py", line 16, in main
    app.run(argv)
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/main/application.py", line 328, in run
    self._run(argv)
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/main/application.py", line 316, in _run
    self.run_checks()
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/main/application.py", line 246, in run_checks
    self.file_checker_manager.run()
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/checker.py", line 317, in run
    self.run_parallel()
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/checker.py", line 286, in run_parallel
    for ret in pool_map:
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/pool.py", line 287, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/pool.py", line 659, in next
    raise value
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 92: ordinal not in range(128)
ERROR: InvocationError: '/home/travis/build/peterjc/biopython/.tox/style/bin/flake8 Bio/'

Likely there is a non-ASCII character in there somewhere, and we've not declared an encoding.

I've been running flake8 locally under Python 3.6 so will seek to reproduce this...

Member

peterjc commented Jun 19, 2017

I'm trying enabling this on our TravisCI/Tox style check - interestingly this failed on TravisCI under Python 2.7,

flake8 Bio/
Traceback (most recent call last):
  File "/home/travis/build/peterjc/biopython/.tox/style/bin/flake8", line 11, in <module>
    sys.exit(main())
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/main/cli.py", line 16, in main
    app.run(argv)
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/main/application.py", line 328, in run
    self._run(argv)
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/main/application.py", line 316, in _run
    self.run_checks()
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/main/application.py", line 246, in run_checks
    self.file_checker_manager.run()
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/checker.py", line 317, in run
    self.run_parallel()
  File "/home/travis/build/peterjc/biopython/.tox/style/lib/python2.7/site-packages/flake8/checker.py", line 286, in run_parallel
    for ret in pool_map:
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/pool.py", line 287, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/pool.py", line 659, in next
    raise value
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 92: ordinal not in range(128)
ERROR: InvocationError: '/home/travis/build/peterjc/biopython/.tox/style/bin/flake8 Bio/'

Likely there is a non-ASCII character in there somewhere, and we've not declared an encoding.

I've been running flake8 locally under Python 3.6 so will seek to reproduce this...

@peterjc

This comment has been minimized.

Show comment
Hide comment
@peterjc

peterjc Feb 3, 2018

Member

I fixed the encoding issue a while ago, and our TravisCI flake8 setup has been using flake8-rst-docstrings successfully for a long time now. Belated closing this issue.

Member

peterjc commented Feb 3, 2018

I fixed the encoding issue a while ago, and our TravisCI flake8 setup has been using flake8-rst-docstrings successfully for a long time now. Belated closing this issue.

@peterjc peterjc closed this Feb 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment