IndentationError while checking doctest's in the sphinx docs

[skirpichev]

Steps to reproduce:

1. install the Diofant from sources
2. install flake8-quotes
3. run flake8-rst docs/tutorial/polys.rst

I got the following traceback:

Traceback (most recent call last):
  File "/home/sk/venv/dev/bin/flake8-rst", line 10, in <module>
    sys.exit(main())
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8_rst/cli.py", line 16, in main
    app.run(argv)
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/main/application.py", line 393, in run
    self._run(argv)
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/main/application.py", line 381, in _run
    self.run_checks()
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/main/application.py", line 300, in run_checks
    self.file_checker_manager.run()
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/checker.py", line 331, in run
    self.run_serial()
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/checker.py", line 315, in run_serial
    checker.run_checks()
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/checker.py", line 598, in run_checks
    self.run_ast_checks()
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/checker.py", line 502, in run_ast_checks
    for (line_number, offset, text, check) in runner:
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8_quotes/__init__.py", line 169, in run
    noqa_line_numbers = self.get_noqa_lines(file_contents)
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8_quotes/__init__.py", line 177, in get_noqa_lines
    tokens = [Token(t) for t in tokenize.generate_tokens(lambda L=iter(file_contents): next(L))]
  File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8_quotes/__init__.py", line 177, in <listcomp>
    tokens = [Token(t) for t in tokenize.generate_tokens(lambda L=iter(file_contents): next(L))]
  File "/usr/lib/python3.7/tokenize.py", line 572, in _tokenize
    ("<tokenize>", lnum, pos, line))
  File "<tokenize>", line 118
    ⎛ 2    ⎞
    ^
IndentationError: unindent does not match any outer indentation level

[twolfson]

I don't see flake8-rst or flake8 in the steps to reproduce. Where do those come in? Also, have you verified that flake8-quotes and Diofant are required?

…

On Wed, Mar 25, 2020, 1:16 AM Sergey B Kirpichev ***@***.***> wrote: Steps to reproduce: 1. install the Diofant <https://github.com/diofant/diofant/> from sources <https://diofant.readthedocs.io/en/latest/install.html#from-sources> 2. install flake8-quotes 3. run flake8-rst docs/tutorial/polys.rst I got the following traceback: Traceback (most recent call last): File "/home/sk/venv/dev/bin/flake8-rst", line 10, in <module> sys.exit(main()) File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8_rst/cli.py", line 16, in main app.run(argv) File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/main/application.py", line 393, in run self._run(argv) File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/main/application.py", line 381, in _run self.run_checks() File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/main/application.py", line 300, in run_checks self.file_checker_manager.run() File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/checker.py", line 331, in run self.run_serial() File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/checker.py", line 315, in run_serial checker.run_checks() File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/checker.py", line 598, in run_checks self.run_ast_checks() File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8/checker.py", line 502, in run_ast_checks for (line_number, offset, text, check) in runner: File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8_quotes/__init__.py", line 169, in run noqa_line_numbers = self.get_noqa_lines(file_contents) File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8_quotes/__init__.py", line 177, in get_noqa_lines tokens = [Token(t) for t in tokenize.generate_tokens(lambda L=iter(file_contents): next(L))] File "/home/sk/venv/dev/lib/python3.7/site-packages/flake8_quotes/__init__.py", line 177, in <listcomp> tokens = [Token(t) for t in tokenize.generate_tokens(lambda L=iter(file_contents): next(L))] File "/usr/lib/python3.7/tokenize.py", line 572, in _tokenize ("<tokenize>", lnum, pos, line)) File "<tokenize>", line 118 ⎛ 2 ⎞ ^ IndentationError: unindent does not match any outer indentation level — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#92>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG4KWA77BNPJZUY2UTDSTDRJG4WNANCNFSM4LTISFCQ> .

[skirpichev]

I don't see flake8-rst or flake8 in the steps to reproduce.

It's among of the development requirements, its installation was documented in the documentation, which quoted in the step 1.

Also, have you verified that flake8-quotes and Diofant are required?

I doubt. But I'm figuring out the minimal example. In the case above the error was triggered by the following fragment from rst:

Square-free factorization
=========================

The square-free factorization of a univariate polynomial is the product of all
factors (not necessarily irreducible) of degree 1, 2 etc

    >>> sqf(2*x**2 + 5*x**3 + 4*x**4 + x**5)
                    2
            ⎛ 2    ⎞
    (x + 2)⋅⎝x  + x⎠

[twolfson]

That looks like a bug in flake8-rst or whatever is passing that code block to flake8-quotes. The code block isn't valid Python, it's a Python REPL yes but not valid Python

It looks like flake8-rst expects a python declaration (e.g. sourcecode:: python) so I'm not sure where the issue is

https://pypi.org/project/flake8-rst/

Closing as invalid

[skirpichev]

@twolfson, please reconsider. It seems, for whatever reason - flake8-quotes plugin parses the file as whole, not something flake8/flake8-rst feed to it.

Unfortunately, flake8 can't disable plugins. Maybe flake8-quotes should be disabled by default if it's in such an early alpha stage? That would be a workaround for my problem, as it would be possible to avoid this plugin while testing doctests.

[skirpichev]

BTW, I'm pretty sure - flake8-rst is ok in this example. Here is the output from the RstFileChecker.__init__() in the above example.

>>> pprint.pprint(source_block.source_lines)
[(6,
  'init_printing(pretty_print=True, use_unicode=True)\n',
  '    >>> init_printing(pretty_print=True, use_unicode=True)\n'),
 (27,
  'f, g = 5*x**2 + 10*x + 3, 2*x + 2\n',
  '    >>> f, g = 5*x**2 + 10*x + 3, 2*x + 2\n'),
 (29, 'div(f, g)\n', '    >>> div(f, g)\n'),
...
 (116,
  'sqf(2*x**2 + 5*x**3 + 4*x**4 + x**5)\n',
  '    >>> sqf(2*x**2 + 5*x**3 + 4*x**4 + x**5)\n'),
...

As you can see, @twolfson, the sqf-related block (which was the source of trouble) - was parsed correctly by flake8-rst. But flake8-quotes just ignores that info.

[twolfson]

I believe you have correctly found the source of the issue as described in #93. We use the pep8 infrastructure and expect files to either be entire files or passed in via stdin. I'm unsure there's actually a viable fix within the flake8 infrastructure to handle this (or we likely would have used it)

Going to leave a similar message in #93. Feel free to explore/submit a PR for it but I feel what flake8-rst is doing is likely pushing the limits of flake8 design intentions