Parsing fails with Python 3.5 matrix multiply infix operator

[dgolden1]

pydocstyle fails when parsing files that use the Python 3.5 matrix multiply infix operator @.

Steps to reproduce

Create file blah.py:

import numpy as np
def blah_bad():
    """Blah blah bad"""

    a = np.zeros(1)
    b = np.zeros(1)
    x = a @ b

Run pydocstyle on blah.py:

pydocstyle blah.py

Output:

Traceback (most recent call last):
  File "pydocstyle.py", line 1725, in <module>
    main()
  File "pydocstyle.py", line 1715, in main
    sys.exit(run_pydocstyle(use_pep257))
  File "pydocstyle.py", line 1330, in run_pydocstyle
    errors.extend(check((filename,), select=checked_codes))
  File "pydocstyle.py", line 1267, in check
    for error in PEP257Checker().check_source(source, filename):
  File "pydocstyle.py", line 1368, in check_source
    module = parse(StringIO(source), filename)
  File "pydocstyle.py", line 275, in __call__
    return self.parse_module()
  File "pydocstyle.py", line 428, in parse_module
    children = list(self.parse_definitions(Module, all=True))
  File "pydocstyle.py", line 366, in parse_definitions
    yield self.parse_definition(class_._nest(self.current.value))
  File "pydocstyle.py", line 485, in parse_definition
    class_.__name__, name, self.current.kind,
AttributeError: 'NoneType' object has no attribute 'kind'

Source of problem

The problem is in pydocstyle.Parser#parse_definitions where @ is assumed to exclusively be part of a decorator in the lines:

            elif self.current.kind == tk.OP and self.current.value == '@':
                self.consume(tk.OP)
                self.parse_decorators()

I'm not sufficiently intimately familiar with the valid Python syntax to suggest a fix, but you could presumably solve the problem by assuming that any @ characters encountered while parsing an expression are matrix multiply operators if they're not the first non-whitespace character of the expression.

[sigmavirus24]

you could presumably solve the problem by assuming that any @ characters encountered while parsing an expression are matrix multiply operators if they're not the first non-whitespace character of the expression.

What about

new_matrix = (matrix
              @ another_matrix)

[dgolden1]

The @ is the first character of the line there, but not the first character of the expression. The fix should look at the first character of the expression.

[Nurdok]

Thanks for reporting this.

Since I started maintaining this project, I wanted to replace the underlying parsing engine to something more robust and, mainly, 3rd party. Parsing Python code is no small undertaking and I feel that each bugfix is just putting out fires instead of dealing with the underlying problem.
So I'm now working on replacing the manual parsing of Python code that is currently implemented in pydocstyle, which should solve this bug, and others with it.

My current direction is to use pylint's astroid. I'm playing around with it and I hope that a big PR will be made in the upcoming weeks :)

[ceridwen]

I and @PCManticore have been doing some major refactoring of astroid, partly to make it better to use as a library for projects that aren't pylint. You can see what we have in progress in the 2.0 branch of the main astroid repository and here, https://github.com/PCManticore/astroid. Eventually, astroid will split into two libraries, one with all the static analysis code and one with the code for the ASTs. I haven't had much time to work on that this summer, but I expect at some point to be picking it up and finishing it. If you have questions or need help, I'm sure one of us would be willing to assist you.

[Nurdok]

@ceridwen that sounds good. Right now I'm still refactoring and adding more tests so that it'll be an easier transition to replace the underlying engine, but I will definitely take you up on that help once I'll start to integrate with astroid.

[Nurdok]

Short update: using astroid did not pan out (we need a Full Syntax Tree, not an Abstract one) and neither did redbaron (due to performance issues). However, I was pretty successful in using the jedi parser. I'm still working on this, but I'm 90% there (all tests pass with the new jedi-based parser). This should also fix this issue. Will update again soon.