Getting type error "_SpecialForm" not callable for pyparsing 3.0.8

[aucampia]

When using pyparsing 3.0.8 I'm getting the following type errors on the master branch of rdflib.

$ .venv/bin/python3 -m mypy --show-error-context --show-error-codes
rdflib/plugins/sparql/parser.py:200: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:391: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:440: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:481: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:500: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:556: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:560: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:563: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:567: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:597: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:603: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:609: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:612: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:613: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:631: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:632: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:662: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:675: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:689: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:713: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:736: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1089: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1156: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1171: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1179: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1230: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1232: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1280: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1287: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1288: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1292: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1344: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1347: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1348: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1389: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1394: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1395: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1396: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1397: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1404: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1427: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1462: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1493: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1500: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/parser.py:1501: error: "_SpecialForm" not callable  [operator]
rdflib/plugins/sparql/results/tsvresults.py:44: error: "_SpecialForm" not callable  [operator]
Found 46 errors in 2 files (checked 284 source files)

Files in question are:

• https://github.com/RDFLib/rdflib/blob/1cba9d87c84b40252fcd7bed9d566839c77e078a/rdflib/plugins/sparql/parser.py
• https://github.com/RDFLib/rdflib/blob/1cba9d87c84b40252fcd7bed9d566839c77e078a/rdflib/plugins/sparql/results/tsvresults.py

Errors can be seen in: https://github.com/RDFLib/rdflib/runs/5962843175?check_suite_focus=true

I think the problem somehow relates to pyparsing.Optional being confused with typing.Optional, but I'm not sure about this, I tried to alias it to something else and tried to use it with package name and it did not help. Not sure why this is happening, but I will try figure out. Just recording this while I work on it.

[aucampia]

Using this as a workaround for now:

https://github.com/RDFLib/rdflib/blob/603b7be94d81c87e6851d1b20c8faf198d5f5ff2/setup.cfg#L39-L43

[mypy-pyparsing.*]
# This is here because of an upstream issue with pyparsing:
# https://github.com/pyparsing/pyparsing/issues/385
# Once the issue is fixed this should be removed.
follow_imports = skip

[ptmcg]

Not clear to me what changed in 3.0.8 to cause this. Optional was in 3.0.7 as well.

[aucampia]

Not clear to me what changed in 3.0.8 to cause this. Optional was in 3.0.7 as well.

3.0.8 added py.typed:

• Mark pyparsing as a typed package #364

Before this using pyparsing was basically the same as having:

[mypy-pyparsing.*]
# This is here because of an upstream issue with pyparsing:
# https://github.com/pyparsing/pyparsing/issues/385
# Once the issue is fixed this should be removed.
follow_imports = skip

And IMO it is an okay workaround.

I'm actually not sure my theory is entirely right also (i.e. mypy confusing pyparsing.Optional with typing.Optional), I digged around a bit with trying to make mypy pass for pyparsing, I can't see anything obvious that is causing this, so I'm a bit baffled also, time will tell what others see.

One option is to remove py.typed, but I really would like to see the issue resolved and py.typed remain, so this may be the catalyst for someone to figure it out.

[ptmcg]

I made this change in the rdflib sparql parser.py code:

from pyparsing import (
    Literal,
    Regex,
    Opt as Optional,  # <-- changed
    OneOrMore,
    ZeroOrMore,
    Forward,
    ParseException,
    Suppress,
    Combine,
    restOfLine,
    Group,
    ParseResults,
    delimitedList,
)

And that cleared things up with mypy. But this will only work with pyparsing 3.x, not 2.4.7.

[aucampia]

Thanks @ptmcg - I will see if I can do some conditional import maybe. I don't actually get why mypy gets confused about Optional, this may actually be a mypy bug and not a pyparsing bug.

[domdfcoding]

This issue is that typing.Optional is imported here:

pyparsing/pyparsing/exceptions.py

Line 5 in 4e627f2

from typing import Optional

and in __init__.py everything from exceptions is imported:

pyparsing/pyparsing/__init__.py

Line 138 in 4e627f2

from .exceptions import *

Even though pyparsing.core.Optional is imported further down on line 142 mypy gets confused as to which object has the name "Optional".

The fix is not to import typing.Optional under the "Optional" name. core.py imports it as "OptionalType".

[aucampia]

Just for posterity sake I think it is critical to note this is a mypy issue and not an issue with pyparsing, and #386 is really just a workaround to the mypy issue.

[jtprobst]

Thanks all for the investigation! Has this bug been reported to the Mypy maintainers yet?

[aucampia]

I somewhat recall seeing a similar bug against mypy before, but I can't say for sure if it was that similar, and I don't know about the issue. I would recommend asking on https://gitter.im/python/typing to see if someone knows, it would be good to link this to a specific issue, but otherwise someone should make one.