Antlr4 (target Python 3) generated files and mypy (>= 0.91) type-checking issues

[DagSonntag]

Hi
I have been using mypy together with Antlr4 python generated code for some time. This has mainly worked since I have been able to exclude any Antlr4 generated files from the mypy-check since they have never been fully compliant. However, lately, from mypy 0.91 forward, some errors are raised even for files that are excluded in case it affects files dependent on the excluded files. This, in combination with that the Antlr generated files are massive, creates an issue since the errors from those files bloats the log of any mypy type-checking, making it hard to see the errors in ones own code.

This is certainly an issue for mypy, but I am also wondering if it is desirable to make Antrl4 python generated code more mypy (PEP) typing-friendly?
The main issues I have found are (examples from the Java/Java grammar):

• The usage of * imports.
  - In for example JavaParser the first import statement is from antlr4 import *. This is something mypy does not like, and results in a lot of 'name-defined' errors like JavaParser.py:11145: error: Name "ParseTreeListener" is not defined [name-defined]
  - Could be resolved by explicitly importing the classes instead, replacing the *

(Edit: Two additional issues were also raised previously, but are now moved to separate issues (#4188 #4189 ))

So, my question is. Would these fixes be desirable or have there been some discussions to what level/PEP the generated files should be follow? (i.e. Is this seen as an issue or will provided fixes be rejected)

Last of all; Thank you for a great tool! It is most useful! :)

[ericvergnaud]

Hi, thanks for this. I welcome efforts to improve the usability of generated code if it’s not tied to a particular tool. There are 3 topics here, and maybe they would be better dressed in 3 separate conversations, except the 1st one. Re usage of * for imports, with the current tool’s infrastructure, it is impossible to know exactly which set of classes need to be imported at the time the import statement is generated, because it depends on the grammar. As an example based on your own proposals, import Optional from Typing would only be necessary if the grammar has a repeating rule (such as id+ or id*). Bad news we don’t know if that’s the case when generating the import, and if it’s not you’ll get a warning for an unused import. So the strategy here would rather be to disable PEP import related warnings as part of generating the code (if that’s possible) Would you mind opening separate threads for the 2 other topics ? Eric

…

Le 15 mars 2023 à 15:07, Dag Sonntag ***@***.***> a écrit : Hi I have been using mypy together with Antlr4 python generated code for some time. This has mainly worked since I have been able to exclude any Antlr4 generated files from the mypy-check since they have never been fully compliant. However, lately, from mypy 0.91 forward, some errors are raised even for files that are excluded in case it affects files dependent on the excluded files. This, in combination with that the Antlr generated files are massive, creates an issue since the errors from those files bloats the log of any mypy type-checking, making it hard to see the errors in ones own code. This is certainly an issue for mypy, but I am also wondering if it is desirable to make Antrl4 python generated code more mypy (PEP) typing-friendly? The main issues I have found are (examples from the Java/Java grammar): The usage of * imports. - In for example JavaParser the first import statement is from antlr4 import *. This is something mypy does not like, and results in a lot of 'name-defined' errors like JavaParser.py:11145: error: Name "ParseTreeListener" is not defined [name-defined] - Could be resolved by explicitly importing the classes instead, replacing the * The usage of implicit optional - Implicit optional is used all over the place in the method declarations in the Antlr generated files with for example: def variableModifier(self, i:int=None):. This is not allowed since pep484 and while one can turn off this check in mypy, it seems like an easy fix to just add Optional everywhere (i.e. def variableModifier(self, i:Optional[int]=None): with an extra import of Optional from Typing). Slots seems to be misused in the Antrl generated files. - Examples are JavaParser.py:10217: error: Trying to assign name "bop" that is not in "slots" of type "JavaParser.JavaParser.ExpressionContext" [misc] - Would probably require more work, but these cases should be possible to find and add the appropriate values in slots So, my question is. Would these fixes be desirable or have there been some discussions to what level/PEP the generated files should be follow? (i.e. Is this seen as an issue or will provided fixes be rejected) Last of all; Thank you for a great tool! It is most useful! :) — Reply to this email directly, view it on GitHub <#4183>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNQJGKRGBGS7YXZRWQC5LW4HEKJANCNFSM6AAAAAAV337HUE>. You are receiving this because you are subscribed to this thread.

[DagSonntag]

Hi
Thank you for the fast response.
Hmm, ok, I haven't looked into the code yet, but there would seem to me like there are two ways to do it

• Either have every generated method return what imports it is using
• Look through the document for certain keywords (imports) at the end

Then do a post-edit of the first lines of the generated document/file. You don't think any of those strategies are possible with the current code generation flow? Or do the lines in the file have to be generated in order?

[ericvergnaud]

Hi,
Sorry for the late reply
I think both of these would be very costly in code changes, and thus risky.