Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Annoying warnings with CSharp3 target #152

Open
ericvergnaud opened this issue Feb 20, 2014 · 27 comments
Open

Annoying warnings with CSharp3 target #152

ericvergnaud opened this issue Feb 20, 2014 · 27 comments

Comments

@ericvergnaud
Copy link

Hi,

following our conversations, I've moved to a build where a generic grammar is used to produce language specific grammars, and lexers/parsers are produced programmatically using org.antlr.Tool v 3.5.1.

All is fine except I get annoying warnings when processing grammars for CSharp3 target, such as:
warning(24): template error: context [/outputFile /lexer /lexerRule /lexerRuleLabelDefs] 14:1 no such property or can't access: org.antlr.tool.Rule.charListLabels
Caused by: org.stringtemplate.v4.misc.STNoSuchPropertyException: no such property: org.antlr.tool.Rule.charListLabels
at org.stringtemplate.v4.misc.ObjectModelAdaptor.throwNoSuchProperty(ObjectModelAdaptor.java:106)
at org.stringtemplate.v4.misc.ObjectModelAdaptor.lookupMethod(ObjectModelAdaptor.java:99)
at org.stringtemplate.v4.misc.ObjectModelAdaptor.getProperty(ObjectModelAdaptor.java:67)
at org.stringtemplate.v4.Interpreter.getObjectProperty(Interpreter.java:1158)
at org.stringtemplate.v4.Interpreter._exec(Interpreter.java:210)
at org.stringtemplate.v4.Interpreter.exec(Interpreter.java:145)
at org.stringtemplate.v4.Interpreter.writeObject(Interpreter.java:703)
at org.stringtemplate.v4.Interpreter.writeObjectNoOptions(Interpreter.java:635)
at org.stringtemplate.v4.Interpreter._exec(Interpreter.java:285)
at org.stringtemplate.v4.Interpreter.exec(Interpreter.java:145)
at org.stringtemplate.v4.Interpreter.writeObject(Interpreter.java:703)
at org.stringtemplate.v4.Interpreter.writeIterator(Interpreter.java:733)
at org.stringtemplate.v4.Interpreter.writeObject(Interpreter.java:708)
at org.stringtemplate.v4.Interpreter.writeObjectWithOptions(Interpreter.java:665)
at org.stringtemplate.v4.Interpreter._exec(Interpreter.java:292)
at org.stringtemplate.v4.Interpreter.exec(Interpreter.java:145)
at org.stringtemplate.v4.Interpreter.writeObject(Interpreter.java:703)
at org.stringtemplate.v4.Interpreter.writeObjectNoOptions(Interpreter.java:635)
at org.stringtemplate.v4.Interpreter._exec(Interpreter.java:285)
at org.stringtemplate.v4.Interpreter.exec(Interpreter.java:145)
at org.stringtemplate.v4.ST.write(ST.java:415)
at org.antlr.codegen.CodeGenerator.write(CodeGenerator.java:1296)
at org.antlr.codegen.Target.genRecognizerFile(Target.java:98)
at org.antlr.codegen.CSharp3Target.genRecognizerFile(CSharp3Target.java:109)
at org.antlr.codegen.CodeGenerator.genRecognizer(CodeGenerator.java:459)
at org.antlr.Tool.generateRecognizer(Tool.java:674)
at org.antlr.Tool.process(Tool.java:487)
at presto.generator.ParserProducer.produce(ParserProducer.java:27)
at presto.generator.Application.main(Application.java:36)
Caused by: java.lang.NoSuchFieldException: charListLabels
at java.lang.Class.getField(Class.java:1537)
at org.stringtemplate.v4.misc.ObjectModelAdaptor.lookupMethod(ObjectModelAdaptor.java:88)
... 27 more

warning(24): template error: context [/outputFile /parser /genericParser /rule /returnScope] 1:4 no such property or can't access: null.attributes

Is there something I can do about it ?

Kind regards,

Eric

@sharwell
Copy link
Member

The Java tool cannot be used to generate code for the CSharp2 or CSharp3 targets. You'll need to use Antlr3.exe from that distribution to generate the necessary code.

@ericvergnaud
Copy link
Author

Hi Sam,

thanks for the rapid feedback.

Eric

Le 21 févr. 2014 à 02:04, Sam Harwell notifications@github.com a écrit :

The Java tool cannot be used to generate code for the CSharp2 or CSharp3 targets. You'll need to use Antlr3.exe from that distribution to generate the necessary code.


Reply to this email directly or view it on GitHub.

@ericvergnaud
Copy link
Author

Hi Sam,

btw, is that also true for AntLR4 ?

Eric

Le 21 févr. 2014 à 02:04, Sam Harwell notifications@github.com a écrit :

The Java tool cannot be used to generate code for the CSharp2 or CSharp3 targets. You'll need to use Antlr3.exe from that distribution to generate the necessary code.


Reply to this email directly or view it on GitHub.

@sharwell
Copy link
Member

Yes, for ANTLR 4 you should use the instructions on the following page:
https://github.com/sharwell/antlr4cs

@ericvergnaud
Copy link
Author

Hi Sam,

just by curiosity, if the java tool cannot generate CSharp, why is it required for CSharp generation ?

Eric

Le 22 févr. 2014 à 11:47, Sam Harwell notifications@github.com a écrit :

Yes, for ANTLR 4 you should use the instructions on the following page:
https://github.com/sharwell/antlr4cs


Reply to this email directly or view it on GitHub.

@sharwell
Copy link
Member

The Java library that's included in the C# target distribution is not the same as the library that's released with the reference Java version. It uses a modified Tool class as the entry point and includes a number of custom templates.

This custom library is capable of generating C# code via command line calls, but use outside of the MSBuild integration is not supported and highly discouraged.

@ericvergnaud
Copy link
Author

Hi Sam,

thanks for the clarification.
It's a bit of a pity, since this prevents one from automating the grammar generation for multiple language targets.

Kind regards,

Eric

Le 23 févr. 2014 à 08:32, Sam Harwell notifications@github.com a écrit :

The Java library that's included in the C# target distribution is not the same as the library that's released with the reference Java version. It uses a modified Tool class as the entry point and includes a number of custom templates.


Reply to this email directly or view it on GitHub.

@sharwell
Copy link
Member

Hi Eric,

The build system for different types of projects (e.g. MSBuild, Ant, or Maven) each has its own method of generating parsers during the build process. In each case, we have produced a reliable implementation to help ensure that your compiled binaries are successfully built according to the grammar file(s) you are using in the project. Command line generation which is performed separately from the compilation process cannot provide these same guarantees. Faced with the choice of supporting a reliable build strategy or an unreliable build strategy, I see no incentive to ever pursue the latter. If you are currently generating your parsers via IDE features or manually on the command line, you should switch ASAP to using a more robust method.

@ericvergnaud
Copy link
Author

Hi Sam,

honestly, I'm not sure only one build strategy is valid. It depends a lot on your organization and requirements.
In my case, I need to generate parsers for multiple languages: Java, C#, Python, JavaScript for now.
Given AntLR's current limitations (I'm not complaining here) I ended up creating a multi-language root grammar, from which I generate the per language grammar.
Ensuring that the parser generation is successful for all languages each time I modify the root grammar is key to consistency and quality.
I believe this approach would become even more relevant once I switch to AntLR 4 (which Terence advertised as being more suitable for multi language parsers).
Thanks to the interface callback model, I believe AntLR4 lets you compile the code and link it with the runtime as a standalone object.
This could be done from a single grammar using the -t option.
So I foresee in AntLR4 the ability to build jars/assemblies which contain only the parsing code, and have no dependency on production code.

So indeed building the parser as part of building the executable is a valid build strategy.
But it might not be the only one...

Eric

Le 24 févr. 2014 à 05:58, Sam Harwell notifications@github.com a écrit :

Hi Eric,

The build system for different types of projects (e.g. MSBuild, Ant, or Maven) each has its own method of generating parsers during the build process. In each case, we have produced a reliable implementation to help ensure that your compiled binaries are successfully built according to the grammar file(s) you are using in the project. Command line generation which is performed separately from the compilation process cannot provide these same guarantees. Faced with the choice of supporting a reliable build strategy or an unreliable build strategy, I see no incentive to ever pursue the latter. If you are currently generating your parsers via IDE features or manually on the command line, you should switch ASAP to using a more robust method.


Reply to this email directly or view it on GitHub.

@parrt
Copy link
Member

parrt commented Feb 24, 2014

Hi guys, I'm not sure I understand what's going on here. The tool itself, at least the Java version, should be able to generate different targets without modifying the grammar. Naturally we only have java and C# at this point.
Ter
On Feb 23, 2014, at 4:54 PM, ericvergnaud wrote:

Hi Sam,

honestly, I'm not sure only one build strategy is valid. It depends a lot on your organization and requirements.
In my case, I need to generate parsers for multiple languages: Java, C#, Python, JavaScript for now.
Given AntLR's current limitations (I'm not complaining here) I ended up creating a multi-language root grammar, from which I generate the per language grammar.
Ensuring that the parser generation is successful for all languages each time I modify the root grammar is key to consistency and quality.
I believe this approach would become even more relevant once I switch to AntLR 4 (which Terence advertised as being more suitable for multi language parsers).
Thanks to the interface callback model, I believe AntLR4 lets you compile the code and link it with the runtime as a standalone object.
This could be done from a single grammar using the -t option.
So I foresee in AntLR4 the ability to build jars/assemblies which contain only the parsing code, and have no dependency on production code.

So indeed building the parser as part of building the executable is a valid build strategy.
But it might not be the only one...

Eric

Le 24 févr. 2014 à 05:58, Sam Harwell notifications@github.com a écrit :

Hi Eric,

The build system for different types of projects (e.g. MSBuild, Ant, or Maven) each has its own method of generating parsers during the build process. In each case, we have produced a reliable implementation to help ensure that your compiled binaries are successfully built according to the grammar file(s) you are using in the project. Command line generation which is performed separately from the compilation process cannot provide these same guarantees. Faced with the choice of supporting a reliable build strategy or an unreliable build strategy, I see no incentive to ever pursue the latter. If you are currently generating your parsers via IDE features or manually on the command line, you should switch ASAP to using a more robust method.


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub.

@ericvergnaud
Copy link
Author

Hi Terence, hi Sam

unfortunately this does not seem to be the case:

ANTLR Tool v4.1 (/var/folders/7z/rdw_873x3_s5zr8pb81k9z7c0000gn/T/antlr-4.1-complete.jar)
ArgsLexer.g4 -o /Users/ericvergnaud/Development/Presto/GrammarV4/Grammar/generated/antlr4/presto/utils -listener -no-visitor -package presto.utils -encoding UTF-8
error(31): ANTLR cannot generate 'org.antlr.v4.codegen.CSharpTarget' code as of version 4.1
Exception in thread "main" java.lang.NullPointerException
at org.antlr.v4.codegen.CodeGenerator.getTemplates(CodeGenerator.java:96)
at org.antlr.v4.automata.LexerATNFactory.(LexerATNFactory.java:69)
at org.antlr.v4.Tool.processNonCombinedGrammar(Tool.java:405)
at org.antlr.v4.Tool.process(Tool.java:384)
at org.antlr.v4.Tool.processGrammarsOnCommandLine(Tool.java:343)
at org.antlr.v4.Tool.main(Tool.java:190)
1 error(s)

I'll be happy to help.

Kind regards,

Eric

Le 25 févr. 2014 à 01:27, Terence Parr notifications@github.com a écrit :

Hi guys, I'm not sure I understand what's going on here. The tool itself, at least the Java version, should be able to generate different targets without modifying the grammar. Naturally we only have java and C# at this point.
Ter
On Feb 23, 2014, at 4:54 PM, ericvergnaud wrote:

Hi Sam,

honestly, I'm not sure only one build strategy is valid. It depends a lot on your organization and requirements.
In my case, I need to generate parsers for multiple languages: Java, C#, Python, JavaScript for now.
Given AntLR's current limitations (I'm not complaining here) I ended up creating a multi-language root grammar, from which I generate the per language grammar.
Ensuring that the parser generation is successful for all languages each time I modify the root grammar is key to consistency and quality.
I believe this approach would become even more relevant once I switch to AntLR 4 (which Terence advertised as being more suitable for multi language parsers).
Thanks to the interface callback model, I believe AntLR4 lets you compile the code and link it with the runtime as a standalone object.
This could be done from a single grammar using the -t option.
So I foresee in AntLR4 the ability to build jars/assemblies which contain only the parsing code, and have no dependency on production code.

So indeed building the parser as part of building the executable is a valid build strategy.
But it might not be the only one...

Eric

Le 24 févr. 2014 à 05:58, Sam Harwell notifications@github.com a écrit :

Hi Eric,

The build system for different types of projects (e.g. MSBuild, Ant, or Maven) each has its own method of generating parsers during the build process. In each case, we have produced a reliable implementation to help ensure that your compiled binaries are successfully built according to the grammar file(s) you are using in the project. Command line generation which is performed separately from the compilation process cannot provide these same guarantees. Faced with the choice of supporting a reliable build strategy or an unreliable build strategy, I see no incentive to ever pursue the latter. If you are currently generating your parsers via IDE features or manually on the command line, you should switch ASAP to using a more robust method.


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub.

@sharwell
Copy link
Member

The reference version of ANTLR 4 only contains the templates and output driver for the Java version. The other language targets are now independently maintained and released, to avoid getting into the situation we are in with ANTLR 3 where it's unclear what state each of the targets is in or when they were last updated.

@ericvergnaud
Copy link
Author

mmm
I understand, but this means that if I have 4 targets, I need 4 versions of the tool which may not support the same grammar or have different bugs...
isn't this approach not just pushing the mess to the users ?
Eric

Le 25 févr. 2014 à 19:02, Sam Harwell notifications@github.com a écrit :

The reference version of ANTLR 4 only contains the templates and output driver for the Java version. The other language targets are now independently maintained and released, to avoid getting into the situation we are in with ANTLR 3 where it's unclear what state each of the targets is in or when they were last updated.


Reply to this email directly or view it on GitHub.

@sharwell
Copy link
Member

The language targets are impossible to maintain centrally. Ter and I do not know all of the targets well enough, nor have the time (or money) to ensure that they all stay up to date. The only practical long-term approach is to shift this inevitable risk onto the exceedingly small group of users who are using multiple targets simultaneously. In addition to this group being small, it tends to also be teams using ANTLR as part of a commercially-developed project, allowing additional resources (expertise, time, money) to ensure that the problems associated with independently-developed targets do not hinder their progress.

@ericvergnaud
Copy link
Author

Sam,

maybe I can help in this matter:

  • I have a requirement
  • I can work on: Java, CSharp, Python, Javascript, C++, Objective-C

But this only makes sense if you and Terence have appetite...

Eric

Le 25 févr. 2014 à 20:18, Sam Harwell notifications@github.com a écrit :

The language targets are impossible to maintain centrally. Ter and I do not know all of the targets well enough, nor have the time (or money) to ensure that they all stay up to date. The only practical long-term approach is to shift this inevitable risk onto the exceedingly small group of users who are using multiple targets simultaneously. In addition to this group being small, it tends to also be teams using ANTLR as part of a commercially-developed project, allowing additional resources (expertise, time, money) to ensure that the problems associated with independently-developed targets do not hinder their progress.


Reply to this email directly or view it on GitHub.

@sharwell
Copy link
Member

You would have to send the proposal. As it currently stands, I do not see any problem in need of a solution (aside from the fact that ANTLR 4 doesn't currently have targets for Python, Javascript, or Objective-C).

@parrt
Copy link
Member

parrt commented Feb 25, 2014

Our intention is definitely support multiple targets, which is why we have the -D option to set language. I have vague memories of intending to copy templates from the C# repo to a distribution that could at least generate C# code, though of course the libraries would be maintained separately. It is definitely a goal to allow ANTLR to generate multiple targets from one grammar.

I think the issue here is that you reasonably want to use the same tool. Having to run the Java tool and the C#-based tool to generate different output is contrary to our original intention. I'm not sure how to fix this but it is important.

Sam: it should be a simple matter of copying templates from the C# repo when we create a distribution I suppose. Actually, there is some logic that has to go in there, a subclass of Target, right? How can we solve this problem? We will definitely have the same issue with C++. There will definitely not be a C++ implementation of the tool itself so we have to solve this problem.

@parrt parrt reopened this Feb 25, 2014
@sharwell
Copy link
Member

Specifically regarding the C# target: The build system is standardized (unlike for Java), and the integration is fast, reliable, follows established conventions, and works well within the IDE. In addition, the distribution mechanism for this target is now standardized, so there is no motivation for providing downloads of the target through another source. I have no intention of ever supporting command line builds for this target, as it would only serve to take away from what has long been available through better means.

The situations above may not apply to other targets, so you can evaluate them as necessary.

@parrt
Copy link
Member

parrt commented Feb 25, 2014

I understand completely that the best way to use C# will be to use your complete tool. That said, we have C++ target coming, which is in its own repository. How will we integrate that target? Once we solve that, we could provide a C# target drawing from your repository that would let people generate C# code for use with mono on mac/linux, for example.

@sharwell
Copy link
Member

The ideal distribution would involve a separate .jar file containing the CPPTarget class and the templates for the C++ distribution. You could then include this jar in the classpath when you run the tool, allowing the -D option to locate the appropriate files.

The distribution mechanism I referred to for C# works on all platforms now, so the justification really does apply to any standard C# project whether Microsoft .NET-based or Mono-based.

@parrt
Copy link
Member

parrt commented Feb 25, 2014

I like what you are suggesting. Sounds like something we would might of been talking about. Ok, so you would not support a jar that would hook into the standard distribution for generating C#?

@whitten
Copy link

whitten commented Feb 25, 2014

If I am going to support generating a different language target (MUMPS or
Cache) what is the naming convention, and what Antlr4code would be most
effective to read and understand?

David Whitten
713-870-3834

PS: I expect only a small group of people would ever use this target, but
I'd like to have it on github, probably in a repository of mine since I
can't imagine enough interest to take Sam or Ter's attention, but with
sufficient hooks that it could be maintained long term.

On Tue, Feb 25, 2014 at 12:46 PM, Terence Parr notifications@github.comwrote:

I like what you are suggesting. Sounds like something we would might of
been talking about. Ok, so you would not support a jar that would hook into
the standard distribution for generating C#?

Reply to this email directly or view it on GitHubhttps://github.com//issues/152#issuecomment-36036230
.

@parrt
Copy link
Member

parrt commented Feb 25, 2014

We have no doc I'm afraid. The real issue is that all of the complex analysis is now done at runtime, which means that a language target developer must be able to debug that code as they develop it. Honestly, it's extremely complicated and requires a great deal of parsing knowledge.

@whitten
Copy link

whitten commented Feb 25, 2014

If you must debug and understand the complex runtime analysis code in
Antlr4 to be able to generate the target language code, would it be a
better path to just re-implement Antlr4 in my target language directly,
and from that understanding create the target code ?

David Whitten
713-870-3834

On Tue, Feb 25, 2014 at 1:08 PM, Terence Parr notifications@github.comwrote:

We have no doc I'm afraid. The real issue is that all of the complex
analysis is now done at runtime, which means that a language target
developer must be able to debug that code as they develop it. Honestly,
it's extremely complicated and requires a great deal of parsing knowledge.

Reply to this email directly or view it on GitHubhttps://github.com//issues/152#issuecomment-36038692
.

@parrt
Copy link
Member

parrt commented Feb 25, 2014

Implementing the tool itself won't really help you understand the runtime, I'm afraid. Plus, it would be a lot of work. Here is a link to the technical report on the underlying parsing strategy: http://www.antlr.org/papers/allstar-techreport.pdf It will give you an idea of what you'd be in for. It's the combination of years of research.

@whitten
Copy link

whitten commented Feb 25, 2014

Would a native (target language) version of the StringTemplate library be
needed by the run-time analysis code? Would that be a good starting point ?

On Tue, Feb 25, 2014 at 1:31 PM, Terence Parr notifications@github.comwrote:

Implementing the tool itself won't really help you understand the runtime,
I'm afraid. Plus, it would be a lot of work. Here is a link to the
technical report on the underlying parsing strategy:
http://www.antlr.org/papers/allstar-techreport.pdf It will give you an
idea of what you'd be in for. It's the combination of years of research.

Reply to this email directly or view it on GitHubhttps://github.com//issues/152#issuecomment-36041104
.

@parrt
Copy link
Member

parrt commented Feb 26, 2014

On Feb 25, 2014, at 2:02 PM, David Whitten wrote:

Would a native (target language) version of the StringTemplate library be
needed by the run-time analysis code? Would that be a good starting point ?

nope. The tool is in Java and uses Java ST to gen code :)
Ter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants