How to deal with new language constructs?

[MHumm]

How do we handle new language constructs?
I mean if a tool wants to support several Delphi version and wants to use the grammaer from here.
Maintain different Delphi versions in branches?
Or is there some notation possible within the grammar?

[darnocian]

If a tool is used, it should generate code for the lowest version of delphi we want to support.

Antlr has semantic predicates that can be used in the gramar rules to indicate when paths can be followed based on some context (https://github.com/antlr/antlr4/blob/master/doc/predicates.md). Possibly, this could be a way of controlling features for a given version of delphi. e.g. say the context was set to delphi8, and the predicates for any newer features would evaluate to false and thus not followed. I'll try create a mini example to validate.

[MHumm]

Thanks for that idea already, and yes: do testing/exploring it is a good idea.

[darnocian]

I made a little test with Antlr. Here is a snippet of a mini grammar I created using a predicate rule that I got working. I added a predicate rule around the inline_var in the stmt rule. Using a static variable, I could toggle the feature being supported or not.

`grammar ProtoPas;

program : 'program' ID ';' vars body '.';

body: 'begin' stmts 'end' ;

stmts : stmt ';' stmts
| stmt
;

stmt : assign
| {Delphi.supportsInlineVar}? inline_var
|
;

inline_var: 'var' ID inline_type inline_assign ;

inline_type : ':' ID
|
;

inline_assign : ':=' expr
|
;

vars : ( 'var' ( var )+ )?
;

var : ID ':' ID ';'
;

assign : ID ':=' expr;

expr: ID
| NUM
;`

I created a small config class:
`public class Delphi {

public static boolean supportsInlineVar = false;

}`

I could use Delphi.supportsInlineVar as a control variable to configure the behaviour of the parser.

So we just define a procedure setVersion(double version) where it sets Delphi.supportsInlineVar := version >= 10.3;

Hope this makes sense. I'll commit this tomorrow.

[darnocian]

I've committed the example project https://github.com/darnocian/ProtoPas which illustrates the predicate rule

[chuacw]

inline_assign is optional, and may be present, or not.
Your grammar doesn't allow the possibility of it being absent.

[MHumm]

Is this a basic problem with the grammar system used or is this a problem how it has been used and it can be changed to reflect that?

[darnocian]

@chuacw you may have missed the point of the 'protopas' example above. It was devised to illustrate the use of antlr predicates, which comes into play with the test rule:
stmt : assign
| {Delphi.supportsInlineVar}? inline_var
|
;

the example is not reflective of a correct delphi grammar.

The point of predicates in a system like antlr, is that you can essentially control the rules that are available at runtime. So the question we had was how to deal with new language constructs, and I was trying to illustrate that 'predicate' type functionality can be a way to say enable features in a grammar based on a particular version.

Of course, this feature is only useful if the wider group likes something like antlr, otherwise, the example above is not relevant.

[darnocian]

btw. If you look at the 'stmt' rule:
stmt : assign
| {Delphi.supportsInlineVar}? inline_var
|
;

The above means the following stmt rule can match:

• the assign rule
• the inline_var rule (provide supportsInlineVar is true)
• nothing

if supportsInlineVar is false, the stmt rule only matches:

• assign
• nothing

[darnocian]

So with predicates, to control language features in rules, you could do things 2 ways :

• have feature flags as illustrated, and have some sort of SetVersion(version) method that toggles feature flags. e.g. supportsInlineVar := version >10.2 (or whatever)
• have predicate that could be something like version102andAbove... but I prefer the first option.

Anyways, that was my 10 cents on the topic.