= and +=

[ProjectMoon]

This almost assuredly a problem where I don't understand quite how to use the lexer. My lexer specifies "=" as '=' and "+=" as 'PLUSEQUAL'.

However, the parser will fail with stuff like:

x += 5;

It will say expecting an = sign instead. x = 5; works fine. I'm guessing this has something to do with precedence that I'm missing. I'm using the jscore example. I added a lexer to the file.

The grammar is here: https://gist.github.com/1159333

[zaach]

The lexer will return the first rule to match, so if "+" comes before
"+=", it will return the "+" token and then "=" as separate tokens.
Put the longer rules before the shorter ones to avoid that.

On Sat, Aug 20, 2011 at 12:53 PM, ProjectMoon
reply@reply.github.com
wrote:

This almost assuredly a problem where I don't understand quite how to use the lexer. My lexer specifies "=" as '=' and "+=" as 'PLUSEQUAL'.

However, the parser will fail with stuff like:

x += 5;

It will say expecting an = sign instead. x = 5; works fine. I'm guessing this has something to do with precedence that I'm missing. I'm using the jscore example. I added a lexer to the file.

The grammar is here: https://gist.github.com/1159333

Reply to this email directly or view it on GitHub:
#58

Zach Carter

[ProjectMoon]

I see. Is this in the documentation at all?

[ProjectMoon]

This has fixed the problem.

[zaach]

This is a good point, since flex will use the longest matching rule. I may expose an option to have it behave as flex does. Maybe in the future that behavior will be the default.

[neizod]

I agree that the lexical part should match the longest rule. Since this makes grouping the same attribute easier.

Flex

%%
i(sabella)?  { printf("noun"); }
is           { printf("verb"); }
s            { printf("alpb"); }
%%

Both i and isabella return noun, while is return verb.

Jison

("i"("sabella")?)   alert("noun")
("is")              alert("verb")
("s")               alert("alpb")

Both i and isabella still return noun, but is return noun and then alpb. To fix this, I need to forget the powerful regex and rewrite this code like:

("isabella")        alert("noun")
("is")              alert("verb")
("i")               alert("noun")
("s")               alert("alpb")

That's make the lexical part too heavy.

[ProjectMoon]

Maybe this issue should be reopened since people are commenting on it.

[zaach]

Yes, I'll leave this open until we have an option for longest rule matching.

[zaach]

There's now a "flex" mode where the lexer uses the rule with the logest match. Just put:

%options flex

in your lexer grammar before the rules section.

[neizod]

wow! thank you for improving this. :)