Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changing token.type is invalid #265

Open
mageniao2 opened this issue Apr 30, 2024 · 1 comment
Open

Changing token.type is invalid #265

mageniao2 opened this issue Apr 30, 2024 · 1 comment
Labels

Comments

@mageniao2
Copy link

mageniao2 commented Apr 30, 2024

I added action for LETTERS in the lexer file, which adjusts the token type to VALUE_FUNCTION_NAME (1), but it doesn't take effect, and the output in antlr4-debug is still LETTERS, not VALUE_FUNCTION_NAME.

Tokens:
[@0,0:3='show',<3>,1:0]
[@1,4:4='1',<4>,1:4]
[@2,5:4='<EOF>',<-1>,1:5]

But the syntax tree I get through antlr4-listener outputs the token as VALUE_FUNCTION_NAME
Maybe I got something wrong? My code is as follows:

  • launch.json
{
    "version": "2.0.0",
    "configurations": [
        {
            "name": "Debug ANTLR4 grammar",
            "type": "antlr-debug",
            "request": "launch",
            "input": "example.txt",
            "grammar": "lang/exampleParser.g4",
            "actionFile": "lang/example1.js",
            "startRule": "start",
            "printParseTree": true,
            "visualParseTree": true
        }
    ]
}
  • lexer and parser
lexer grammar exampleLexer;
tokens {
   VALUE_FUNCTION_NAME, DECORATE_FUNCTION_NAME
}
LETTERS : [a-zA-Z]+ { resetBuiltInToken(this) };
DIGITS : [0-9];
WS : [ \t]+ -> channel(HIDDEN);
parser grammar exampleParser;
options {
    tokenVocab=exampleLexer;
}
start : letters digits?;
letters : (LETTERS | VALUE_FUNCTION_NAME| DECORATE_FUNCTION_NAME);
digits : DIGITS+;
  • exampleLexer.tokens
VALUE_FUNCTION_NAME=1
DECORATE_FUNCTION_NAME=2
LETTERS=3
DIGITS=4
WS=5
  • lang/example1.js
function resetBuiltInToken(token) {
        console.log(token + "asdasdasdasds")
        token.type = 1;
    }
@mike-lischke
Copy link
Owner

mike-lischke commented May 2, 2024

Is the resetBuiltInToken function actually called? And the this binding is wrong. That's not the token, but the lexer. If all you want is to change the token type use the type lexer command:

LETTERS : [a-zA-Z]+ -> type(VALUE_FUNCTION_NAME);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants