Propagate column numbers from parser to AST #36

Open
andrewschaaf opened this Issue Jan 9, 2011 · 3 comments

3 participants

@andrewschaaf

No description provided.

@satyr
Owner

Parser doesn't know them. This needs an overhaul of lexer and/or token structure.

@andrewschaaf

Right.

@josher19 josher19 pushed a commit to josher19/LiveScript that referenced this issue Jun 21, 2012
@gkz gkz fixed ops as functions for instanceof, import, in, their negation, etc.
eg. (not in [1 to 5]) 7 #=> true - closes #35, closes #36
9d1be84
@qqueue

Some research:

lexer.co outputs tokens in the form [TAG, value, lineNumber], which is converted to jison variables in by coco.co:

parser import
  yy    : require \./ast
  lexer :
    lex           : -> [tag, @yytext, @yylineno] = @tokens[++@pos] or ['']; tag
    setInput      : -> @pos = -1; @tokens = it
    upcomingInput : -> ''

grammar.co passes yylineno to certain rules with the L function:

Chain:
  o \ID            -> Chain L Var $1

compiled to:

case 1:this.$ = yy.Chain(yy.L(yylineno, yy.Var($$[$0])));

where L is defined in ast.co:

exports.L = (yylineno, node) -> node import line: yylineno + 1

The default lexer for jison keeps track of column location in yylloc, per the bison spec. See also zaach/jison#59

Thus, for column numbers, the following needs to be done:

  1. lexer.co needs to keep track of column numbers for each token, e.g. [TAG, value, lineNo, colStart, colEnd]
  2. coco.co needs to set yyloc properly from the lexer's token format.
  3. grammar.co needs to pass yyloc into the AST nodes it creates somehow, perhaps implicitly as part of the grammar DSL.

Those steps will propogate column numbers far enough to allow carp compile-time errors to have column information.

For source maps, more work would have to be done, since coco AST compiles straight to javascript code as strings. If coco AST instead compiled to JS AST, decorated with source line and column information, then the source map generation task could be offloaded to a separate code generator, such as escodegen. Factoring out the JS code generation could also make optimization/cleanup/shaping of the compiled code easier, such as #115.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment