Propagate column numbers from parser to AST #36

andrewschaaf opened this Issue Jan 9, 2011 · 3 comments

3 participants


No description provided.


Parser doesn't know them. This needs an overhaul of lexer and/or token structure.



@josher19 josher19 pushed a commit to josher19/LiveScript that referenced this issue Jun 21, 2012
@gkz gkz fixed ops as functions for instanceof, import, in, their negation, etc.
eg. (not in [1 to 5]) 7 #=> true - closes #35, closes #36

Some research: outputs tokens in the form [TAG, value, lineNumber], which is converted to jison variables in by

parser import
  yy    : require \./ast
  lexer :
    lex           : -> [tag, @yytext, @yylineno] = @tokens[++@pos] or ['']; tag
    setInput      : -> @pos = -1; @tokens = it
    upcomingInput : -> '' passes yylineno to certain rules with the L function:

  o \ID            -> Chain L Var $1

compiled to:

case 1:this.$ = yy.Chain(yy.L(yylineno, yy.Var($$[$0])));

where L is defined in

exports.L = (yylineno, node) -> node import line: yylineno + 1

The default lexer for jison keeps track of column location in yylloc, per the bison spec. See also zaach/jison#59

Thus, for column numbers, the following needs to be done:

  1. needs to keep track of column numbers for each token, e.g. [TAG, value, lineNo, colStart, colEnd]
  2. needs to set yyloc properly from the lexer's token format.
  3. needs to pass yyloc into the AST nodes it creates somehow, perhaps implicitly as part of the grammar DSL.

Those steps will propogate column numbers far enough to allow carp compile-time errors to have column information.

For source maps, more work would have to be done, since coco AST compiles straight to javascript code as strings. If coco AST instead compiled to JS AST, decorated with source line and column information, then the source map generation task could be offloaded to a separate code generator, such as escodegen. Factoring out the JS code generation could also make optimization/cleanup/shaping of the compiled code easier, such as #115.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment