Line/column location is wrong with JSX

[moroshko]

Trying to parse the following code using loc: true, jsx: true:

function App() {
  return (
    <Container>
      <Text>Hello</Text>
    </Container>
  );
}

gives an unexpected location for <Text>:

"loc": {
  "start": {
    "line": 3,
    "column": 23
  },
  "end": {
    "line": 3,
    "column": 27
  }
}

I expect it to start on line 4.

Am I missing something?

[KFlash]

Obviously an bug!

cc/ @3cp Do you have any idea for an quick solution?

[3cp]

Not just Text, Container closing tag is wrong too.
It looks like the jsx parser missed all the line feeds to reset line and column. Do you know where is the problematic lines?

[KFlash]

@3cp I think it happened around here

[3cp]

It looks like scanJSXToken is called twice to scan the same source location (index 3).

The duplicated scan is probably a bug.

The first scan picked up two new lines correctly, the second scan missed the first new line, it strangely thinks the char at index 3 is a < instead of \n. I guess this comes as a side effect of the duplicated scan.

export function scanJSXToken(parser: ParserState): Token {
  parser.startPos = parser.tokenPos = parser.index;
  parser.startColumn = parser.colPos = parser.column;
  parser.startLine = parser.linePos = parser.line;

  if (parser.index >= parser.end) return (parser.token = Token.EOF);

  const token = TokenLookup[parser.source.charCodeAt(parser.index)];
  console.log('index: ' + parser.index);
  switch (token) {
    // '<'
    case Token.LessThan: {
      advanceChar(parser);
      if (parser.currentChar === Chars.Slash) {
        advanceChar(parser);
        parser.token = Token.JSXClose;
      } else {
        parser.token = Token.LessThan;
      }

      break;
    }
    // '{'
    case Token.LeftBrace: {
      advanceChar(parser);
      parser.token = Token.LeftBrace;
      break;
    }
    default:
      let state = LexerState.None;

      while (parser.index < parser.end) {
        const type = CharTypes[parser.currentChar];
        console.log('char ' + JSON.stringify(String.fromCharCode(parser.currentChar)));

        if (type & CharFlags.CarriageReturn) {
          state |= LexerState.NewLine | LexerState.LastIsCR;
          scanNewLine(parser);
          console.log('scanNewLine');
        } else if (type & CharFlags.LineFeed) {
          console.log('consumeLineFeed');
          consumeLineFeed(parser, state);
          state = (state & ~LexerState.LastIsCR) | LexerState.NewLine;
        } else {
          advanceChar(parser);
        }

        if (CharTypes[parser.currentChar] & CharFlags.JSXToken) break;
      }

      parser.tokenValue = parser.source.slice(parser.tokenPos, parser.index);
      parser.token = Token.JSXText;
      console.log('tokenValue: ' + JSON.stringify(parser.tokenValue));
  }

  return parser.token;
}

> const {parse} = require('./dist/meriyah.cjs.js');
undefined
> 
> const code = `<p>
... hello
... </p>
... `;
undefined
> 
> const ast = parse(code, {
...   loc: true, jsx: true
... });
index: 3
char "\n"
scanNewLine
char "h"
char "e"
char "l"
char "l"
char "o"
char "\n"
scanNewLine
tokenValue: "\nhello\n"
index: 3
char "<"
char "h"
char "e"
char "l"
char "l"
char "o"
char "\n"
scanNewLine
tokenValue: "\nhello\n"
index: 10
undefined
> 

[KFlash]

Any ideas on how to solve it?

[3cp]

No idea now. The src/parser.ts file is bit deep.

[KFlash]

Far as I know jsx is separated from rest of the code in parser.ts. So I guess only 5 methods related to jsx. Everyone have jsx in the func name

[3cp]

I will debug it when I got more time.

[3cp]

I didn't fix the duplicated scan, but figured out how to bypass the wrong currentChar.