Skip to content

lexer/parser: JSX text tokens not emitted in .tsx (JSX-vs-generic ambiguity) #70

Description

@ericsssan

Summary

Follow-up to #61 (fixed for .jsx in PR #69). JSX child text in .tsx files is still tokenized as ordinary JS tokens — the single-jsx_text-token fix is gated to plain JSX.

Why .tsx was excluded

The lexer-side fix (#69) tracks JSX element nesting to know when it is in text context. In TSX, a <T> is genuinely ambiguous between:

  • a JSX element <T>…</T>,
  • generic type arguments f<T>(…),
  • a generic arrow <T,>() => ….

Only the parser's speculative parse (try generic, backtrack to JSX) can disambiguate. Committing to "this < opens a JSX body" in the context-free lexer regressed 14 tsx/ts conformance cases (tsxGenericArrowFunctionParsing, tsxTypeArgumentsJsxPreserveOutput, <Foo<T>>, …). So #69 gates element-body tracking off for TS (jsx_text_mode = is_jsx and !is_ts), leaving the TSX token stream as it was.

Direction

The parser already disambiguates correctly and builds jsx_text_node AST nodes with the right ranges for both jsx and tsx. The likely fix is parser-driven: a post-lex token-stream rebuild that rewrites each jsx_text_node range into a single jsx_text token (approach "C" from the #61 scoping). This works for jsx + tsx and sidesteps the lexer ambiguity. It could either layer on top of the #69 lexer fix (jsx via lexer, tsx via rebuild) or replace it (one mechanism for both).

Impact

Same as #61 (indent and any JSXText-anchored rule), but for .tsx files specifically.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions