Difference in highlighting some TypeScript React files (Sublime vs. Syntect)

[slimsag]

Syntect fails to properly syntax highlight some TypeScript React (tsx) files in contrast to Sublime Text 3 which highlights them fine.

To reproduce this:

1. You'll need to first download Microsoft's official TypeScriptReact.tmLanguage file and save it somewhere.
2. Convert that .tmLanguage file to a .sublime-syntax file, using Sublime Text 3's builtin Plugin Development: Convert Syntax to .sublime-syntax command (important: only shows up when viewing .tmLanguage files)
3. Save the new .sublime-syntax file into e.g. testdata/Packages/TypeScript/TypeScriptReact.sublime-syntax
4. Run make assets.

Alternatively, my fork's branch here has the above steps done so you don't have to perform them. The .sublime-syntax file is also in the root of the repo.

Once you've done the above or cloned my fork,

1. Save this BasePicker.tsx test file anywhere.
2. Save TypeScriptReact.sublime-syntax into e.g. ~/Library/Application\ Support/Sublime\ Text\ 3/Packages/User/ on Mac (and uninstall the TypeScript Sublime plugin, to prevent any concern over the tmLanguage -> sublime-syntax conversion potentially being the issue).

When the file is viewed in Sublime, it looks like this:

In contrast, the HTML rendered by cargo run --example synhtml ./BasePicker.tsx > out.html ends up looking like this:

Of course, the above two look different due to the theme in use, but notice that even just in the syntect result alone there are two different colors in use for the attributes:

I found that changing this line of code to remove the addition operator:

                suggestedDisplayValue={ suggestedDisplayValue }
-               aria-activedescendant={ 'sug-' + this.suggestionStore.currentIndex }
+               aria-activedescendant={ this.suggestionStore.currentIndex }
                aria-owns='suggestion-list'

Fixes the highlighting in Syntect:

If it wasn't for Sublime rendering this file with the same sublime-syntax, I would assume the syntax file was broken somehow -- but given the difference between Sublime and Syntect's rendering, I assume this must be a bug in Syntect (but I do not understand Syntect well enough to even begin tracking it down :) )

[trishume]

Interesting, I wonder if this is related to the ASP syntax tests still failing. I never got around to fixing all of those. See #51.

[robinst]

(For anyone wondering, v2.0.0 still has this issue, just tested.)

[robinst]

Was wondering whether #146 changes this, it doesn't.

But what I noticed is that cargo run --example syncat doesn't have the same problem, only synhtml. I wonder what they do differently, it might give a clue as to what the problem is. But it also might just be a bug in syncat or synhtml :)..

[trishume]

@robinst huh weird, yah probably a bug in synhtml.

[trishume]

Okay I diagnosed this hoping to fix it before the next release, unfortunately it's tricky.

Basically synhtml uses the nonewlines mode, which uses hacky regex rewriting, which is incomplete. The \s metacharacter matches newlines and we don't do anything about it.

That causes this rule to fail to match on the \s+ in this regex:

  jsx-tag-in-expression:
    - match: |-
        (?x)
        (?<=[({\[,?=>:*]|&&|\|\||\?|\Wreturn|^return|\Wdefault|^)\s*
        (?!<\s*[_$[:alpha:]][_$[:alnum:]]*((\s+extends\s+[^=>])|,)) # look ahead is not type parameter of arrow
        (?=(<)\s*
        ([_$a-zA-Z][-$\w.]*(?<!\.|-))
        (?=\s+(?!\?)|/?>))

A minimal test case file that differs in parsing (in parsing on the first line, in highlighting with base16-ocean-dark only on the second) is:

<div
ref="hi"

For now the workaround is just to use newlines mode. It's just better.

[robinst]

Ah nice, that makes sense. Should we also make synhtml use newlines mode by default?

[trishume]

@robinst probably, and maybe in the mean time change the docs to suggest that nonewlines is known to be mostly working but a little bit broken and point to this issue.

I wonder if eventually a mode could be added to fancy-regex to hallucinate a \n at the end of strings, although that may run into the problem of regex not supporting it.

[Enselic]

A similar problem still exists in latest syntect master with the latest TypeScriptReact.tmLanguage:

cargo run --example syncat -- ~/Downloads/BasePicker.tsx --no-newlines

However, as before, not using nonewlines behaves correct:

cargo run --example syncat -- ~/Downloads/BasePicker.tsx

And by now there is clear documentation that avoiding nonewlines is recommended.

So I think we can go ahead and close this issue. Feel free to reopen if you disagree, of course.