Wrong syntax highlighting for latex?

[colasbd]

It seems that there is an issue with this library.

For the text

\begin{document}
This is a test
$test$
\end{document}

\begin{document}
$x+y$
\end{document}

and the theme Rainbow_Dash, I get the colours

These are not (I guess) the correct colours.

The .tmLanguage file that I use is given here.

More precisely, here is the NSAttributedString I get:

\documentclass{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 0.172549 0.364706 0.803922 1";
    caret = "#000000";
    fontStyle = bold;
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}{article}
{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 0.301961 0.301961 0.301961 1";
    caret = "#000000";
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}\begin{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 0.501961 0 1";
    caret = "#000000";
    fontStyle = bold;
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}{{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 0.301961 0.301961 0.301961 1";
    caret = "#000000";
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}document{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 0.301961 0.301961 0.301961 1";
    caret = "#000000";
    fontStyle = italic;
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}}
Ceci est mon test?
$test$
{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 0.301961 0.301961 0.301961 1";
    caret = "#000000";
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}\end{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 0.501961 0 1";
    caret = "#000000";
    fontStyle = bold;
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}{{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 0.301961 0.301961 0.301961 1";
    caret = "#000000";
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}document{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 0.301961 0.301961 0.301961 1";
    caret = "#000000";
    fontStyle = italic;
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}}{
    NSBackgroundColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 1 1 1 1";
    NSColor = "NSCustomColorSpace sRGB IEC61966-2.1 colorspace 0.301961 0.301961 0.301961 1";
    caret = "#000000";
    invisibles = "#BFBFBF";
    lineHighlight = "#00000012";
    selection = "#4D97FF54";
}

[alehed]

Can you test it with the reference implementation (textmate)?

[colasbd]

Do you mean textmate uses SyntaxKit?
Here is the result:

[alehed]

No, but it implements highlighting with TmLanguage files correctly. In theory SyntaxKit tries to emulate TextMate's highlighting (there are small differences). Just add the rainbow dash theme to textmate and post the screenshot. Also does SyntaxKit emit any messages?

[colasbd]

No message from SyntaxKit in the console.
I can't find how to add a theme to text mate.
(adding it to /Application Support/textmate/Themes does not work)

[alehed]

http://manual.macromates.com/en/bundles#installing_a_bundle

[colasbd]

OK, I've installed the Twilight theme in my project.
Here is the result:

left: textmate
right: SyntaxKit

[alehed]

I'll test it. Looks like there is a problem with math and the article.

[colasbd]

Yep. Thanks.
Which class(es) will you look at? So that I can cast an eye too ;-)

[alehed]

First I will take a look at the grammar (you have to eliminate \G). Then you just have to do a manual parsing pass and compare it to what the parse does. You can familiarize yourself with the parsing alogrithm if you want to.

[colasbd]

Thanks @alehed .
I don't have time to dive into this part of the code. Do you plan looking at it?

[alehed]

I'll take the code for a spin in the next few days.

[alehed]

For me the expression in $$ is highlighted just fine. All you have to do is make sure that it can also find the tex grammar (the latex grammar includes the tex grammar). I still have to figure out why the article is not colored.

[colasbd]

So I have to include the tex grammar as a resource? No need to load it?

Thanks

[alehed]

Just put the tex.tmlanguage in a place where your callback will find it (probably next to the latex.tmlanguage). When SyntaxKit sees the include, it tries to include the language grammar automatically.

[colasbd]

About the article part maybe it is related to another grammar, for document classes le something. I remember that in the latex.tmbundle there were one file like that.

There is no latex.tmLanguage file in this bundle but only a latex.plist

I guess if I add to the resources the .tmbundle it will not find the .plist file as a grammar, won't it?

[alehed]

It doesn't matter what extension, just add the TeX.plist to the same place as you already have the LaTeX.plist and make sure your function finds it there (give it the correct name).

[colasbd]

Ok 👌
I have seen in your code that the resource should have extension .tmLanguage, no ?

[alehed]

The reason I don't like plist is because it can mean anything (xml, json based or binary). In the code you gave me it looks specifically for a .tmLanguage though. The following code would look for a .plist:

let bundleManager = BundleManager { (identifier, kind) -> (URL?) in
    let name = kind == .Language ? identifier._split(separator: ".")[1] : identifier
    return bundle.url(forResource: name.capitalized, withExtension: ".plist") ?? URL(fileURLWithPath: "")
}

[colasbd]

Thanks 👍

[alehed]

I think this issue can be closed. The newest version produces the following output for the first test (it does need to find the tex.tmLanguage):

Currently the only difference between SyntaxKit and TextMate is that the article in \document class{article} is not highlighted, but according to the tex grammar at hand it should not be highlighted. So TextMate seems to do some magic behind the scenes.

[colasbd]

Well done!