Problems syntax coloring TeX and LaTeX files

[edreamleo]

From this thread:

• .tex files imports create @language latex directives.
  [EKR]: The .tex and .latex extensions should be not be treated differently, which may mean adding or changing existing importers.

• The constructs \lae, \cbstart and others color as expected, but words after the closing \ are improperly colorized. This needs fixing, because the "closing" "\ " designates [a real?] macro, which ends with the blank (and expands to a blank), since latex macros gobble up any trailing blank. I have such "\ " (backslash followed by a blank) macros all over the text.

[dalcolmo]

"The .tex and .latex extensions should not be treated differently. tex and latex share the same syntax. Indeed, latex is tex with some additional macros.

[dalcolmo]

clarification to the "\ ": yes this is a real macro, which expands to a (non-reducible) blank.
Most latex macros start with a backslash and then consist only of letters, which means the macro ends before any special character, before any number and before any blank etc.

However every latex macro consumes a blank that may follow, which is why you have to use such a non-reducible blank after a macro, if you want a blank in the output.

I would say the rule for highlighting macros should be kept simple: any backslash followed by any sequence of (upper- or lowercase) ASCII letters should be a considered a macro.
Exceptions to this rule may be some single-character macros like "\\" and "\_", but I would not even bother highlighting them.

Alternatively one could highlight only the backslash and the first character after the backslash if the first character after the backslash is not an ASCII letter. This would cover all single-character macros.

Of course, with these simple rules one cannot know the difference between a valid macro and something which just resembles a macro, but is not defined - but there is no easy way to achieve that.

It is possible to define macros in latex, which defy these rules, but in that case it becomes impossible to decide which is a macro and which is not. Some macros are only valid inside certain environments, for example in math mode. There are some tech editors which try to scan all \newcommand, \def, ... , but that too is prone to fail because there are many ways to define a new macro in latex. It would require to scan all loaded packages too. Fortunately most user commands follow the simple rule I named above and I would just leave it at that.

What may be much more useful than a complicated syntax highlighter is highlighting the brace pair, parenthesis or brackets which are immediately before the cursor, as long as the cursor is there, not just flashing the pair shortly when typing a closing brace.

• Josef

[edreamleo]

@dalcolmo Thanks for these clarifications.

[largo84]

I agree with Josef that tex and latex syntax should work the same way. Possible solution would be to have only one @language directive (suggest tex) and only one mode file, tex.py. Implications of that solution:

• Change importers to only use the valid @language tex.
• Might cause confusion for users who already have @language latex if it goes away.
• Only one file to maintain, not two.

Is it possible to have Leo refer both @language tex and latex to one mode file?

[dalcolmo]

I would like this to be re-opened. Although pygments coloring does a decent job on latex code, I mix many different languages in the same Leo file, and find Leo's own syntax coloring better for languages other than latex. So, why not fixing the syntax highlighting for latex in Leo, and stick with that colorizer?

The coloring of tex/latex on Leo's own colorizer seems based on a rather incomplete command list. Such a list is always going to be incomplete, because latex allows to define new commands via various mechanisms. Therefore, either one allows the user to easily extend the list (globally, and per-project separately, since the valid command list may be different for each project), or one bases the coloring on a simple rule instead of a list of commands. TexStudio tries to figure out the command list, by scanning the code (also in loaded packages), but this does not work for me as I also \input several files which define (lots of) new commands, and TexStudio does not seem to follow these.

The simplest coloring of latex commands would be to color everything that starts with a backslash and ends with a non-alphabetic, non-ascii character or whitespace. That non-alphabetic, non-ASCII character termination character should NOT be highlighted any more (it does not belong to the command).

EKR: The following should be easy enough to do with regex-based rules.

A command will start with a backslash followed by either:

1. One or more characters in the set [a-zA-Z], or,
2. Exactly one non-alphabetic, non-ASCII character.
   This single character should be highlighted too, because this covers the single-character macros.

This would not cover all possible latex commands and this would highlight some stuff which isn't defined as a latex command (e.g. a misspelling), covering everything right would be an enormous task, so let's keep it simple.

All the other highlighting for tex/latex (e.g. comments, ...) should remain as it is.

[edreamleo]

@dalcolmo Ok. I'll put this on the list for 6.1

[edreamleo]

From the TeX FAQ: "Macro names are conventionally built from a \ followed by a sequence of letters, which may be upper or lower case (as in \TeX, mentioned above). They may also be any single character, which allows all sorts of oddities, [including] \space meaning “insert a space here”.

This seems clear enough. The point is that we can ignore "unconventional" macros.

[edreamleo]

@largo84 Yes, it would be possible to use a single file for both TeX and LaTeX. It would involve a special case in Leo's colorizer, but that's no big deal.

[edreamleo]

Completed at 5e467b6. The highlights:

• Both @language tex and @language latex use leo/modes/tex.py. This ignores the many special cases in latex.py. I'll retain leo/modes/latex.py in case this choice becomes controversial.
• Only slight changes were needed to tex.py. Similar changes could be added to latex.py.

Code changes:

• bjc.init_mode makes @language latex be equivalent to @language tex.
• The new jedit.match_tex_backslash easily handles the nuances of conventional latex macros.