LP Syntax Highlighting

Syntax highlighting

This page is a guide to implement the syntax highlighter for a new esolang. It is expected that you've followed the steps in the Getting Started page and can open the Esolang Park page for your new esolang.

Esolang Park uses the open-source Monaco editor as the code editor. Monaco in turn uses a library called Monarch for syntax highlighting. Monarch allows you to specify the syntax highlighting specifics of your language as the rules of a pushdown automata.

Before you close this tab and delete your repository fork, you don't need to understand pushdown automata. Depending on your esolang, you may not even need to read most of this page. But in any case, pushdown automata are really just a computer science term for processing the input left-to-right while maintaining a stack and following rules of the current state. And Monarch does the heavy lifting - we just need to make the rules.

The common basics

Cropping out the details, syntax highlighting in editors works in the following way:

• Editors have a notion of tokens - types of "things" in source code. Some examples of token names are variable, operator and constant.
• Each language's syntax highlighter is responsible for defining the rules - what kind of text is a variable or operator or constant. This is usually done by making the rules as mentioned above.
• Editor color themes provide the mapping between tokens and colors. In essence, the color of a variable token and other tokens is provided by the color theme.

We need to do the second part - making the rules about what text corresponds to what token type. To see a list of tokens currently supported in Esolang Park, refer to the rules field in any of the theme files. It is a rather small list, and given the weirdness of esolangs, the token names often make no sense actually. You'll need the token names for the below steps, though.

For simple languages

If the syntax of your esolang is exceptionally simple, you won't need to read the other sections at all. Consider languages like Brainfuck, Deadfish and Befunge. These languages have no notion of syntactic scopes or complex nesting - it's just "assume , is a keyword and + is an operator". For such languages, you just need to define a mapping between regular expressions and corresponding token names.

For instance, here's the syntax highlighter for Deadfish, in all its entirety:

export const editorTokensProvider: MonacoTokensProvider = {
  tokenizer: {
    root: [
      [/i/, "identifier"],
      [/d/, "variable"],
      [/s/, "meta"],
      [/o/, "tag"],
    ],
  },
  defaultToken: "comment",
};

That is it. i is tokenized (and colored) as an identifier, and similar for the other three operators. Characters that don't match anything are tokenized as comment. Note that the token names aren't supposed to make sense here - there are no identifiers, variables, meta or tags in Deadfish - and are picked pretty much randomly (my personal preference may have played a role).

If your esolang falls in this category, define the rules similar to the above example and check the results. Remember that the rules are followed in array order, so you may need to reorder rules accordingly if there's some overlap.

The Monarch state machine

If the above doesn't cut it, we need some statefulness. In this example, we'll slightly dip our faces into the capabilities of Monarch in stateful syntax highlighting. But before that, let's understand how the Monarch tokenizer works.

As mentioned at the start, the Monarch tokenizer works on the model of a pushdown automata. It parses the input from start to end, and at each point tries to match the "remaining" input on the rules we specify. Whenever a rule matches, the matched text is tokenized according to the rule and Monarch then skips ahead to the rest of the text. If no rule matches, the character is tokenized as specified in the defaultToken field and skipped past.

Monarch also maintains a stack of states. The topmost state of the stack is considered the current state of the Monarch tokenizer, and only the rules for this state are followed at any point. There are special operations to push and pop states on the stack.

To illustrate this, consider a variant of Deadfish that supports the /** ... */ comment syntax. Note that the idso characters inside a comment should not be highlighted. Here's the token provider for such a language:

export const editorTokensProvider: MonacoTokensProvider = {
  tokenizer: {
    root: [
      [/\/\*\*/, { token: "comment", next: "myComment" }],
      [/i/, "identifier"],
      [/d/, "variable"],
      [/s/, "meta"],
      [/o/, "tag"],
    ],
    myComment: [
      [/\*\//, { token: "comment", next: "@pop" }],
      [/[^\*]+/, "comment"],
    ],
  },
  defaultToken: "comment",
};

Here's a description of the changes:

1. A new field myComment is added. Each field of the tokenizer represents a possible state of the Monarch tokenizer. The myComment state represents the state of the tokenizer when it's inside a comment.

2. A new rule is added to the root state. The regex matches the comment-opening syntax /**, and pushes the myComment state onto the Monarch tokenizer's state stack. After this match, the tokenizer's current state (the topmost state of the stack) becomes myComment and it will now only follow the rules of that state.

3. Two rules are added in the myComment state. The first rule matches the comment-closing syntax */ and pops the topmost state in the stack (myComment), taking the tokenizer back to root state. The second rule matches everything other than a *.

Note how Deadfish idso operations inside comments won't be highlighted - those rules aren't followed when the tokenizer is inside a comment. This is a basic example of how you can implement stateful syntax highlighters.

Cool - what else?

Monarch is quite powerful - the Monarch page contains much more than what I can write in this wiki, so make sure to go to the page and scroll down to read the official documentation. It's not super-user-friendly for someone new to syntax highlighting or pushdown automata, but with the above example it should be easier to follow. Trying things out while reading the doc will certainly help.

For more examples, refer to the Chef and Shakespeare syntax highlighters which use states to highlight different sections of the code accordingly. If it still seems too difficult, feel free to skip to the other components of the language provider and return to this part later on. You don't need to make a perfect syntax highlighter - it's probably fine as long as it covers most cases.