Parse, validate, and highlight code blocks

[xsebek]

• part of Highlight commands in descriptions #545

As a quick improvement, we should be able to parse markdown code blocks and highlight those in the game UI.

If the first and last line of Text is ```, the lines in the middle should be (parsed as swarm code and) highlighted.

Before:

  - 'Example:'
  - '  move; turn left; move; turn right'

After:

  - 'Example:'
  - |
    ```
    move; turn left; move; turn right
    ```

As a bonus, we could easily check that the code parses in the CI. #309

[byorgey]

As I mentioned in #545, I do not actually think highlighting code blocks will be easy, although I would be very happy to be proven wrong. But I agree we should do this and at least get #309 .

[xsebek]

Well for start I would not semantically highlight the code - just make it bold or italic or something like that - simple attribute of the txt block. 🙂 And of course drop the ```.

Highlighting code semantically would require highlighting individual words like in #545, so that is harder even without the extra work of matching parsed elements to code spans and highlighting those.

[byorgey]

Ah, that makes sense, yes, that should be possible.

[xsebek]

Turns out there is a library for this:

• https://hackage.haskell.org/package/brick-skylighting

In particular, this function to which we can pass tokenized lines:

renderRawSource
  :: (T.Text -> Widget n) -- just put txt function here
  -> [Sky.SourceLine]     -- [SourceLine] == [[Token]] == [[(TokenType, Text)]] 
  -> Widget n

So we can use our already processed terms and just label them.

[xsebek]

Well, it also needs to work with our pretty printing. 🤔

The way to do that seems to be to:

1. annotate the Doc.
2. produce a SimpleDocStream (we can specify line length)
3. turn that into a list (lines) of lists of text segments each with an assigned TokenType.
   • consider provided helper functions like renderSimplyDecorated

[xsebek]

Since I think the implementation is clear now, I will do semantic highlighting too.

Notes:

1. start by splitting the descriptions into plaintext paragraphs and code blocks
2. parse the code blocks
   • if it does not keep the original text so we can print it plainly (e.g. while developing)
3. pretty print the code blocks (consider line length)
4. set up an integration test to check that all code blocks are valid
5. follow the steps in the comment above to highlight the output
6. profit

[xsebek]

I looked into this and ended up quite confused. 😕 (See WIP in 98ee109)

Adding annotations is simple enough, but there are more steps to the Doc TokenType -> Widget Name conversion:

• we already produce SimpleDocStream
• but I have no idea how to easily turn it to [[(TokenType, Text)]]
• for the brick library to highlight our code, we would need some Style definition, which attrMappingsForStyle would turn to brick attribute tree that we would add to our current one
  • given style needs to be loaded from XML with loadSyntaxFromFile
  • this seems like it should be configurable by users, but with default provided in our data

[xsebek]

@byorgey if you have some idea how to approach this, I would be glad to hear it. 😅

The good news is that once we figure this out, we should get nicely coloured code widgets everywhere.
Also the Parse and Validate parts will be done as part of #1106.

The bad news is that highlighting is more work than I expected and I will not implement it in #1106. I will probably have to render inline code simply and blocks can get properly highlighted later. @byorgey so I will probably rename this issue back to "Highlight code blocks". 😅

[byorgey]

The bad news is that highlighting is more work than I expected and I will not implement it in #1106. I will probably have to render inline code simply and blocks can get properly highlighted later. @byorgey so I will probably rename this issue back to "Highlight code blocks". 😅

Makes sense. I don't have any particular ideas off the top of my head. If you wanted to schedule a meeting sometime to think about it together I'd be happy to do that.