Remove newline when highlighting random chunks of code

[mrsdizzie]

By default Chroma will always add new lines for any pieces of code for certain file formats. See :

https://github.com/alecthomas/chroma/blob/bac6996317c811706b2ade529f31d276a1accae8/regexp.go#L433-L435

And

https://github.com/alecthomas/chroma/blob/bac6996317c811706b2ade529f31d276a1accae8/lexers/m/make.go#L16

This breaks the way we render diff when one of these formats also has a line of added/deleted code within a new/del line. Make sure this doesn't happen by overriding the default lexer options and setting the Nested flag to true.

Fixes #12172

[silverwind]

Tokens never span across multiple lines, right? Should probably report this to chroma.

[mrsdizzie]

Correct and it should never see more than one real line of code at a time anyway because of how we handle diffs and process them one line at a time.

I think this might be intended chroma behavior and this is just a separate case on our end that I didn't fully account for where we process one line of code in sections.

I'll look further to see if something on their end needs to change, but this will fix the problem for us.

[mrsdizzie]

actually see another issue need to fix first before merging

[mrsdizzie]

@silverwind for a better answer and why I didn't notice when testing:

Whats actually happening here is that only certain lexers are configured to always add a new line for code when there isn't one there. See:

https://github.com/alecthomas/chroma/blob/bac6996317c811706b2ade529f31d276a1accae8/regexp.go#L433-L435

And

https://github.com/alecthomas/chroma/blob/bac6996317c811706b2ade529f31d276a1accae8/lexers/m/make.go#L16

This is inherited from Pygments and the idea is that the lexers aren't able to parse arbitrary lines of code for certain formats without a newline since it is part of the matching. So thats why it happens in the combination example case of being a Makefile AND having a section of added/deleted code within the diff line.

We don't need the newline after the code has been processed so OK to remove it. Added check to not process a line that is only a newline as well to preserve those

[silverwind]

This is inherited from Pygments and the idea is that the lexers aren't able to parse arbitrary lines of code for certain formats without a newline since it is part of the matching

I guess we could get a better highlighting result by passing the diff's before/after as multi-line at the cost of reduced performance of course. The current diff highlighting still seems good so maybe not worth the effort.

[lafriks]

🚀