Should Kakoune have a parser?

[alexherbo2]

Use case:

• PEG to handle syntax highlighting

Example:

; JSON grammar
; see http://json.org

JSON ← S? ( Object / Array / String / True / False / Null / Number ) S?

Object ← "{"
             ( String ":" JSON ( "," String ":" JSON )*
             / S? )
         "}"

Array ← "["
            ( JSON ( "," JSON )*
            / S? )
        "]"

String ← S? ["] ( [^ " \ U+0000-U+001F ] / Escape )* ["] S?

Escape ← [\] ( [ " / \ b f n r t ] / UnicodeEscape )

UnicodeEscape ← "u" [0-9A-Fa-f]{4}

True ← "true"

False ← "false"

Null ← "null"

Number ← Minus? IntegralPart FractionalPart? ExponentPart?

Minus ← "-"

IntegralPart ← "0" / [1-9] [0-9]*

FractionalPart ← "." [0-9]+

ExponentPart ← ( "e" / "E" ) ( "+" / "-" )? [0-9]+

S ← [ U+0009 U+000A U+000D U+0020 ]+

[alexherbo2]

Yi (repository) has strong, incremental parsers for highlighting.

[mawww]

Supporting grammars is seducing, however having a performant, data driven, incremental, tolerant (as most of the time you are working on an invalid program) parser seems more like a research project. It is not a goal at the moment for Kakoune.

[alexherbo2]

I was expecting this answer. The reason why I open this issue is because of
your TODO. Regex are simple, definitely, but perfectible to describe well a
language. Regex highlighters are just a quick and dirty helper to please our
eyes.

[alexherbo2]

Have you look at Yi? It is an interesting project, like Kakoune. It still needs more love.

[mawww]

I did, parsers are coded in Haskell and compiled in, so not really data driven.

[haarts]

I know this is an ancient issue. But recently I found this project: http://tree-sitter.github.io/tree-sitter/ (created by the Github organisation). It seems to address exactly the issues raised in #50 (comment) by @mawww. From the website:

• General enough to parse any programming language
• Fast enough to parse on every keystroke in a text editor
• Robust enough to provide useful results even in the presence of syntax errors
• Dependency-free so that the runtime library (which is written in pure C) can be embedded in any application

It has been around for a while now speaking to it's maturity.

[Delapouite]

Related ongoing PR about tree-sitter: #3160
Also https://github.com/ul/kak-tree for selectors

[haarts]

That's pretty spectacular. I hadn't even thought about selections! Syntax highlighting was my annoyance. It's just too damn hard to write regexes (think code in string interpolations, ugh).