Having a tree-sitter parser for cppfront

[uyha]

For those who do not know, tree-sitter is a parser generator that is designed for highlighting code. Since cppfront's grammar is stable (discussion for numbered release is going on), I'd like to have a discussion regarding having a tree-sitter parser for cppfront (I'm planning to start it). I've tried to look but I don't think there is something like BNF specification for the syntax yet.

[gregmarr]

See also #910 and #951

[hsutter]

Thanks! I've written this on the sister threads @gregmarr mentioned: Without rewriting the grammar, are there any blockers right now to mechanically generate EBNF/PEG/Tree-sitter from the //G comment grammar I'm using now (and which likely still has a few oversight corrections needed)? If so is there anything incremental I can do to unblock that mechanical transformation?

[uyha]

what is the difference between //G and //GT?

[zaucy]

I believe GT means it's a TODO

[hsutter]

I believe GT means it's a TODO

Exactly -- sorry, that was a bit obscure because those were notes to myself primarily while first implementing the grammar. I left them in as things to come back and think about, but not necessarily to add (many may not be needed).

[uyha]

Not a problem, I was just confused since there are actual //GTODO so I thought //GT meant something else.

[hsutter]

Ah, true. I'm not always consistent over time!

[uyha]

I started writing the parser and encountered this grammar

parameter-declaration:
    this-specifier? parameter-direction? declaration

parameter-declaration-list:
    '(' parameter-declaration-seq? ','? ')'

parameter-declaration-seq:
    parameter-declaration
    parameter-declaration-seq ',' parameter-declaration

function-type:
    parameter-declaration-list throws-specifier? return-list? contract-seq?

unnamed-declaration:
    ':' meta-functions? template-parameters? function-type requires-clause? '=' statement
    ':' meta-functions? template-parameters? function-type statement
    ':' meta-functions? template-parameters? type-id? requires-clause? '=' statement
    ':' meta-functions? template-parameters? type-id
    ':' meta-functions? template-parameters? 'final'? 'type' requires-clause? '=' statement
    ':' 'namespace' '=' statement

declaration:
    access-specifier? identifier '...'? unnamed-declaration
    access-specifier? identifier alias

which is saying that some syntax like the following is valid

main : (a: namespace = {}) = {}

I know it's invalid, but it seems like the grammar is not very accurate also.

[hsutter]

Thanks! Good question.

The grammar is intentionally general, and I think it's as intended in this case. There are semantic rules that make some grammatical programs not valid. It could be that I'm using semantic restrictions for grammar subsetting more than in other languages.

One reason for having all declarations be identifier : kind = statement is for generality, including to pursue new directions such as "passing types as values" in the future.

main : (a: namespace = {}) = {}

I could restrict this case now, but it's possible I could want to allow that in the future, for example to have a metafunction that takes a type or namespace as a non-template parameter.

If the goal is to write something like a syntax highlighter, I don't know of a reason to special-case this and restrict it arbitrarily? It can just highlight the general grammar attractively, and cppfront will reject some cases today if they're not yet legal. That also makes the syntax highlighter more future-proof as Cpp2 supports more uses of the general grammar.

Does that help?

[uyha]

yup, it's more of a curiosity, it doesn't affect writing the parser much.

[tsoj]

I wrote a tree-sitter here: https://github.com/tsoj/tree-sitter-cpp2
Since this is my first time doing this kind of parser stuff, it is a bit messy.

[zaucy]

Awesome I gave it a try in neovim and it works fairly well. The other ones didn't seem to work so huge props :)

Example

[geoffreyvanwyk]

Thanks for doing that. I know it is not easy.

[MaxSagebaum]

Looks nice!