ISPC grammar for tree-sitter language parser

[fab4100]

Dear All,

I have written some grammar for an ISPC language parser in tree-sitter. The grammar and parser source can be found in https://github.com/fab4100/tree-sitter-ispc.

The parser and various queries are now also integrated in nvim-treesitter.

[dbabokin]

Thanks for this contribution!

Looking at the repo, I see that you have some tests, but they are not fully covering the language. It's probably ok, but i would suggest specifically covering floating point literals - we parse them differently as in C/C++. You can use these to tests for the examples of the literals that need to be parsed: float_literals.ispc and float_literals_value.ispc

Also, I'm wondering what is the process for updating the grammar? I think we are not going to add anything new in the short term, but would be good to understand the process.

[fab4100]

I have only added tests for the grammar rules that were needed on top of what is implemented for C already, but you are correct that tests for number literals are missing. I will add these tests since I did implement a new rule for number literals in ISPC. The current grammar parses everything in stdlib.ispc and all code below tests/ without errors in the syntax trees. There are a few exceptions in tests/lit-tests: there are some syntax tree errors that are excepted (e.g. in float_literals_errors.ispc) and some are due to interplay between pre-processor constructs and ISPC code (which would result in correct IPSC code after the cpp pass). One other issue that currently results in syntax tree error is foreach (int i = 0...N) versus foreach (int i = 0 ... N) the compiler accepts both but the grammar currently understands only the latter correctly (due to miss interpretation of integral 0 and float 0.).

I currently do not have a proper process for updating the grammar, my plan was to update on the fly as new ISPC releases are published (if grammar update is required). The generated EBNF is also very helpful (#2529) which I will use to improve the current grammar as I get to it.

Unfortunately there are no universal queries for tree-sitter as different tools/editors have their own flavors. I currently maintain the queries in https://github.com/nvim-treesitter/nvim-treesitter/tree/master/queries/ispc which work for nvim, if somebody wants to use the parser in emacs for example they would need to add their own set of queries. I have added a basic standalone query in https://github.com/fab4100/tree-sitter-ispc/blob/master/queries/highlights.scm which I further plan to use for ISPC syntax highlighting on github (https://github.com/github-linguist/linguist/blob/master/vendor/README.md) because current highlighting is broken (e.g. https://github.com/ispc/ispc/blob/main/stdlib.ispc#L81)

[dbabokin]

Thanks for detailed answer.

Is tree-sitter grammar compatible with linguist? I'm working on Textmate grammar for the purpose of introducing it to linguist. It's also is needed for better support through VSCode. But I don't know if automatic conversion from Bison to Textmate is possible and I'm trying to implement it manually.

[fab4100]

Is tree-sitter grammar compatible with linguist?

Yes. Tree-sitter is also used for other features than syntax highlighting on github (e.g. https://github.blog/2021-12-09-introducing-stack-graphs/#creating-stack-graphs-using-tree-sitter).

I'm working on Textmate grammar for the purpose of introducing it to linguist

I do not want to interfere here, but I think for github functionality a tree-sitter parser for ISPC may be more valuable. Let me know what you think.

FYI: this PR is currently pending: tree-sitter/tree-sitter#2282

[fab4100]

Update: I wanted to play around with linguist but it seems they are not doing syntax highlighting (which was my initial understanding) and they seem not be converged right now with tree sitter altough they mention some tree-sitter grammars they use.

github-linguist/linguist#6073

[pbrubaker]

Hey @fab4100, it seems we've duplicated some effort here. My team (@ethandev) also completed a tree-sitter grammar about two weeks ago! hah! We are going to be using it in our LSP (Language Server Protocol) for better Intellisense like functionality in both vscode and Visual Studio.

We chose to go with a TextMate grammar for syntax highlighting as that's what is used in vscode, and it's supported in Visual Studio. Those are our main two targets. But I'm not against using tree-sitter for syntax highlighting.

What are your thoughts on the future maintenance of the ISPC grammar? Do you want to continue maintaining it and we'll use yours? Should we merge your work with ours and maintain it as part of the ISPC repo and maintain it from there?

[fab4100]

What are the next steps for migrating the parser repository? (I can change license and push the code to the new repo)

[dbabokin]

I've created a private repo and sent you an invite. I'll need to do some paperwork to switch it to public repo, but this should not take too long.

My primary concern is the license in the sense that we need to understand where the prior work comes from, so we are 100% clean. We'll need to have a "Bill of Material", which explains where the third party code comes from. Either @pbrubaker or myself will need to work with you to build this BOM.

I think you can migrate your repo to this new one and we'll go from there.

[fab4100]

I have pushed the code and added a few remarks https://github.com/ispc/ispc.tree-sitter#todo. The Rust crate item is optional and it requires an account on crates.io (GitHub login should work). The crate can be created via CI but I currently used a crates.io account linked to my GitHub account.

[MarijnS95]

I'll need to do some paperwork to switch it to public repo, but this should not take too long.

Is this still in the works? I'm adding ISPC support to helix in helix-editor/helix#8678 and pointing to @fab4100's repository now.

[pbrubaker]

Hey @MarijnS95, yes, still in the works. This fell down my todo list but I will follow up and get it done as quickly as possible.