Provide grammars and LSP

[GoldsteinE]

As far as I understand, there’s currently no support for editors other than typst.app. Some things that could improve this situation:

1. Providing a TextMate grammar, which is not very powerful, but supported by a wide range of editors and tooling.
2. Providing a TreeSitter grammar, which is supported primarily by Neovim and allows for advanced editor integration.
3. Providing a full-fledged LSP, which is supported by basically everyone and allows for all sorts of IDE-like features.

[SeniorMars]

I'm working on number 2.

[LaurenzV]

TextMate grammar can be found here.

[KaranAhlawat]

2. Providing a TreeSitter grammar, which is supported primarily by Neovim and allows for advanced editor integration.

Also supported by Emacs, Helix, and a couple of other editors :)

[imatpot]

Development of a Typst LSP (3.) seems to have been started over at nvarner/typst-lsp

[KiruyaMomochi]

Just found that typst.app compiles files locally.

Does this mean we can have a VS Code extension that not only

1. works in VS Code for the Web, but also
2. compiles document in almost real-time through WebAssembly?

And if it's true, I finally can use github.dev to write my paper without downloading anything, and collaborate with others.

That is fantastic.

[reknih]

We've got an LSP at nvarner/typst-lsp and an (admittedly a bit outdated) TextMate grammar here. @SeniorMars can you share a link to your WIP Tree Sitter grammar? Once we collected that here, we'll close this issue, any further issues shall be filed on the respective projects.

[laurmaedje]

There's also @frozolotl's grammar here: https://github.com/frozolotl/tree-sitter-typst

[RubixDev]

@reknih, the tree-sitter grammar from @SeniorMars can be found here: https://github.com/SeniorMars/tree-sitter-typst

[laurmaedje]

As there are multiple grammars and a third-party LSP, I'm going to close this.

[GoldsteinE]

Are there plans on blessing some solutions as official-ish (while keeping them community-maintained)? Right now there’s a naming problem: for example, nixpkgs currently includes typst-lsp, which may be not ideal if another project with that name emerges. Blessing one typst-lsp and one tree-sitter-typst as official would resolve the issue “what to include by default” and “what gets priority while reserving names”.

[laurmaedje]

I think its a little early for that, but I can imagine that in the medium term. Also depends on what the maintainers of these projects want and plan.

[SeniorMars]

I agree. While my grammar right now has more stars - it is nowhere complete, I think we should wait until they become feature-proof.

[uben0]

I am writing a TreeSitter grammar that took me several months to come by, yet, it is not finished.

Nonetheless, it seems more advanced than the two evoked earlier, as it parses correctly more tricky cases.

https://github.com/uben0/tree-sitter-typst

As mentioned in the README, two issues remain, and I need help resolving them. I am looking at you @SeniorMars 👀

Sincerely, it seems like the TreeSitter tool is not meant for that kind of language. It feels like this idea of having optional ; to terminate an inline expression is terrible design. Having it mandatory would solve everything.

Difficult cases:

#a._ b _

#if a {} elseeee {}

#{
   2
   + 2
}

[frozolotl]

I am writing a TreeSitter grammar that took me several months to come by, yet, it is not finished.

Nonetheless, it seems more advanced than the two evoked earlier, as it parses correctly more tricky cases.

https://gitlab.com/uben0/tree-sitter-typst

It seems the repository is set to private.

As mentioned in the README, two issues remain, and I need help resolving them. I am looking at you @SeniorMars 👀

Sincerely, it seems like the TreeSitter tool is not meant for that kind of language. It feels like this idea of having optional ; to terminate an inline expression is terrible design. Having it mandatory would solve everything.

Difficult cases:

#a._ b _

Here I'm not sure what the issue would be iih.
EDIT: I think it's about parsing emphasis? #a._ isn't valid Typst code anyway, so either field accesses or identifiers might have to be parsed differently from how they are parsed now. Another way of solving this could be setting a precedence for that rule, but I don't believe that is necessary.

#if a {} elseeee {}

This one should be solved by providing word: https://github.com/frozolotl/tree-sitter-typst/blob/master/grammar.js#L29

#{
   2
   + 2
}

In my grammar, I haven't implemented binary operators yet, but I believe this should just be a case of only allowing trivia that isn't a newline. For instance, here I differ between trivia that allow newlines and those that do not.

[uben0]

@frozolotl Thanks for the feedback! I'll test the possible fixes and keep this response updated.

By the way, #a._ b _ is valid code (it compiles without error) and should have the following tree:

(source_file
    (ident)
    (text)
    (emphasis
        (text)))

[uben0]

@frozolotl Can I use parts of your code in the grammar I am writing ? I'm thinking about the regex for XID_Continue without _. If yes, do you have conditions?

[frozolotl]

@frozolotl Can I use parts of your code in the grammar I am writing ? I'm thinking about the regex for XID_Continue without _. If yes, do you have conditions?

You are allowed to use all the code in frozolotl/tree-sitter-typst under the conditions given in the LICENSE.
The license shouldn't be difficult to read and GitHub does provide a summary at the top, though that one is not legally binding.

I would prefer that we kept this conversation and other ones not related to this issue somewhere else, though. So, if you have any further questions or want to discuss my tree-sitter grammar, you should do that there. Alternatively, you can contact me via email, via matrix, or on discord (@frozolotl, or the typst discord).