Spellchecking based on Tree-Sitter queries?

[connorlay]

For a bit of background: I've been writing a lot of documentation lately for Phoenix Live View and Helix is my daily driver 🧬 ⚗️

Would the maintainers be interested in accepting a PR to Helix core that adds a built-in spellchecker that uses tree-sitter queries to determine what parts of the document to spellcheck? For example, consider the following Elixir code:

@doc "This documentation contains a tyyyypo"
def this_function_also_contains_a_tyyyypo do
  # mooooore typooooos
end

As a user, I would like the text inside of the comment string and the @doc to be spellchecked against my preferred language (in this case, en-US), but not the Elixir identifiers themselves. The tree-sitter queries could look something like this:

(comment) @spellcheck

This would allow for syntax-aware spell checking for each language and would be customizable for each user. Typos could be represented as diagnostics, possibly even with a list of close matches to a known dictionary?

See https://github.com/lewis6991/spellsitter.nvim for an example of what I am thinking of 🙂

[David-Else]

Sounds great, my only thought is that https://github.com/valentjn/ltex-ls has proven to be the ultimate spell and grammar checker, could you integrate that somehow? At the moment it needs some special features to work that are not part of the lsp spec. There is a plugin version for nvim, https://github.com/vigoux/ltex-ls.nvim that implements those features, could it be ported to Helix?

[connorlay]

Interesting, I haven't seen the LTeX language server before. I don't see why this could be added to Helix once #2507 is merged? Supporting non-LSP features would probably need to be handled by a plugin, or a wrapper similar to how https://github.com/typescript-language-server/typescript-language-server wraps the non-LSP features of tsserver.

That being said, I would not personally use such a language server and would rather have something simple and built-in to Helix. I don't want or need an automatic proof reader, all I want is a simple spell checker to help me since I am not the greatest touch typist (or speller) 😅

[mandarvaze]

💯 for ltex-ls I use it everywhere (Emacs, neovim) I would love to see helix support it.

[getreu]

A short blog post about Helix and ltex-ls: Jens Getreu's blog - Note talking with Helix, Tp-Note and LanguageTool

Comment in Discussion #3756 · helix-editor/helix

[the-mikedavis]

Using tree-sitter to determine what should be spell checked sounds like a good idea to me 👍

Spell-checking in general though is a question mark: there isn't anything built-in at the moment or a way to plug in a checker like aspell. It would probably be worth some discussion whether that even belongs in the core or as a plugin.

I think it would be handy if LSP could be used to provide spell-checking capabilities. Maybe languages could inject a language like we do for tree-sitter-comment or markdown in docs and the spelling language server could act on the injected ranges (although currently injected content can't be adressed by a language server afaik).

[connorlay]

@the-mikedavis One approach we could take if we wanted this to be built-in is to use Enchant via the enchant crate. Apparently enchant is the spell checker that Emacs uses, and it supports multiple different back-ends so it should be fairly portable and customizable if needed. Should this be in core? Based on the design goal of "Modern and sane builtin features" mentioned on https://helix-editor.com I think so 🙂

[connorlay]

If we wanted something simpler, there is the ispell crate that we could use to call ispell, aspell, or hunspell.

[the-mikedavis]

I worry that building in spell checking would take a lot of work/code for something that can already be done by LSP (although it would need #2507). Enchant doesn't look like too much code but I'm not sure how the dictionaries fit in - would we need to distribute dictionaries for languages?

With supporting multiple languages, I think spell checking has the possibilty to be too large in scope to be good for core. Like a magit mode or #1976, it would be great if it were in core but scope-wise it may be large enough to be a project on its own.

[connorlay]

@the-mikedavis I started a proof-of-concept draft PR exploring what a built-in spell checker could look like using the ispell crate and aspell: #3830 There's still much to do, but you can get the idea of how things look if you want 🙂

Regardless if you decide this approach makes sense or not to include in core, I'm learning a lot by implementing it 😁

[kirawi]

Microsoft recently released a blog post on how they introduced spell checking to Visual Studio: https://devblogs.microsoft.com/visualstudio/visual-studio-spell-checker-preview-now-available/

[the-mikedavis]

I've come around to the other side on this since my last comment ☝️: I think it's reasonable to have spell-checking as a core feature. Finding system-installed dictionaries is fairly straight-forward and fits well into helix-loader, plus it's easy to manually install dictionaries from a source like JetBrains/hunspell-dictionaries if necessary. Diagnostics for incorrect words can fit squarely into the LSP diagnostics code and we can re-use much of the code-actions code for suggesting corrections and providing a way to allow/forbid words in your personal dictionary. Delegating to a language server works well for prose languages but tokenization is difficult for code (i.e. splitting code into words that can be looked up in the dictionary), and we should be in a nice spot to do this robustly per-language with tree-sitter.

Spell checking also requires some other advancements in core that we want for other features like running tree-sitter queries across language injections (like for rainbow brackets, textobjects, indents) and segmenting a whole buffer into words efficiently (useful for- but maybe different than lexical autocomplete).

Specifically about spell-checking, we really only need the ability to: create an in-memory dictionary from Hunspell .dic and .aff files, check if a word is in the dictionary, and suggest corrections. Personal dictionaries would be nice to include but could be a follow-up.

The existing options we have (hunspell, nuspell, enchant) are a bit annoying from a build perspective: we would need to depend on the presence of libenchant or hunspell as a library at build-time and use FFI, so spell checking as a feature would probably need to be optional. I'd really like to avoid complicating the build or increasing our dependency on C(++) codebases so as archseer mentions, I'm working on a pure-Rust spell-checker compatible with Hunspell dictionaries. I'll make the repo public and push up an integration branch once the project matures some more.

[jerabaul29]

As #2507 has been released as part of 23.10, could / should this discussion be revisited? :) In my experience, good spellcheck also in a coding environment (not just pure markdown or text) per se is already quite useful at catching a number of "stupid" bugs :) .

My questions:

• I guess that there is no reason any longer given Add support for multiple language servers per language #2507 to not use (if need be custom per language) LSP(s) for providing spellcheck on all all kinds of relevant files?
• have some users special experience with using LSP-based spellcheck across file types and have recommendations? Ideally, could a tiny bit of extra config be added to get spellcheck-through-LSP out of the box / could such an example be added to the documentation for n00bs like me to just copy paste? :)
• I understand that @the-mikedavis solution above sounds more attractive on the long run, but would a slightly "ad hoc" solution based on other existing LSPs or similar be possible in the meantime?
• should spellcheck status / support be reported in ```helix --health````? (I guess it does not make sense to report it on a per language basis if it can be shared across languages?)?

[jerabaul29]

Sounds nice, I agree pushing a MVP even if it is a bit limited is better than nothing, and it makes it easier to get feedback / iterate as usual :) .

[akhilman]

I have managed to add CSpell as a spell checker using diagnostic-language-server. Both run by Deno runtime.
Here is my languages.conf:

# Languages

[[language]]
name = "git-commit"
language-servers = [ "dls" ]

[[language]]
name = "markdown"
language-servers = [ "marksman", "dls" ]

[[language]]
name = "rust"
language-servers = [ "rust-analyzer", "dls" ]

# Language servers

# Diagnostic language server.
# See https://github.com/iamcco/diagnostic-languageserver .

[language-server.dls]
command = "deno"
args = [ "run", "--allow-read", "--allow-env", "--allow-run", "--allow-sys", "npm:diagnostic-languageserver", "--stdio", "--log-level=2" ]

[language-server.dls.config.filetypes]
"*" = "cspell"

# CSpell https://cspell.org/docs/getting-started/ .

[language-server.dls.config.linters.cspell]
command = "deno"
args = [ "run", "--allow-read", "--allow-env", "npm:cspell", "stdin" ]
sourceName = "cspell"
rootPatterns = [
  ".cspell.json",
  "cspell.json",
  ".cSpell.json",
  "cSpell.json",
  "cspell.config.js",
  "cspell.config.cjs",
  "cspell.config.json",
  "cspell.config.yaml",
  "cspell.config.yml",
  "cspell.yaml",
  "cspell.yml",
]
debounce = 1000
formatLines = 1
formatPattern = [
  "^[^:]+:(\\d+):(\\d+) - (.*) \\((.*)\\)$",
  { line = 1, column = 2, security = 3, message = [3, " (", 4, ")"] },
]
securities = { "Unknown word" = "warning" }

[weakish]

@akhilman Does diagnostic-language-server support code actions (e.g. replace the word with suggestions, add word to dictionary, etc)?

[jerabaul29]

Yep, the repo is private for now and I'm squatting the crate name :P. Spellbook can currently spell-check reasonably well (missing some features for compatibility with aspell) but suggesting corrections will take a while to work out. [...]

@the-mikedavis just being curious: would you be ok to make the corresponding repo public, even if it is just in a very early form? I am trying to look a bit about the different options available and it would be very useful to see your work on this, even if it is preliminary :) (for example, I am still a bit unsure if I want to import a big LSP that offers suggestions, or if I am happy to start with something small that just highlights misspellings that I then 99% of the time can fix myself easily :) ).

[the-mikedavis]

I'm working on Spellbook actively again but I'm rewriting it based on Nuspell rather than Spylls for performance and correctness. The part about "Spellbook can currently spell-check reasonably well" is not the case at the moment but my plan is to open up the repo after the check algorithm is done. Without a full implementation of check though the project is pretty useless. There would be way too many false positives.

Suggestions and personal dictionaries could be done as follow-up. Plus the suggestion algorithm is simpler than check. My plan is to start small with a :spell typable command that segments the words in the current view, or maybe the whole buffer. But this is still a ways out so for now you're better off relying on a language server.

[akhilman]

Nope, it does not. New words could be added to a custom dictionary file in the project directory.

…

On Fri, 2024-01-26 at 06:01 -0800, weakish wrote: @akhilman Does diagnostic-language-server support code actions (e.g. replace the word with suggestions, add word to dictionary, etc)?

[David-Else]

Vale now has a language server, and tree sitter based spell checking for comments in programming languages like Rust!

https://github.com/errata-ai/vale/releases/tag/v3.5.0

https://github.com/errata-ai/vale-ls (needs vale installed)

I have not tried it in Helix yet, but it looks like it could be perfect.

[sbromberger]

I got vale working in Helix. It's really good.

One other one to try out is https://github.com/tekumara/typos-lsp - I was using this beforehand. It has really low false-positive rate but a higher false-negative rate due to the fact that it has a predefined list of common misspellings and only alerts on those. It's fairly comprehensive, though!

[mandarvaze]

@sbromberger

I got vale working in Helix. It's really good.

Do you mind sharing details/links/helix config ?

[sbromberger]

Sure. Here's the languages.toml entry:

[language-server.vale]
command = "vale-ls"

Then you just add it to whatever language config you want.

As far as installation goes, you'll need both vale and vale-lsp binaries on your path. I used a release of vale from the github releases page, but built vale-lsp from a git clone (using cargo install . from the cloned source directory). Then I created ~/.config/vale/.vale.ini using the contents of the configuration generator (https://vale.sh/generator), and ran vale sync from the terminal. Everything just worked.