feat(matlab)!: replace parser

[acristoffers]

I wrote a MATLAB parser which has more features than the current one:

• It detects commands. The code

variable P(1, 2, 3) semidefinte

returns a function_call token with 2 argument tokens, which is how matlab sees it. I use a scanner to correctly detect the line, so it won't mix it with a function call followed by a identifier.

• It detects string escapes and formatting. This returns a string with two tokens inside, differentiating escape sequences from formatting options. Unfortunately there is an issue with tree-sitter that prevents me from doing it with single-quoted string at the moment, but it works fine with double-quoted ones.

"Format: \t %2.3f"

• It detects line continuations, lambdas, parfor statement, arguments statements (a kind of MATLAB block), two more operators, superclass function call, and when it comes to classes, it correctly detects and generates specific tokens for everything (MATLAB has some weird constructs, like the properties block).

The parser also generates more tokens, which allows to better handle not only syntax highlighting, but specially textobjects. For instance, matrices and cells are composed of rows with values, which allows the creation of textobjects to operate on those, which is something I was missing a lot. At this point I believe I have implemented everything in MATLAB's grammar.

[clason]

• Commit message should be feat(matlab)!: replace parser (breaking change).

• tags.scm are not used in this project, so please drop that file.

• Consider adding an injections.scm (at least for comments, but ideally for any language -- pure C? -- where it makes sense).

• Consider adding @spell captures where appropriate.

The full list of allowed captures (which is different from upstream/Helix) is here: https://github.com/nvim-treesitter/nvim-treesitter/blob/master/CONTRIBUTING.md#highlights.

[acristoffers]

Commit message should be feat(matlab)!: replace parser (breaking change).

Fixed.

tags.scm are not used in this project, so please drop that file.

Removed.

Consider adding an injections.scm (at least for comments, but ideally for any language -- pure C? -- where it makes sense).

Was already there, only comments. MATLAB does not allow injections.

Consider adding @spell captures where appropriate.

Was already there, at the end of the file. For strings and comments only, as most other projects do.

The full list of allowed captures (which is different from upstream/Helix) is here: https://github.com/nvim-treesitter/nvim-treesitter/blob/master/CONTRIBUTING.md#highlights.

I fixed the function call one, but it seems to be the only trully different from what I had already.

[clason]

We also have indents; you could try adding at least initial support for them? (Not required.)

[clason]

You also need to change the commit hash in the lockfile.json.

[acristoffers]

We also have indents; you could try adding at least initial support for them? (Not required.)

I forgot about it, thank you for the reminder. It's in the repo now. And I also changed the commit in the lockfile.

[acristoffers]

Just did two small fixes to use the @boolean and @error captures.

[clason]

Note that the locals captures also differ from upstream: https://github.com/nvim-treesitter/nvim-treesitter/blob/master/CONTRIBUTING.md#locals

[lucario387]

indents.scm looks fine to me now 👍

[clason]

@structure query is illegal in nvim-treesitter (for the moment).

[acristoffers]

@structure query is illegal in nvim-treesitter (for the moment).

Just saw it. Already changed it to be just @keyword.

[acristoffers]

I had a calloc in the parser which was unnecessary, so I changed the code to just use the stack instead.

[acristoffers]

When I first wrote this I was thinking about taking screenshots of the existing grammar and mine to show the difference, and then I thought it would be too much trouble and just wrote that text. Now I just realized that Helix uses the existing one too, so I only had to open helix and neovim to compare both. Night and day.

[acristoffers]

I added textobjects to some classes constructs, which required a small grammar adaptation.

[acristoffers]

The change in order revealed that I had a named node and some aliases with the same name, but totally different contexts. Fixed that.

[acristoffers]

MATLAB accepts cell-indexing as a function identifier, a{1}(1). Fixed that.

[acristoffers]

Fixed the matrix parsing, it now follows MATLAB way of doing it a bit closer.

[amaanq]

There are some design choices and many issues with the grammar and queries that I do not agree with/are incorrect and I think are better fixed upstream before adding here. Let me know if you'd be welcome to a PR on my end, that'd be easier than writing a bunch of comments here

[acristoffers]

There are some design choices and many issues with the grammar and queries that I do not agree with/are incorrect and I think are better fixed upstream before adding here. Let me know if you'd be welcome to a PR on my end, that'd be easier than writing a bunch of comments here

Sure, go ahead.

[amaanq]

acristoffers/tree-sitter-matlab#1

[acristoffers]

I added a file that should not be in the commit. The last force-push only removes that file.

[amaanq]

Is the pr ready to be merged?

[acristoffers]

Yes.

[acristoffers]

I just created nvim-treesitter/nvim-treesitter-context#292 and nvim-treesitter/nvim-treesitter-textobjects#468 to follow this one.