LSP: make autocompletion etc work better when the file has changed since the last successful compilation

[lpil]

Currently the language server only gets information about the code after successful compilation, so there is a period of drift between the state of the code in the LS and in the source file while code has errors.

This can result in:

• Autocompletions not triggering as the LS thinks the cursor is currently outside of a function
• Incorrect autocompletions triggering as the LS thinks the cursor is in some other context (e.g. it thinks it is in a function when you're writing an import)
• Go-to definition going to the wrong place if the destination has been moved up or down lines by edits
• Go-to definition not working as the location of the item being clicked on has changed slightly by edits

And so on.

Long term we want to make everything fault tolerant, but this needs to be an incremental process.
What can we do to improve the experience in the near future?

I believe this will be a very high value and impactful area of work.

[lpil]

Tagging @giacomocavalieri @rawhat @Acepie who I believe are interested in working on the LSP

[Acepie]

So my understanding of this (if I am reading correctly) is that in the short term the goal is to get lower hanging fruit wins by doing things that are possible without going to implementing fault tolerant parsing? So since we have access to the modified src we can do local changes relatively safely (i.e. things like using the src byte_index/cursor pos for certain actions instead of the saved one, things that ultimately amount to line level simple parses, directly fixing edit positions for completions, maybe confirming a definition is still at a location before doing a goto?). I assume anything beyond that and we might as well just implement fault tolerant parsing since we'd have to do partial parses anyway.

As for the fault tolerant parsing, I "theoretically" understand the core idea of what needs to change but I'm not entirely sure about how to plan it out into chunks. From my general knowledge on the topic/what I've read so far the strategy at a high level would be something like:

• Update the existing parse function to return a list of error sites instead of 1 error site (this for now would just be the type change to prepare by making it a list and adjusting the call sites to be ready for multiple errors)
• Determine which parse function failures are continuable/have clear ending markers
• Update parser logic so that if a continuable parse failure occurs, append to our stored list of errors, eat characters to a recoverable ending marker, and then continue
• Update ast to include "error hole" types for places where we would expect a specific parsable thing that is missing
• Allow access to ast for lsp even in cases where errors occur (which i guess would mean updating the error type to include the error holed ast)
• Update lsp to handle cases with "error hole" types

Does that seem generally right? With the idea being that over time we improve how granularly we put error holes/how well we handle those error holes in the lsp/diagnostics

[lpil]

Aye, it would be good to find ways to improve this in the short term as making every compiler pass fault tolerant is a large amount of work, and we would also need to make it capable of integrating state from previous compilation passes so we have more context in code with errors, which would likely involve changing how various syntaxes (mainly pipes and use) are compiled in order to preserve more information.

Update the existing parse function to return a list of error sites instead of 1 error site (this for now would just be the type change to prepare by making it a list and adjusting the call sites to be ready for multiple errors)

This is incidental. Fault tolerant parsing compilation analysis would make this possible, but it's not a requirement.

Does that seem generally right? With the idea being that over time we improve how granularly we put error holes/how well we handle those error holes in the lsp/diagnostics

Yes but I think this is not the majority of the work. Almost always while the programmer is typing Gleam code will parse successfully, it is analysis that will fail.

I wager we could not have fault tolerant parsing at all and so long as we are able to use information from previous analysis with the new positional information then we'll have the good experience we want.

[lpil]

Replacing with the linked issues above