refactor: decouple diagnostics from handlers into a hook

[radeksimko]

Background

Originally we would only publish diagnostics directly from the relevant (RPC request) handlers, such as from textDocument/didOpen, textDocument/didChange or textDocument/didClose.

This in principle introduced the requirement on the task scheduler to support blocking/synchronous execution as we didn't want to be publishing stale diagnostics before the parsing and decoding is actually finished. However, this execution combined with the amount of operations we need to run in different contexts makes this a complex implementation hard to reason about and harder to maintain. Consequently there is at least one bug 🐛 where duplicate operations are attempted to be scheduled and one of them will remain blocking forever, effectively causing a deadlock 🙀 . This bug was first discovered in #700 which adds more operations to ensure cross-module references are discovered and therefore makes the deadlock situation more likely to occur.

This PR therefore aims to remove the whole concept of blocking/sync execution and assume that all operations will be executed asynchronously.

UX Implications

This comes with a (subtle?) UX difference where we would publish diagnostics for all files as they are indexed (and changed), not just the ones which are open on the client. We could still provide the existing UX if we wanted to, but there's no clear benefit we'd gain by the additional complexity.

Also we now avoid publishing any diagnostics for non-autoloaded tfvars files, which more or less aligns the behaviour with gopls ' treatment of Go build tags. I will create a separate issue to track this problem.

See #715

LSP

There is currently no clear guidance in LSP spec around whether servers should publish diagnostics for unopened files. The only related part of the spec talks about clearing of diagnostics on close:

Diagnostics are “owned” by the server so it is the server’s responsibility to clear them if necessary. The following rule is used for VS Code servers that generate diagnostics:

• if a language is single file only (for example HTML) then diagnostics are cleared by the server when the file is closed. Please note that open / close events don’t necessarily reflect what the user sees in the user interface. These events are ownership events. So with the current version of the specification it is possible that problems are not cleared although the file is not visible in the user interface since the client has not closed the file yet.
• if a language has a project system (for example C#) diagnostics are not cleared when a file closes. When a project is opened all diagnostics for all files are recomputed (or read from a cache).

When a file changes it is the server’s responsibility to re-compute diagnostics and push them to the client. If the computed set is empty it has to push the empty array to clear former diagnostics. Newly pushed diagnostics always replace previously pushed diagnostics. There is no merging that happens on the client side.

Language Clients & gopls

Although LSP doesn't provide clear guidance, it is clear that sending diagnostics for unopened files is a common practice in gopls. Clients account for this possibility differently, probably catering different user bases with different expectations:

• Sublime Text LSP will hide any diagnostics which are not for unopened files
• VS Code will display all diagnostics

Future TODOs

Versioned diagnostics

LSP allows us to send diagnostics with document versions, which presumably aids the client as it can just drop the ones which are associated with an old version of the document and effectively prevent ever publishing a stale diagnostic.

Attaching the document version at this point would not be trivial as we'd probably need to start tracking document versions in memdb alongside diagnostics. This PR therefore does increase the likelihood of us publishing outdated diagnostics, but they would always get updated/cleared as part of another module change hook execution anyway, so it's rather a "temporary blip".

See #716

Only publishing changed diagnostics

In an ideal scenario we would only publish diagnostics when there are different ones to publish and avoid publishing the same ones multiple times. In order to achieve this however we would need to check whether all diagnostics are equal. Currently we use the upstream hcl.Diagnostic to represent a diagnostic and the upstream implementation does not have an equality function itself. We could implement all this relatively easily downstream, but we can't actually take advantage of it until we figure out a better way of publishing and clearing validate diagnostics, which currently get just cleared whenever other diagnostics are published, to avoid stale diagnostics.

See #717

[rchl]

* Sublime Text LSP will hide any diagnostics which are not for unopened files

This is no longer true (fixed by sublimelsp/LSP#1881).

ST LSP still hides diagnostics for files that are outside of the workspace but that's also not entirely wanted so might change in the future.

[radeksimko]

@rchl Thanks for the note, that further supports the theory that people do want to see diagnostics even for unopened files then 👍🏻

[dbanck]

Awesome work 🌟 The diagnostics have improved and there is even less code now

There is one strange behavior I need to show you, please don't merge yet

[dbanck]

7611112 fixes the before mentioned strange behavior 🎉

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.