[Tracking] Reduce LSP Memory Footprint

[JoshuaBatty]

Tracking issue for reducing the large RAM usage of the language server.

• Store compressed data in the language servers memory #6221
• Use a LRU mechanism for “forgetting” old sessions to reduce RAM usage for large workspaces. #6222
• Audit the bytecode size for everything that the server stores #6223
• Reduce the complexity of data we store for each token #6224
• Consider only storing interned values of types rather that the whole AstNode #6225
• forc-lsp consuming a ton of memory #5645
• Allow a single Engines to be used across a workspace #5856
• Use a string interner in the compiler to reduce memory footprint #6272

[tritao]

Might be interesting to look into https://docs.rs/cstree/latest/cstree/, which implements some of the syntax tree design approaches initially presented by Roslyn (C# compiler) and Swift libsyntax.

Specifically:

cstree trees are trees over interned strings. This means cstree will deduplicate the text of tokens with the same source string, such as identifiers with the same name. In this position, rowan stores each token’s text together with its metadata as a custom DST (dynamically-sized type).

It would allow us to better intern and re-use parts of our syntax tree nodes.