Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full)

[soulomoon]

Working out #1654
specification: specification

motivation

It enhance the original syntax highlighting by differing tokens semanticly.

What is implemented

• The current implementation only textDocument/semanticTokens/full.
• [future] textDocument/semanticTokens/full/delta and workspace/semanticTokens/refresh Not planning in this pull request
  we need to store last computed tokens for files in Ide state. I will try to implement them in the future after getting a deeper understanding of how hls works.

Features:

• Supported PluginMethodHandler

  • textDocument/semanticTokens/full.
  • [future] textDocument/semanticTokens/full/delta
  • [future] workspace/semanticTokens/refresh

• Supported semantic tokens type:

  • class and class method
  • type family name (data family)
  • data constructor name (not distinguishing record and normal data, and GADT)
  • type constructor name (GADT)
  • record field name
  • type synonym
  • pattern synonym
  • pattern bindings In favor of differing functions and none-functions from its type
  • value bindings In favor of differing functions and none-functions from its type
  • functions
  • none-function variables
  • imported name

• Supported modifiers(planning):

  • [future] declaration (as in class declearations, type definition and type family)
  • [future] definition (as in class instance declaration, left hand side value binding, and type family instance)
  • [future] modification (as in rec field update)

• Support unicode

Implementation details:

• Compute visible names from renamedsource
• Compute NameSemanticMap for imported and top level name tokens using HscEnv(with deps) and type checked result
• Compute current module NameSemanticMap using RefMap a from the result of GetHieAst
• Compute all visible (Name, Span) in current module, in turn compute their semantic token using the combination map of the above two NameSemanticMap
• use default legends, Candidates map of token type with default token type: Maps to default token types
• add args support to turn the plugin on and off
• enhence test Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892 (comment)
• enhence error reporting. Implement semantic tokens plugin to support semantic highlighting(textDocument/semanticTokens/full) #3892 (comment)
• [future] make use of modifiers
• [future] hadling customize legends using server capabilities (how?)

The focus of this implementation of semantic tokens is to support semantic highlight.
modifiers does not seems to provided as much use in pure language as in impure language.
I decided to ignore the modifier support for now, and schedule to the future.

Ps
I am using the following configuration to test the stuff out in vscode.

    "editor.semanticTokenColorCustomizations": {
        "enabled": true, // enable for all themes
        "rules": {
            "interface": {
                "foreground": "#daf049"
            }
            ,
            "type": {
                "foreground": "#f81eba"
            }
            ,
            "typeParameter": {
                "foreground": "#cd4ca0"
            }
            ,
            "property": {
                "foreground": "#f81e"
            },
            "method": {
                "foreground": "#1ef1f8"
            },
            "class": {
                "foreground": "#ffe100"
            }
        }
    },

[soulomoon]

I would be great if we could transfer some of the reasoning you've written in the PR into code comments. In particular, we've had a bit of discussion about:

1. What token types to use for what kinds of Haskell entity
2. What information we can and can't get from what sources

This seems likely to be useful to future people working on this plugin and will get lost if it's just in PR comments.

sure, I'll collect them some times later

[Bodigrim]

but since text reading involved, the conversion taken place one by one is not efficient, maybe we can create a position mapping for the whole file at one. And cache that mapping to do the conversion instead.

To avoid performance degradation text-rope supports mixed indexing mode in Data.Text.Utf16.Rope.Mixed. It would be great if someone revived haskell/lsp#436 and switched lsp to use it.

[soulomoon]

but since text reading involved, the conversion taken place one by one is not efficient, maybe we can create a position mapping for the whole file at one. And cache that mapping to do the conversion instead.

To avoid performance degradation text-rope supports mixed indexing mode in Data.Text.Utf16.Rope.Mixed. It would be great if someone revived haskell/lsp#436 and switched lsp to use it.

Sure, I can take a look at them in the near future to see if I could be helpful.

[michaelpj]

Could you make this disabled by default, like the stan plugin here? #3917

I think given how big and new this is it's sensible to do that for one release and get some testing.

Then I think this is close to being mergeable. I don't know if there's more tidying you want to do - maybe a bit more documentation/cleaning up commented out code etc.

And we should remember to make issues for the bits of future work you have in mind.

[soulomoon]

Could you make this disabled by default, like the stan plugin here? #3917

I think given how big and new this is it's sensible to do that for one release and get some testing.

Then I think this is close to being mergeable. I don't know if there's more tidying you want to do - maybe a bit more documentation/cleaning up commented out code etc.

Yep, Plugin is off by default now and tidying is done.

And we should remember to make issues for the bits of future work you have in mind.

definitely, I will open issues to track them down.

[soulomoon]

track list have been made in #3931

[soulomoon]

it seems a [Required] test is failling accidentally. Maybe it need to be rerun @michaelpj

ghcide
  diagnostics
    typecheck-all-parents-of-interest: FAIL (0.36s)
      test\exe\DiagnosticTests.hs:493: