Support structured diagnostics

[dylan-thinnes]

Addresses #2014

Should hopefully enable #3246

TODOs:

• Resolve TODOs in code
• Add diagnostic codes to more tests
• Make PR on GHC side so that we can eventually remove ghcide/src/Development/IDE/GHC/Compat/Driver.hs
  • Issue https://gitlab.haskell.org/ghc/ghc/-/issues/24996 and MR https://gitlab.haskell.org/ghc/ghc/-/merge_requests/12891
• Deal with CPP to make this work properly on all versions
• Fix hls-plugin-api using _code field set by withWarnings to determine which warning to use.

Additional work (future PRs):

• Replacing regexes with structured diagnostic checks
• Improve typing so that it can be known that an error has a structured diagnostic, instead of defaulting to Maybe

[michaelpj]

Possibly insane suggestion: could we write some orphan JSON instances for GHC's message types and stuff the GhcMessage into the data field on the Diagnostic? Then we don't have to carry anything else around, and we won't lose things going via the DiagnosticStore?

[dylan-thinnes]

Re: deriving JSON instances, the messages often contain fragments of AST and other things which aren't terribly friendly to ToJSON/FromJSON. I tried deriving an instance and immediately came up against

No instance for ‘ToJSON
                         (Language.Haskell.Syntax.Binds.HsBindLR
                            GHC.Hs.Extension.GhcRn GHC.Hs.Extension.GhcRn)’

which makes me think this approach is not going to be easy.

[dylan-thinnes]

Gonna take a break for today, but two big TODOs remain, and a few open questions.

[michaelpj]

Re: deriving JSON instances, the messages often contain fragments of AST and other things which aren't terribly friendly to ToJSON/FromJSON.

This tickled my brain and I remembered that GHC has its own JSON conversion functions! https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/Utils/Json.hs?ref_type=heads

And they definitely have instances for diagnostics, since they're used in the JSON output for GHC, see e.g. here: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11093

Having written this I realise that that only handles one direction, so it doesn't help us much, oh well.

[michaelpj]

They might have another serialization format, though. There's nothing stopping us from serializing them to bytes and the smuggling a bytestring as a JSON string in data 😂

[wz1000]

I don't think we should serialise the raw data over the wire, instead we should stash it in some sort of Map and send over an identifier/key which we can later use to look up the value from the Map. It requires care to ensure that the values are properly garbage collected when no longer valid though.

[michaelpj]

I don't think we should serialise the raw data over the wire, instead we should stash it in some sort of Map and send over an identifier/key which we can later use to look up the value from the Map. It requires care to ensure that the values are properly garbage collected when no longer valid though.

There's no viable serialization format anyway. I do think it would be nice if there was one, though.

That said, I don't think we should need this info in diagnostics that we e.g. get back from the client. So the main issue is when we e.g. ask for the diagnostics covering a range so we can then match on them to work out code actions. At that point we need the GHC diagnostics. So I don't think we need a separate store, at worst we might have to augment our server-side diagnostic store.

[dylan-thinnes]

One remaining point to address from review: #4311 (comment), have made a commit for it 414c845

I'm looking into the CI failures now - I assume some CPP shenanigans are at least part of the issue.

[jhrcek]

Seems like this section can now be nicely DRYied like this (also taking advantage of the fact that we recently dropped support for GHCs < 9.4, so MIN_VERSION_ghc(9,3,0) will always be true going forward):

    let closure_errs = maybeToList $ checkHomeUnitsClosed' (hsc_unit_env hscEnv') (hsc_all_home_unit_ids hscEnv')
        closure_err_to_multi_err diag = ideErrorWithSource
            (Just "cradle")
            (Just DiagnosticSeverity_Warning)
            _cfp
            (T.pack (Compat.printWithoutUniques (singleMessage diag)))
#if MIN_VERSION_ghc(9,6,1)
            (Just (fmap GhcDriverMessage diag))
#else
            Nothing
#endif
        multi_errs = map closure_err_to_multi_err closure_errs
        bad_units = OS.fromList $ concat $ do
            x <- map errMsgDiagnostic closure_errs
            DriverHomePackagesNotClosed us <- pure x
            pure us
        isBad ci = (homeUnitId_ (componentDynFlags ci)) `OS.member` bad_units