Generate types from metamodel #458

michaelpj · 2022-09-19T21:27:32Z

This generates all the LSP types from the LSP metamodel, a new machine-readable format they have produced.

This requires a pretty formidable amount of TH, but in return we get a hopefully much better maintenance process, and we won't always be behind or wrong as we have been. There are also some nice touches, like on GHC >9.2
we can now actually attach doucmentation to things we generate with
TH, so we can transfer all the documentation from the metamodel to Haddock.

This is going to result in a lot of churn, enough so that I think it should be a new top-major version of lsp-types. In particular:

Various inconsistencies with the spec got fixed.
We aim for scrupulously matching the spec, even when that is weird, so there's a lot more A |? Null rather than Maybe A than there used to be.
The new version makes heavy use of anonymous records to handle the anonymous structure literals in the spec.
I took a fairly heavy-handed approach to avoiding name clashes in a reasonably future-proof way, but that means that a bunch of names changed, notably the constructors of Method/SMethod.

This still needs a bunch of work:

lsp-test tests are broken for some reason
More documentation and tidying in CodeGen
Changelog
Explain how to update the metamodel in the README
Tidy up the generated documentation so that links go to the generated types not the LSP spec.
Ensure everything builds on all versions of GHC we support.

Plus a bunch of other TODOs in the code. Any help would be welcome, but here are the top things I'm interested in feedback on:

How does the new version look? Is the organization okay? I'll try and get some Haddock with the new documentation in place soonish so people can look at that without building it.
The gnarly TH. Anyone who wants to dive in is more than welcome, it's not too bad but it would be nice if we could figure out a way to make it less gnarly.

This generates all the LSP types from the LSP metamodel, a new machine-readable format they have produced. This requires a pretty formidable amount of TH, but in return we get a hopefully much better maintenance process, and we won't always be behind or wrong as we have been. There are also some nice touches, like on GHC >9.2 we can now actually attach doucmentation to things we generate with TH, so we can transfer all the documentation from the metamodel to Haddock. This is going to result in a _lot_ of churn, enough so that I think it should be a new top-major version of `lsp-types`. In particular: - Various inconsistencies with the spec got fixed. - We aim for scrupulously matching the spec, even when that is weird, so there's a lot more `A |? Null` rather than `Maybe A` than there used to be. - The new version makes heavy use of anonymous records to handle the anonymous structure literals in the spec. - I took a fairly heavy-handed approach to avoiding name clashes in a reasonably future-proof way, but that means that a bunch of names changed, notably the constructors of `Method`/`SMethod`. This still needs a bunch of work: - [ ] `lsp-test` tests are broken for some reason - [ ] More documentation and tidying in `CodeGen` - [ ] Changelog - [ ] Explain how to update the metamodel in the README - [ ] Tidy up the generated documentation so that links go to the generated types not the LSP spec. - [ ] Ensure everything builds on all versions of GHC we support. Plus a bunch of other TODOs in the code. Any help would be welcome, but here are the top things I'm interested in feedback on: 1. How does the new version look? Is the organization okay? I'll try and get some Haddock with the new documentation in place soonish so people can look at that without building it. 2. The gnarly TH. Anyone who wants to dive in is more than welcome, it's not _too_ bad but it would be nice if we could figure out a way to make it less gnarly.

michaelpj · 2022-09-19T21:28:10Z

@thomasjm @drsooch might be of interest

michaelpj · 2022-09-21T11:54:06Z

@sjakobi I think you're the main maintainer of the dhall language server, which is one of the main non-HLS users of this package that I know of. I'd really appreciate your thoughts!

michaelpj · 2022-09-21T13:47:38Z

Guide for reviewers:

Metamodel stuff is in the Metamodel hierarchy, including the types for reading the metamodel and the code generation itself, which is the most gnarly part
- The easiest way to review the generated code at the moment is either to build the haddock or to use -ddump-splices on the Generated module.
Most of the rest of the Language.LSP.Types hierarchy is just a cut down version of what was there before due to much of it being generated
- There is some new stuff in Message to handle some typed/untyped versions of things
Everything in the rest of the project is just adapting to the changes. Might be interesting to take a glance at to see the sort of changes that this will cause.

sjakobi · 2022-09-26T11:42:34Z

@sjakobi I think you're the main maintainer of the dhall language server, which is one of the main non-HLS users of this package that I know of. I'd really appreciate your thoughts!

Unfortunately, I'm not very familiar with the dhall-lsp-server internals at all – I've mostly been bumping bounds. Maybe @Gabriella439 has thoughts on this change.

Gabriella439 · 2022-09-26T23:34:13Z

My take is that you should just do whatever you need to do. We'll figure out a way to adapt downstream

drsooch · 2022-09-30T02:26:02Z

lsp-test/src/Language/LSP/Test.hs


 -- | Applies formatting to the specified document.
 formatDoc :: TextDocumentIdentifier -> FormattingOptions -> Session ()
 formatDoc doc opts = do
  let params = DocumentFormattingParams Nothing doc opts
-  edits <- getResponseResult <$> request STextDocumentFormatting params
+  edits <- absorbNull . getResponseResult <$> request SMethod_TextDocumentFormatting params


I know this is lsp-test code but this line pops up a bunch

drsooch · 2022-09-30T03:37:38Z

lsp-types/src/Language/LSP/MetaModel/Types.hs

+import           Data.Aeson         hiding (Null, String)
+import qualified Data.Aeson         as JSON
+import qualified Data.Aeson.TH      as JSON


nit: some mixture between qualified and unqualified aeson functions in this file

drsooch · 2022-10-02T02:46:41Z

lsp-types/src/Language/LSP/MetaModel/CodeGen.hs

+    Nothing -> fail $ "no name for " ++ show name
+  addDocumentation tn documentation since proposed
+
+  ctor <- lift $ makeToplevelName name


can just use tn here? Or does it have to be a new name? or does it even matter?

Then in genMetaModel no need to un-mangle names?

nvm it breaks TH - got it

drsooch · 2022-10-02T19:23:43Z

lsp-types/src/Language/LSP/MetaModel/CodeGen.hs

+-- | Generate a type declaration corresponding to a top-level named struct.
+genStruct :: Structure -> CodeGenM [TH.Dec]
+genStruct s@Structure{name, documentation, since, proposed} = do
+  let structName = name


nit: structName is used in one place only

drsooch · 2022-10-02T19:43:15Z

lsp-types/src/Language/LSP/MetaModel/CodeGen.hs

+
+  -- Result type is Maybe unless X allows custom values, in which case we can always interpret
+  -- a base value as a custom member
+  -- valueToX :: <base type> -> Maybe X


do we want a partial function for non-custom enums?

drsooch · 2022-10-02T19:53:02Z

lsp-types/src/Language/LSP/MetaModel/Types.hs

+    "both"           -> pure Both
+    t                -> fail $ "unknown message direction " ++ show t
+
+data BaseTypeName = URI | DocumentUri | Integer | UInteger | Decimal | RegExp | String | Boolean | Null


nit: URI -> Uri :) Unless there is some name clashing?

drsooch · 2022-10-02T20:10:00Z

lsp-types/src/Language/LSP/MetaModel/CodeGen.hs

+      n <- lift $ TH.newName "arg"
+      pairE <- case optional of
+        Just True -> lift [| $(TH.litE $ TH.stringL $ T.unpack name) .=? $(TH.varE n) |]
+        _ -> lift [| [$(TH.litE $ TH.stringL $ T.unpack name) J..= $(TH.varE n)] |]


The spec defining "optional" to be False if it's omitted is confusing and weird... but it's in the spec so

drsooch · 2022-10-02T20:17:31Z

lsp-types/src/Language/LSP/MetaModel/CodeGen.hs

+    extends' = fromMaybe [] extends
+    mixins' = fromMaybe [] mixins
+    supertypes = extends' ++ mixins'


I wonder if it's ever possible for an extended type or mixin type to have duplicate fields? Is it worth de-duping superProps (aside from local name de-duping)

drsooch · 2022-10-02T22:25:45Z

lsp-types/src/Language/LSP/Types/Capabilities.hs

+
+{-
+TODO: this is out-of-date/needs an audit
+TODO: can we generate this? process the 'since' annotations in the metamodel?


Seems like a good idea to generate this as well. I'm going cross-eyed looking at it. I can give it a shot if you want

drsooch · 2022-10-07T00:48:16Z

I honestly, don't think there is much to change. The TH generation seems sensible, I don't think there's much to change honestly. Codegen is pretty much straightforward, no matter what language you are using. I made some general code nit picks that honestly you can just ignore. There were one or two actual suggestions that I think are worth it. Otherwise lgtm.

drsooch · 2022-10-07T00:52:00Z

lsp-types/src/Language/LSP/MetaModel/Types.hs

+
+data Request = Request
+  { method              :: Text
+  , params              :: Maybe Type -- typescript says it can be [Type], but it never is so whatever


I'd argue this should match the spec, better to match the spec and reduce churn later imo

drsooch · 2022-10-07T00:55:44Z

lsp-types/src/Language/LSP/MetaModel/CodeGen.hs

+sanitizeName n =
+  -- Names can't start with underscores! Replace that with a 'U' for lack
+  -- of a better idea
+  let n' = if "_" `T.isPrefixOf` n then T.cons 'U' $ T.tail n else n


What about Abstract or Base or Internal or something like that. Looks like most usages are "abstract" classes in the traditional OOP sense.

drsooch · 2022-10-07T01:16:06Z

Also not sure why the operator-like constructors appear like this on my Haddock. It seems to just be for Record Fields

July541 · 2023-03-19T04:53:36Z

Any update here? I'm going to address some lsp-test parts, some new features from 3.16 spec attracted me.

michaelpj · 2023-03-22T18:21:49Z

I have a partially-rewritten version that generates Haskell source instead. I wasn't able to get this version to work with GHC 9, and the thought of trying to make it work with multiple GHC versions made me scared. So I decided to switch to generating source. I need to finish it, basically...

michaelpj · 2023-05-19T15:18:22Z

Closing in favour of #478

This was referenced Sep 19, 2022

Generate types via new metaModel.json #421

Closed

How to improve the documentation of lsp-types #419

Open

drsooch reviewed Sep 30, 2022

View reviewed changes

drsooch reviewed Oct 2, 2022

View reviewed changes

drsooch reviewed Oct 7, 2022

View reviewed changes

michaelpj mentioned this pull request Dec 9, 2022

Consider switching to microlens or overloaded dot syntax? #465

Open

michaelpj mentioned this pull request May 19, 2023

Generate types from the metamodel (attempt 2) #478

Merged

michaelpj closed this May 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate types from metamodel #458

Generate types from metamodel #458

michaelpj commented Sep 19, 2022 •

edited

michaelpj commented Sep 19, 2022

michaelpj commented Sep 21, 2022

michaelpj commented Sep 21, 2022

sjakobi commented Sep 26, 2022

Gabriella439 commented Sep 26, 2022

drsooch Sep 30, 2022

drsooch Sep 30, 2022

drsooch Oct 2, 2022

drsooch Oct 2, 2022

drsooch Oct 2, 2022

drsooch Oct 2, 2022

drsooch Oct 2, 2022 •

edited

drsooch Oct 2, 2022

drsooch Oct 2, 2022

drsooch Oct 2, 2022

drsooch Oct 2, 2022

drsooch commented Oct 7, 2022

drsooch Oct 7, 2022

drsooch Oct 7, 2022

drsooch commented Oct 7, 2022

July541 commented Mar 19, 2023

michaelpj commented Mar 22, 2023

michaelpj commented May 19, 2023

Generate types from metamodel #458

Generate types from metamodel #458

Conversation

michaelpj commented Sep 19, 2022 • edited

michaelpj commented Sep 19, 2022

michaelpj commented Sep 21, 2022

michaelpj commented Sep 21, 2022

sjakobi commented Sep 26, 2022

Gabriella439 commented Sep 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drsooch Oct 2, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drsooch commented Oct 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drsooch commented Oct 7, 2022

July541 commented Mar 19, 2023

michaelpj commented Mar 22, 2023

michaelpj commented May 19, 2023

michaelpj commented Sep 19, 2022 •

edited

drsooch Oct 2, 2022 •

edited