Representation of arbitrary types for a plugin

[alanz]

My thinking is that we have an "inner" and an "outer" protocol. The inner/logical one is processed by all the plugins, the outer one comes about when a specific inner message is wrapped in a transport for delivery on the wire.

A possible version of the inner protocol is defined in https://github.com/haskell/haskell-ide-engine/blob/plugins-definitions-play/haskell-ide-plugin-api/Haskell/Ide/PluginDescriptor.hs.

A IdeRequest is defined as

data IdeRequest = IdeRequest
  { ideCommand :: CommandName
  , ideSession :: SessionContext
  , ideContext :: CommandContext
  , ideParams  :: Map.Map ParamId ParamVal
  } deriving Show

type ParamId = String
type ParamVal = String

This initially expects any additional parameters required for a command are String values.

The response is defined as

data IdeResponse = IdeResponseOk String -- ^ Command Succeeded
                 | IdeResponseFail String -- ^ Command Failed
                 | IdeResponseError String -- ^ some error in haskell-ide-engine
                                           -- driver. Equivalent to HTTP 500
                                           -- status
                 deriving Show

The String type is a problem.

What is a good way to represent the value for the IdeResponse inner protocol so that it can be serialised/deserialied via an arbitrary transport mechanism, such as JSON, MSGPACK, etc.

[saep]

Wouldn't it be easier if the inner protocol is plain Haskell and the outer protocol is something that can easily be called from any language?

Since I'm familiar with the msgpack-rpc protocol, I'll elaborate what I mean with that. Assuming the IDE process contains plugins that run a linter on a file and another one that gives completion hints for a partial identifier.

type Error = Text

data LintSuggestion = LintSuggestion 
    { file :: FilePath
    , line :: Int
    , column :: Int
    , suggestion :: Text
    } deriving (MSGPACK) -- The instance will just be a map with the record fields as keys.

lintFiles :: [FilePath] -> Either Error [LintSuggestions]

-- | Apply the given suggestion. Make sure to reload the file in the editor when this function returns without an error. 
applyLintSuggestion :: LintSuggestion -> Either Error ()

data CompletionSuggestion = CompletionSuggestion
    { completion :: Text
    , doc :: HTML -- ^ i.e. 'Text' in html format (This is just an example.) 
    } deriving (MSGPACK)

-- | Return a list of possible completions for the given partial identifier 
completions :: FilePath -- ^ absolute filepath to determine the context
            -> Text -- ^ partial identifier
            -> Either Error [CompletionSuggestion]

(The result type probably has to be wrapped in an ReaderT Something IO-ish monad in which Something stores appropriate objects to dispatch the function calls to the desired , but let's ignore that part for now.)

The cabal/stack/.. context should be deferrable from the filepath given to the arguments of the functions, but if more specific metadata is required, I would simply put it as required arguments.

You can write a few lines of general documentation that explains how record types and result types are mapped to MSGPACK, and let haddock take care of the rest of the documentation.

Say, you want to call lintFiles from vimL. It would look similar to this:

try
let lintSuggestions = callRemoteFunction(haskellIDE, "lintFiles", [ expand('%:p') ])
for s in lintSuggestions
   setmarker(s['file'], s['line'], s['suggestion']) 
endfor
endtry

What do we get for this specific case over the transport? According to the msgpack-rpc specification, we get a 4 element array which contains an integer value that identifies the message as a function call, an integer which serves as an identifier to match the result to the function call, the name of the function that has to be called and finally an array/list of function arguments.
So assuming that Object represents a MSGPACK object, we must find a way to execute the function identified by its name and apply it to the transmitted arguments. In essence, we must translate a Haskell function like the ones above to a function of type: [Object] -> Object. Fortunately, this is not too difficult and I have done it here already.

So, technically it shouldn't be a problem to use Haskell functions rather directly instead of defining a lot of boilerplate data types that have to be handled within Haskell and by the frontends. The arguments to the externally exposed Haskell functions are merely limited to primitive types (e.g. Text, Int, FilePath) and types that can be represented as maps (e.g. LintSuggestion, CompletionSuggestion, Map Text Object).

Although I have specifically used MSGPACK as a example here, JSON or other formats can easily be used here as well.

[bitemyapp]

@saep tend to agree. Rather plain-old-Haskell interally with an HTTP+JSON API exposed for clients.

[alanz]

The intention is that the inner protocol is just plain haskell. But the plugin does not communicate with the IDE directly, this goes through a dispatcher/router first.

So the idea is to come up with a common interface that any plugin can honour, to be able to work within the router/dispatcher. As such there needs to be a common type for the request to the plugin and for the response from the plugin.

In my POC these are IdeRequest and IdeResponse.

@gracjan has suggested that we use Data.Aeson.Value for this, because it is is general and represents types that can be converted.

Each transport could then provide a means to serialize/deserialize the Value to the specific wire format, be it MSGPACK, SExps, JSON, or something else.

[alanz]

The current POC uses Data.Aeson.Value as the internal data structure

[JPMoresmau]

Could we have a more explicit structure for Fail and Error, so that we avoid the "let's parse some cryptic error message to try to have tools deal with it" problem we have with ghc? I think a data type that contains the plugin name, command name, an error code (an arbitrary text), explicit parameters and finally a human readable error would be good.

[alanz]

The intention is that the IdeResponse is tied to the IdeRequest via the RequestId

type RequestId = Int

data ChannelRequest = CReq
  { cinPlugin    :: PluginId
  , cinReqId     :: RequestId -- ^An identifier for the request, can tie back to
                              -- e.g. a promise id. It is returned with the
                              -- ChannelResponse.
  , cinReq       :: IdeRequest
  , cinReplyChan :: Chan ChannelResponse
  } deriving Show

data ChannelResponse = CResp
  { couPlugin :: PluginId
  , coutReqId :: RequestId
  , coutResp  :: IdeResponse
  } deriving Show

This sorts out

• the plugin name,
• command name,
• explicit parameters

The distinction between

• an error code (an arbitrary text),
• a human readable error

needs to be decided/managed.

Of course this assumes that the transport used makes the RequestId visible for the particular request/response interaction.

[JPMoresmau]

Ah, ok. Because I was trying the console, to run ghc-mod:check in it, and it wasn't pleasant, because of

error $ "GhcModPlugin.checkCmd: got unexpected file param:" ++ show x

And the console I think parses all parameters as ParamText, so we always fail. So here we use error instead of returning IDEResponseFail and we repeat the plugin and command name. What I means by parameters is that we would need to outline here that the file parameter is the one that is wrong, since our command could have several. So really we should have an standardize way of building errors, in that case passing file and the actual parsed parameter. I suppose then the console could show the pluginid and request information on errors.

[alanz]

The console is basically useless at the moment. Not sure if the head version still has Context in the IdeRequest, but there is no way to set it in the console.

The new version treats them just as params, so should work better. See my PR from a short while ago

[alanz]

Closing this as it was basically a discussion, and we now have a working way of doing it, via ValidResponse