Support interactive code execution

[bennofs]

Right now, when using ghc-mod, I still have an GHCi instance open because I like to test my code in the repl. But this means that I need to have 2 instances of GHC running! Also, the features of each overlap: ghc-modi does syntax checking, but GHCi can also do that. However, GHCi doesn't have such a good emacs integration and also doesn't support so many completion-related features.

So I'd really like to see support for interactive code execution like in GHCi in ghci-modi. This would allow me to avoid running an extra GHCi instance.

[DanielG]

It's funny you should mention the overlap between ghc-mod and ghci right now, I was talking to @alanz about that.

Honestly at this point I'm questioning why we're not just trying to contribute the missing features to GHC/ghci, adding network RPC support and porting the emacs frontend to that.

[bennofs]

Yeah, I also came to that conclusion after trying to write my own on-the-fly syntax checker a few times :) But on the other hand, it might be a good idea to separate the development of ghc-mod from main GHC, to allow it to move faster than GHC releases.

[DanielG]

We could still do something along the lines of ghci-ng.

[DanielG]

(i.e. backporting features form GHC HEAD into a standalone package)

[DanielG]

Ok I've been thinking about this a lot and I think it's best to just go forward with ghc-mod development instead of trying to add stuff to ghci.

How about this:

We add network RPC capabilities to ghc-mod and then develop a frontend that looks exactly like ghci, so tools that use ghci now can just have the ghci executable name modified to use the ghc-mod frontend. (I hope that makes sense, I'm tired ;)

[bennofs]

Why do you need network RPC in ghc-mod itself? In my experience, having network RPC is a big source of bugs and I would try to avoid that.

[DanielG]

Well how else would you do it? We need some way that ghc-mod can communicate with multiple other processes at the same time.. stdio just doesn't cut it for that (unless we spawn a netcat process or something). By RPC I really just mean a request-response protocol that communicates the command type and arguments, nothing fancy.

[bennofs]

Let the editor do the session managment? It has much more information about projects, etc. That's how cabal repl with haskell-mode in emacs currently works. Then you only need to spawn one non-daemon process per session, no RPC necessary. Then you don't even need cabal support in ghc-mod: We'd only need a --with-ghci option for cabal repl, so you can do cabal repl --with-ghci=ghc-mod. This will also handle preprocessing of .hsc files and compiling / linking of necessary c files better.

[DanielG]

And how does ghc-mod communicate with the daemon then?

[bennofs]

Emacs will just spawn one ghc-mod process per session, and then send commands via stdout / stdin.

[DanielG]

But then we're wasting resources again. The ghc-mod frontend would have to communicate with the daemon that actually does all the GHC API interaction.

I really like the --with-ghci idea though that literally removes the need for any cabal support in ghc-mod.

[DanielG]

I think cabal repl spawns ghci via ghc's --interactive flag, so we just need to pass cabal repl --with-ghc=ghc-mod after we support the options cabal passes to us then this should work.

[DanielG]

TODO:

• Let ghc-mod be driven by cabal repl so we don't even have to mess around with cabal
• Add a ghci compatible frontend so tools supporting ghci can be effortlessly switched to use ghc-mod

[RubenAstudillo]

I am kind of against this. Sure running two instances of ghc (ghc-mod, ghci) is bad, but this blurs the line between ghci and ghc-mod. I had in my mind separated use cases for each of them:

• Ghci: interactive evaluation of expressions and programs plus a debugger for projects.
• Ghc-Mod: Contextual information of code (even malformed one).

But that was just me. If we don't have a specific goal for Ghc-mod we can end up being a jack of all trades in hopes of no duplicating running the ghc-api.

[DanielG]

The line between ghci and ghc-mod is already pretty blurred.

• syntax checking: ghci does it
• type info: ghci does it
• browsing modules: ghci does it
• etc.

There are only a few things that ghc-mod can do that ghci can't. Getting the type of an arbitrary subexpressions as instantiated by the surrounding code comes to mind. Because of this overlap I like to think of ghc-mod as more of an extension to ghci.

In theory nothing is stopping ghci from supporting all the operations its missing compared to ghc-mod but I think the problem is that ghci is too closely coupled to GHC's development which makes 1) trying to modify it rather intimidating and 2) makes it somewhat inflexible (for example the ghc developers don't like introducing new dependencies).

If it were possible to add all the niceness that we need to conviniently do haskell development to ghci that'd be great! Everyone using ghc would automatically get all that stuff, no need to install anything, just awesome. Alas because of the way ghci is developed I don't think that is feasible.

So what should we do if we want new, complex features in ghci but can't actually add them there? This is, in my mind, exactly where ghc-mod should come in.

In my vision ghc-mod would provide an interface that looks and behaves exactly like ghci BUT supports additional operations (like our type command). This way we could almost completely deprecated ghc-mod's Emacs/Vim frontends and just contribute support for ghc-mod specific features to existing ghci based tools.

This is why I'm for interactive code evaluation, so we can pretend to be ghci but with more features.

[RubenAstudillo]

You make a good case, the overlap exist and the different use cases were just in my head. Seeing ghc-mod as as a better ghci with editor-friendly features serves as a good goal. On that note makes total sense having interactive code execution.

I will make myself familiar with ghc-api needed to implement this :-).

[DanielG]

Just look at the ghci source code. The interresting bits are in the GHC repo in ghc/InteractiveUI.hs and ghc/GhciMonad.hs.

Also you should swing by IRC if you want to get stuff moving ;)

[RubenAstudillo]

It seems our local times don't match :-(.

About the interactive execution, I've been reading the InteractiveUI.hs code and getting familiar with it. I think I can help in the implementation of this, but I want to be sure what are going to implement. Just loading local modules and evaluating expressions on them? What about interactive debugging? What is the scope of features that we ought to implement.

When this is answered we could start a branch on mine or yours repo and start working on it.

[DanielG]

Didn't you get my email? Well I'm around now if you have time :)

My plan is to basically copy ghci in terms of features. Maybe we leave out debugging for now but pretty much everything else.

[RubenAstudillo]

I am in IRC #ghc-mod right now!

[RubenAstudillo]

I tried importing the ghci api into the ghc-mod tree. I did a script that grabbed the interesting files of the ghc tree for interactive execution (GhciTags.hs InteractiveUI.hs GhciMonad.hs) extended the export list, change the qualified names in the imported modules. The script is simply a mess :-) , is basically a 5 piped sed command together, but it works kind of reliably.

The imported files depend on an include HsVersion.h which in turn depend on ghcautoconf.h. This last file is generated at compile time in the ghc tree and is necessary to declare the size of different data-types as Int. If we want to support interactive execution as ghci with the same interface and semantics we will have to complicate our build system to support files like that (we can't use the same file everywhere). The alternative is abandon the goal of being semantically like ghci and implement a similar kind of look & feel but implemented all on top of InteractiveEval which is a module exported by ghc and thus will all the compile time options already set.

This obviously is just my experience, would love to be proven wrong :-)

[DanielG]

I'm pretty sure InteractiveUI doesn't actually use most of the stuff in HsVersion.h ghci-ng just has a dummy version of that file (https://github.com/hvr/ghci-ng/blob/master/ghc/HsVersions.h). Can't you just use that?

The part of the ghci source we're trying to import is only the frontend driver, it doesn't do any code generation or anything fancy it just calls the GHC API and tells it what to do. So it shouldn't need those constants.

[RubenAstudillo]

You're right, I can use just the HsVersion.h from ghci-ng. It works with little errors.

[DanielG]

Cool :)

[DanielG]

Damn it looks like we can't do this after all. haskelline pulls in terminfo as a dependency which needs the native library libncurses-dev, which I really don't want to be a requirement for installing ghc-mod :/

On the brighter side I looked more closely at the ghci source and it looks like most of the functions are in the InputT/Ghci monads and don't return anything useful anyways so maybe outright copying the code is the best we can do anyways.

@RubenAstudillo sorry I didn't realize this before :x

[RubenAstudillo]

It's OK, I'm glad you caught that one, I didn't like all the extensions we had to put anyways. I learned a lot.

[DanielG]

ETIMEDOUT, I have no time to implement this. If someone else steps up we could have another run at this but otherwise I don't see it happening any time soon.