Support for Common Lisp

[smolck]

Hello! So after seeing #92 and starting to use common lisp, I'd really like to implement a client for common lisp myself (that I could PR here) because I love the dev experience with Conjure, but I have some questions about that which hopefully you wouldn't mind answering?

1. Would you accept a PR adding common lisp support? If not I can create a separate plugin for it, but if you'd merge it here I'd like to do that.
2. How would you go about implementing a common lisp client? My idea was to create a repl (say via sbcl) and then communicate back and forth with that. Would that work, or is there a different/better approach?

I know there are plugins like vim-slime and such, but most of these seem old or outdated and they also don't make use of Neovim-specific features like floating windows (those are Neovim-specific, right?), unlike Conjure. So I'd much prefer to have a common lisp client for Conjure than use one of those.

[Olical]

Hello! This is really exciting, I've been hoping someone would show interest in adding a client for a while 😄

Q1

Absolutely, 100%. As long as it's in line with the other clients and works well / has a good UX I'm more than happy to add it. I'm okay with a PR or a plugin, whatever works for you really.

If you go down the PR route you get to lean on everything that's already set up in the Conjure repo with regards to Aniseed Fennel compiling. If you write your own plugin (presumably with Aniseed too) you have more control over the marketing / readme / docs / tooling side. What do you think you'd prefer?

Since this is the first time someone has approached me about adding a new client I'm not sure on the best course. I have a feeling just integrating everything directly into Conjure is the way to go for ease of use by users. I'm leaning towards a fork that is eventually merged in.

Q2

So I try to find some sort of socket based transport for each language, like netrepl for Janet and nREPL for Clojure. This way the user can start the process with all of the env vars and magical things they require, they can even start it in docker! Then the client just uses some port file or known good default port to try and connect when you enter the filetype for the first time.

I think starting sbcl under Neovim and reading/writing to that process is doable and I'd use that as a last resort if there's no good known network REPL solution in that language's community. I want to encourage getting your project running with a dev hook in it, then you start Neovim on the side and it automatically connects.

If Neovim starts the process you then tie them together, it becomes "jack in" support pretty quickly which requires so much config for users to get things running just how they want them. I want jack in one day but I'm putting it off as long as possible.

I noticed https://github.com/sjl/cl-nrepl, I'll tweet sjl and see what their thoughts are on networked CL now. I don't think I've ever run a single form of CL so it's still a bit of a mystery land to me!

Config side note

I will be rewriting the config system some time soon, so if you think it's kind of gross right now, it won't be that way forever. If you add a client I'll migrate that over when I get around to it. I'm basically going to find a way to have g:conjure_config be the single source of truth since it can be updated from VimL, Lua or ex commands with the same ease.

Neovim specifics

Yeah, the floating window API is Neovim only, but I think Vim 8+ has a similar idea under another name? Just a different API I guess. The main benefit with Neovim for me is LuaJIT as the first class runtime.

[smolck]

Thanks for the quick response! 😃

If you go down the PR route you get to lean on everything that's already set up in the Conjure repo with regards to Aniseed Fennel compiling. If you write your own plugin (presumably with Aniseed too) you have more control over the marketing / readme / docs / tooling side. What do you think you'd prefer?

I think I'd probably prefer the PR route, since not only is that simpler for me but it would also help increase Conjure's usage, which would be awesome 😉

I have a feeling just integrating everything directly into Conjure is the way to go for ease of use by users. I'm leaning towards a fork that is eventually merged in.

Yeah I have no problems doing this and I think I agree.

So I try to find some sort of socket based transport for each language, like netrepl for Janet and nREPL for Clojure. This way the user can start the process with all of the env vars and magical things they require, they can even start it in docker! Then the client just uses some port file or known good default port to try and connect when you enter the filetype for the first time.

Okay, so I was talking with @theHamsta from Neovim's gitter, and he told me about swank-client, which is used by vlime and can be connected to via a socket. So it seems like that's the way to go, unless it turns out not to work well in which case I could maybe communicate with sbcl w/a process or try out cl-nrepl.

As far as the implementation goes though, it might be easier to parse everything from within common lisp, which would mean using cl-neovim to connect neovim w/common lisp. There's also other benefits to using common lisp for the implementation, but I think I'll see how far I can get w/Fennel and go from there. I imagine if I need to depend on cl-neovim maybe this would make more sense as a separate plugin? But hopefully I won't have to; and with Fennel it might not be that much more difficult.

[smolck]

So, one question I have is concerning connecting to a process via socket; how is that done here, specifically with the Clojure client? I haven’t looked into the code for the Clojure client yet, but if you wouldn’t mind giving a somewhat high-level overview of how that works (i.e. how to connect, where to store the ref to the socket if that’s necessary, how to communicate back and forth with the process, etc.), I’d really appreciate it.

Or if you’d rather just point me to the files/parts of the files that I should look at for the answers to those things I’m cool with that too 😉

[theHamsta]

No, swank-client is not used by vlime. swank-client, vlime, slima and slime. All implement the same Slime protocol. It is very simple and nrepl would not be needed. nrepl would just be an unnecessary dependency.

There are two alternatives:

• using cl-neovim that is talking via RPC with Neovim. Then, you would directly eval things in the remote process
• using a simple socket communication. Then, everything would be implemented just the way vlime and slime does it.

[Olical]

it might be easier to parse everything from within common lisp

Is there any parsing to be done? Almost all of the interactions with all current clients just send code for evaluation and then display the result. The Clojure + nREPL client does encode and decode for the bencode transport, but the results contained within those messages are used directly, Clojure is never parsed in Neovim.

You should just be able to send code and append results to the log buffer. Although you may encounter the problem of "what's stdout and what's data" which Janet has. So I just display them all in the log buffer uncommented, as opposed to Clojure where I can tell, so I comment out stdout/err.

So, one question I have is concerning connecting to a process via socket

conjure.client.clojure.nrepl.server handles all of this, so does conjure.client.janet.netrepl.server, it uses (vim.loop.new_tcp) to connect, the handle is stored for later disconnection. I actually want to refactor the TCP part out into a module that can be reused a little easier (although there's not much in it that is super reusable) and then build a reusable nREPL client module on top of that.

For now, I'd recommend copying the Janet server and modifying from there, it's pretty simple even though it does encode/decode netrepl messages.

There are two alternatives

I would lean towards the simple socket so the Neovim and CL processes aren't tied together and can be started or restarted independently.

[smolck]

No, swank-client is not used by vlime. swank-client, vlime, slima and slime. All implement the same Slime protocol. It is very simple and nrepl would not be needed. nrepl would just be an unnecessary dependency.

Ugh, my mistake! Swank-client is just a client that implements the protocol . . . facepalm
I see now, thank you for pointing this out. Not sure why I thought that.

There are two alternatives

I would lean towards the simple socket so the Neovim and CL processes aren't tied together and can be started or restarted independently.

This would also mean I have several examples to base my work on, so this would maybe be easier too.

conjure.client.clojure.nrepl.server handles all of this, so does conjure.client.janet.netrepl.server, it uses (vim.loop.new_tcp) to connect, the handle is stored for later disconnection. I actually want to refactor the TCP part out into a module that can be reused a little easier (although there's not much in it that is super reusable) and then build a reusable nREPL client module on top of that.

Great, thank you!

For now, I'd recommend copying the Janet server and modifying from there, it's pretty simple even though it does encode/decode netrepl messages.

Will do!

Is there any parsing to be done?

Yes, I believe so, although maybe not a whole lot. AFAIK (and I could be wrong) the response from swank is 6 bytes long, and it is either (:ok (response)) or something along the lines of (:error (what-went-wrong)) I think. I can verify that once I get started though so the details aren’t super important, what is is that there will be some parsing, but I don’t think it’s bad enough where it would be difficult to implement in Fennel.

[theHamsta]

It's always 6 bytes with the message length including (?) the 6 bytes in upper case hex characters filled up with zeros. You should be able to load the s-expressions directly with fennel.

[rafaeldelboni]

Hi folks, as a new year resolution I'm willing to study CL in 2022, and my experience with conjure it's so awesome that is almost a requirement for me to start fiddling with the language.

I was reading this discussion here and to summarize it in a few words:

• Start a remote swank-server repl following this?
• Implement what this file or this one does in conjure?

The developer experience with conjure+swank would be something like this:
(considering one does have sbcl and quicklisp installed in their system)

1. Initiate the swank server

  (ql:quickload :swank)
  (swank:create-server)

2. :ConjureConnect 4005

Is that right?

[Olical]

That sounds right to me! This'd be incredible to get working and may finally get me interested in CL some day.

While implementing, you'll want to implement a "remote" like this one https://github.com/Olical/conjure/blob/master/fnl/conjure/remote/nrepl.fnl which would depend on a "transport" like this https://github.com/Olical/conjure/blob/master/fnl/conjure/remote/transport/netrepl.fnl then you can write the "client" module that depends on that underlying system. just to give you an idea of what code goes where.

So we have a client that depends on a remote (kind of remote REPL) that depends on a transport for encoding and decoding. The naming may not be perfect but I hope that helps with the terminology and structure. A client can be a copy/paste/modify of any other similar one, that's just a module that exposes a few functions that fit the client interface.

You can find docs for that interface in :h conjure or look at any of the simpler clients to get a good idea for them. They all vary but they all fit the same interface while giving you room to get creative adding more functions and mappings your client may need.

[Olical]

It's in! 🎉