Add Steel as an optional plugin system - #8675
Conversation
|
I've been using this Steel + Helix integration for a bit. My config is here: https://github.com/zetashift/helix-config The main downside here imho is that Helix isn't as great for Lisps(or Scheme in this case?) as Emacs for example. But besides that this worked nicely. I have no real complaints and I actually did not need to script so many things as with my VSCode or NeoVim configs because a lot just works ™️ . |
I think we can definitely improve, this is just the first prototype after all. You can't do everything at once. For example, I want to fully revamp the config system in the future (and remove the toml based config) so that is works even better with scheme |
Oh! I'm very sorry if I came over as demanding. I meant more like, in my experience this was what I stumbled upon, and it's not even a permanent thing, because a LSP for Steel might be in the works as well. I completely agree that you can't do everything at once, nor should somebody be obliged to do it. And maybe some lower hanging fruit can be picked up by people like me. For example if the tree-sitter grammar for scheme can improve things for Steel than that's another avenue. |
|
No worries I didn't read it as negative. I ws more trying to highlight that this is just an initial prototype that can & will Improve (although Mathew already did a great job!) |
This is something that people with more familiarity with tree sitter and indents could be able to answer - are indent queries capable of matching a proper lisp indent mode? I did spend some time trying to mimic it but without much success |
There was a problem hiding this comment.
Github says the ending newline is missing in this file.
I don't know Scheme / Lisp but it looks like the indentation should not be too difficult to model with tree-sitter indent queries (just using (list) @indent
; Align lists to the second element (except if the list starts with `define`)
(list . (symbol) @first . (_) @anchor
(#not-eq? @first "define")
(#set! "scope" "tail")) @alignIn order to improve this, I'd need to understand in which cases lists are aligned to the first, in which to the second element and when they are not aligned at all. Is this documented somewhere? If I find the time, I'll also try to understand the scheme indent implementation you wrote. Now that the plugin system is making progress, it might finally be time for me to learn the language 😄 |
This is great, I really need to learn more about tree sitter idents! I used this as a reference to get started: https://github.com/ds26gte/scmindent#how-subforms-are-indented Note, my implementation is not complete and was mostly an exercise in 1. seeing if it could be done and 2. getting something far enough to make editing scheme code more pleasant, I would not verbatim use it as a canonical reference. |
|
Thanks a lot for the reference @mattwparas. I created a PR (#8720) with indent queries that should cover everything from it except for some simplifications regarding keywords. Feel free to try it out and report any issues it has (you can just copy the |
There was a problem hiding this comment.
already left some notes but haven't got trough everything.
I think we should be mindful of not exposing too many implementation details and not tying the API too closely to how helix is currently implemented (as that would make evolution harder and may feel clunky).
In the longterm I would like a lot more focus on configuration but to get a good steel API for that we probably need to revamp our config system internally first so that its not tied to closely to toml/serde (And that there is a separate config of a mutable buffer config) so it may be better to do that in future PRs.
Ideally I would like something like the following to work eventually:
(config
(theme "everforest")
(soft-wrap
enable #true
wrap-at-text-width #true)
(text-width 85)
(keymap "normal"
("=" write)
("j" my_custom_function)))| @@ -0,0 +1,265 @@ | |||
| use helix_core::syntax::Configuration; | |||
There was a problem hiding this comment.
nit: I think the name engine is a bit ambiguous. Could we call it something like script_engine to make it clearer what the file actually does
| } | ||
|
|
||
| // Attempt to fetch the keymap for the extension | ||
| fn get_keymap_for_extension<'a>(cx: &'a mut Context) -> Option<SteelVal> { |
There was a problem hiding this comment.
this is too specical cased IMO.
While nice for a proof of concept the editor should have a generic concept of per buffer keymaps and then plugins should just interface with that
There was a problem hiding this comment.
I would agree, it requires a little hacking from the steel side to make it work as you can tell. Do you think it is worth investing in that machinery to expose that from the editor side now?
| } | ||
| } | ||
|
|
||
| thread_local! { |
There was a problem hiding this comment.
I am not sure how suitable threadlocals really are.
I think tokio can and does move our main loop to different threads (and I also don't want to constrain interaction with plugins to a single thread).
In the worst case these could be globals but for almost all of these it feels like its really adding more extensability to the editor here instead of in the editor itself.
I think we should instead take the approach of making the core editor more extensible and then exposing those APIs to plugins.
These APIs would ofcourse require access to somekind of context. This would be similar to the context problem you mentinoed in your description. I think the best way to handle that one would be if we stored the context as a global inside the engine (using run_with_reference).
In any callbacks that want access to the context we check for that global (even user code could do it themselves right?) and throw an error if a function that needs a context is called without context access.
There was a problem hiding this comment.
I'm currently refactoring the implementation to instead just from the steel side of things, have one variable that refers to the context. Whenever we call steel code, we'll do like you said - just refresh the context variable with the current context for the duration of the steel code scope, otherwise it'll be some poisoned value that will error.
With respect to thread locals - I'm really not sure how else to implement the interaction with the steel engine if not through a thread local, and any callbacks that I set up are done on a thread local queue that tokio is processing.
Can you elaborate on:
I also don't want to constrain interaction with plugins to a single thread
Does this mean you want the engine to be utilizing multiple threads? Or you want the main loop to be able to be moved to different threads (or both)?
As it stands, the engine implementation is not thread safe, which would make this rather tricky to implement. I don't know if tokio is actively moving the main loop to another thread or not, as I haven't run into any strange behavior with the interaction between the editor and the engine up to this point.
There was a problem hiding this comment.
To put it another way - if we didn't want to have the main thread interact with the scripting engine, then it would have to take the context by value for this to work (at least, that is what I'm currently thinking)
There was a problem hiding this comment.
If tokio lets you hold the engine struct across await points, I think the main loop only ever runs on a single thread. I would like to be able to eventually make Engine Send so we can send it across thread. I could definitely imagine that we would like to run some plugin code in the background (or in response to some async event).
Requiring engine to be Sync won't work since you would need to pub mutexes everywhere which would be excessively slow. But Send could be reasonable so we can have Mutex<Engine> and share that across threads. That is usually a reasonably easy requirement. The main thing you can't do is expose something like Rc to the user. I know you use refcounting internally (so Send is currently not implemented) but it could be safe to unsafe impl Send for Engine assuming that you never hand any refcounted types out to rust (or if you do make sure to use Arc or a non-clonable reference).
I think thread_locals are quite alright for the Context since the context would be explicitly provided everytime we call into the Engine.
There was a problem hiding this comment.
So that the current moment, I've added some code to have the whole editor panic if the engine gets used on another thread to see if it is happening. I have been using this as a daily driver for a few weeks like this and haven't yet run into it. So while I agree thread local isn't the best, it also seems to be acting as intended, which I think is just by coincidence
| module.register_fn("editor-cursor", Editor::cursor); | ||
|
|
||
| module.register_fn("cx->cursor", |cx: &mut Context| cx.editor.cursor()); |
There was a problem hiding this comment.
this is an instance of exposing too many implementation details.
As I said in the other comment, context should be somekind of hidden global (which can have different capabilities depending on callsite).
The user should only know that he can do X/Y at a certain callsite. the existance of context and editor struct is an implementation detail that plugins should not be aware of
There was a problem hiding this comment.
So just to clarify - don't expose the editor struct at all, any details go directly through the context?
I'm fine with that, just confirming the intent here.
There was a problem hiding this comment.
yeah don't expose theses structs at all. These should not be editor-cursor and cx->cursor but instead just active-cursor and the internal structs are handled in the background.
|
|
||
| // TODO: | ||
| // Position related functions. These probably should be defined alongside the actual impl for Custom in the core crate | ||
| module.register_fn("Position::new", helix_core::Position::new); |
There was a problem hiding this comment.
what are the naming conventions in steel? These look like rust paths to me
There was a problem hiding this comment.
There aren't any explicit naming conventions, although in general I prefer kebab case. At some points I tend to adopt Rust naming conventions in order to make interaction with the system more obvious. This is certainly open for discussion and I don't have any strong opinions.
There was a problem hiding this comment.
hmm I am not sure how I fell about using rust naming conventions. I think syntactically it feels nice to stay a bit with conventions typically used in scheme/lisp. For example racket conventions could work well: https://docs.racket-lang.org/style/Textual_Matters.html
I think in this case the name new is not needed at all and we could just call it (position row col) following the naming conventions for constructors in racket (https://docs.racket-lang.org/reference/define-struct.html)
This comment was marked as off-topic.
This comment was marked as off-topic.
|
this PR is not intended to discuss the choice of plugin language. We already made our choice after lengthy discussion. This PR is only intended for reviewing and discussing this particular implementation. I will mark such comments offtopic |
|
As an aside question, I recall that we were talking about sandboxing on Matrix a while back. Is this still planned/implemented here? |
Not fully, right now there are no restrictions what a VM van do but I talked about this a while ago with Mathew and it should be possible to add capability based security to plygins in the future. This would entail a one VM per plugin approach (and how to handle interop in that case) but those should be solvable problems (the same problems would have occurred with any sandboxing including wasm) |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Hi, what's the status of this pr? Really looking forward more progress on plugin system. |
I'm still working on it! I made some reasonably involved changes to avoid passing around a context to all the functions which needs some clean up. I've also made a relatively well functioning terminal emulator with the existing component API. Stay tuned! |
|
I haven't looked into the code. Did you avoid passing around a context with state monad? When I wrote haskell, I used state monad to carry implicit global state. |
|
Would you consider this ready to play around with and write some plugins for, or is it still too early for that? |
Certainly functional to use and write plugins with. I've been daily driving on it for a long time now. I'll be rebasing to the latest stuff later today, and will update the installation instructions. If you decide to start using it, please don't expect any guarantees around anything. I'd expect the configuration API to change before its all said and done. |
|
|
||
| (require-builtin helix/core/typable as helix.) | ||
| (require-builtin helix/core/static as helix.static.) | ||
| (require-builtin helix/core/keybindings as helix.keybindings.) |
There was a problem hiding this comment.
helix/core/keybindings seems renamed to helix/core/keymaps
|
What is needed to integrate this with helix master ? |
|
Just to add my voice here. I really like the idea of having a plugins.toml. As for scripts, instead of writing them directly in toml, you could also add a path to the steel script you want to run. The plugin branch seems stable et could be merged soon so I would not be un favour of trying to implement it in the current PR. It's been long enough and some people seem to be waiting for this to switch from neovim to helix. What you propose is a more user friendly way to use something that still isn't merged and could be usable. If you want you can also join the discord where most discussions take place. I believe there's a link somewhere in this PR. |
|
I'd prefer not to discuss this design further here since this is already a large PR. My opinion is that the implementation as provided here allows you to do basically anything you want, entirely in user space. I have no intentions of changing that at the current time. If someone would like to put together a plugin that boot straps a Also to note, steel can be turned off both at build time, and at runtime, and the existing configuration strategies still work exactly the same as before. Migration should be relatively trivial (if its not, its probably a bug) |
|
As I explained above, But ig it could be added afterwards. No need to delay this. |
|
I think we should move away from toml support in few more releases. Its better to have configuration in a single unified language for everything, wrt. keeping some in toml and other in steel. Plugins will come with their own set of keybindings, so it is quite logical to have everything within one language. Also it opens a whole another level of possibilities. Maybe we can embed macros, maybe we can achieve context based behavior, to do something like this (not correct syntax): ;; goes to line start if cursor is in first non whitespace else goes to first whitespace char
(if (not (string=? "cursor" "first_non_whitespace_char"))
"goto_first_nonwhitespace"
"goto_line_start")and the list goes on... |
|
What about assigning a config file for each plugin and letting the extension pick if it wants to use steel or toml? So in say plugins.toml: [plugin_name]
src = "github.com/foo/bar"
config_file = "plugins/extension.scm"
[plugin_name2]
src = "codeberg.org/foo2/bar2"
config_file = "plugins/extension.toml" |
|
As a nix user the only pro i see in a plugins.toml would be the possible ability to configure it in nix same way we can do it with config and languages.toml. But i don't really consider it something groundbreaking |
|
what about a plugins folder. Autoload whatever is sits in
and store the unused pile in
and if you want to have a dev version to override a current active plugin, add it to
|
|
Imho this is a discussion for a later point in time. We as community can write scripts to try out different approaches and whatever works best gets included as fallback/default script. |
Override nixpkgs' helix-unwrapped with mattwparas' steel-event-system fork (upstream helix-editor/helix#8675), reusing nixpkgs' hash-pinned grammar farm. This replaces the old zellij/broot keystroke-injection sidebar: forest.hx runs inside Helix as a Steel plugin, so opening a file is a function call rather than synthesized keystrokes. Cogs (forest.hx + its notify/glyph deps) are git submodules under steel-cogs/, symlinked into $STEEL_HOME/cogs by sync.py's new build_cog_symlinks. STEEL_HOME is set in the fish config so it reloads without a re-login; init.scm binds space-e to :forest-open.
|
How about something similar to yazi's plugin implementation. The argument for still having toml in configs would it On plugins.toml why not treat it like a manifest/lockfile that outlines installed plugins/packages (and perhaps named to package.toml) Since currently, when downloading plugins with steel forge, it doesn't track what it downloaded. This makes it harder to reproduce a helix environment on another machine plus a dedicated helix steel plugin installation directory would be nice, in case someone uses steel for something else edit: |
Writing parsers in scheme is really easy. Writing a toml parser should be possible within a day. So if you really want to, you can bring that functionality back with a trivial plugin. |
These links are to vague comments that are three years old. I would not take them to reflect the details of the current plan. I really don't think the maintainers intend to break all existing users' setups with the release of dynamic config. |
|
@mattwparas sorry if this has been asked before but it is kinda hard to go through the churn of comments. what is the state of this PR and what is it missing for it to be ready for review / merging? you may also want to update the PR description with this info since i suppose this is a common question thanks for your hard work! |
|
The only thing remaining that I'm working on (other than bugs) is building in some integration tests. I have some uncommitted work that I'm still polishing before pushing that sets up a framework for that in tree and add a suite with some existing plugins that have been made. Otherwise it is the usual make improvements on steel -> patch them over here, address individual bug reports here, etc. I think there is some love in the API that I can polish in a backwards compatible way that I'll do after I get the test framework set up. I'd like to make these changes more easily with more confidence before progressing. There have been many questions about boot up time, I am working on it. The compilation model with macros makes this difficult to arbitrarily cache intermediate compilation artifacts and load them willy nilly, but I'm investigating. It will require a modest overhaul to some of passes in steel to accommodate, but I think in general it can be done. Steel will be better for it at the end in exchange for some pain along the way. Once I land the integration test stuff I'll update the PR description to reflect the new state. Overall given the number of plugins that have managed to be made not by me, I think it is generally in a good position. |
Stupid question: Why do you have to worry about backwards compatibility before anything has even been released? |
|
There are many plugins which exist today and I'd prefer to not break them without good reason (including my own plugins). I've been pretty mindful about this and haven't broken anything intentionally in a long time. What I was mostly referring to is that there are a lot of built in functions that don't accept arguments because they're static functions designed in a pre plugin world. I'd like to add optional arguments there. I don't think there are going to be any earth shattering changes here, and I reserve the right to do so, but also I'd like to be respectful of the time people have put in to making plugins and be sure that I'm not making the wrong decision when breaking things. Also just setting myself and others up for the future so we don't accidentally break things. |
This comment was marked as low quality.
This comment was marked as low quality.
A highly professional public comment as a representative of your company. Additionally I would check out your companies website. It is broken on multiple levels. I wonder why... |
|
I know you've heard a million of these comments already, but thanks for the hard work :) . I am really excited to see this feature get released onto master when you feel confident in it. 3 years of work is no small amount of work (especially for unpaid labor), keep trucking along <3 |

Notes:
Opening this just to track progress on the effort and gather some feedback. There is still work to be done but I would like to gather some opinions on the direction before I continue more.
You can see my currently functioning helix config here and there are instructions listed in the
STEEL.mdfile. The main repo for steel lives here, however much documentation is in works and will be added soon.The bulk of the implementation lies in the
engine.rsandscheme.rsfiles.Design
Given prior conversation about developing a custom language implementation, I attempted to make the integration with Steel as agnostic of the engine as possible to keep that door open.
The interface I ended up with (which is subject to change and would love feedback on) is the following:
If you can implement this, the engine should be able to be embedded within Helix. On top of that, I believe what I have allows the coexistence of multiple scripting engines, with a built in priority for resolving commands / configurations / etc.
As a result, Steel here is entirely optional and also remains completely backwards compatible with the existing toml configuration. Steel is just another layer on the existing configuration chain, and as such will be applied last. This applies to both the
config.tomland thelanguages.toml. Keybindings can be defined via Steel as well, and these can be buffer specific, language specific, or global. Themes can also be defined from Steel code and enabled, although this is not as rigorously tested and is a relatively recent addition. Otherwise, I have been using this as my daily driver to develop for the last few months.I opted for a two tiered approach, centered around a handful of design ideas that I'd like feedback on:
The first, there is a
init.scmand ahelix.scmfile - thehelix.scmmodule is where you define any commands that you would like to use at all. Any function exposed via that module is eligible to be used as a typed command or via a keybinding. For example:This would then make the command
:shellavailable, and it will just replace the%with the current file. The documentation listed in the@docdoc comment will also pop up explaining what the command does:Once the
helix.scmmodule isrequire'd - then theinit.scmfile is run. One thing to note is that thehelix.scmmodule does not have direct access to a running helix context. It must act entirely stateless of anything related to the helix context object. Runninginit.scmgives access to a helix object, currently defined as*helix.cx*. This is something I'm not sure I particularly love, as it makes async function calls a bit odd - I think it might make more sense to make the helix context just a global inside of a module. This would also save the hassle that every function exposed has to accept acxparameter - this ends up with a great deal of boilerplate that I don't love. Consider the following:Every function call to helix built ins requires passing in the
cxobject - I think just having them be able to reference the global behind the scenes would make this a bit ergonomic. The integration with the helix runtime would make sure whether that variable actually points to a legal context, since we pass this in via reference, so it is only alive for the duration of the call to the engine.Async functions
Steel has support for async functions, and has successfully been integrated with the tokio runtime used within helix, however it requires constructing manually the callback function yourself, rather than elegantly being able to use something like
await. More to come on this, since the eventual design will depend on the decision to use a local context variable vs a global one.Built in functions
The basic built in functions are first all of the function that are typed and static - i.e. everything here:
However, these functions don't return values so aren't particularly useful for anything but their side effects to the editor state. As a result, I've taken the liberty of defining functions as I've needed/wanted them. Some care will need to be decided what those functions actually exposed are.
Examples
Here are some examples of plugins that I have developed using Steel:
File tree
Source can be found here
filetree.webm
Recent file picker
Source can be found here
recent-files.webm
This persists your recent files between sessions.
Scheme indent
Since steel is a scheme, there is a relatively okay scheme indent mode that only applied on
.scmfiles, which can be found here. The implementation requires a little love, but worked enough for me to use helix to write scheme code 😄Terminal emulator
I did manage to whip up a terminal emulator, however paused the development of it while focusing on other things. When I get it back into working shape, I will post a video of it here. I am not sure what the status is with respect to a built in terminal emulator, but the one I got working did not attempt to do complete emulation, but rather just maintained a shell to interact with non-interactively (e.g. don't try to launch helix in it, you'll have a bad time 😄 )
Steel as a choice for a language
I understand that there is skepticism around something like Steel, however I have been working diligently on improving it. My current projects include shoring up the documentation, and working on an LSP for it to make development easier - but I will do that in parallel with maintaining this PR. If Steel is not chosen and a different language is picked, in theory the API I've exposed should do the trick at least with matching the implementation behavior that I've outlined here.
Pure rust plugins
As part of this, I spent some time trying to expose a C ABI from helix to do rust to rust plugins directly in helix without a scripting engine, with little success. Steel supports loading dylibs over a stable abi (will link to documentation once I've written it). I used this to develop the proof of concept terminal emulator. So, you might not be a huge fan of scheme code, but in theory you can write mostly Rust and use Steel as glue if you'd like - you would just be limited to the abi compatible types.
System compatibility
I develop off of Linux and Mac - but have not tested on windows. I have access to a windows system, and will get around to testing on that when the time comes.