Help needed: libvterm integration for remacs

[brotzeit]

@akermu already contacted me because he is interested in contributing. I want to invite everybody to join us: remacs/remacs#1130.

If you have any questions you can ask in our gitter room.

[deb0ch]

@brotzeit Sounds great, I've been meaning to try Remacs for a long time, maybe I will get to it now 😁

But I tought Remacs was trying to stay on-par and compatible with Emacs, is there so many breaking changes that require so much work to integrate emacs-libvterm ?

Why is all that work required ?

[brotzeit]

But I tought Remacs was trying to stay on-par and compatible with Emacs

terminal emulation is kind of special, since it doesn't break compatibility(at least I hope so). I had the idea to implement libvterm support while I was porting some terminal related C code to rust and I thought it would be better to use an external library. Then I found libvterm and this module and gave it a try.

Why is all that work required ?

I don't understand this question, but the goal of remacs is to get rid of the C code. Since emacs-libvterm already works I used it as a starting point. I had the choice between porting the unpopular terminal emulation code from emacs or try something different.

Currently it's about 50:50 rust to C, but it's not important to get rid of the C code immediately. It doesn't have to be completely in rust to get merged. What I need now is help to fix some bugs and to add more features.

[deb0ch]

Oh, I thought at first that, just like in Emacs for now, emacs-libvterm was to be used as a package, i.e. get it from Melpa (maybe in the future, for now git clone it), require it, vterm-module-compile it and you're good to go.

But actually you are forking it to integrate it as a part of the codebase, right ?

[brotzeit]

But actually you are forking it to integrate it as a part of the codebase, right ?

Exactly. When I get some help it's realistic that it gets merged in a few weeks.

[brotzeit]

For everybody who is interested. Just clone this branch https://github.com/brotzeit/remacs/tree/vterm , build it with libvterm installed and call vterm.

[deb0ch]

I know that neither of these projects are mine, but as a contributor and maintainer of Emacs' ecosystem I am concerned about it in general.

Although it is great to hear that Remacs will ship with a decent terminal, something that never happened in the entire Emacs history and is not even close to happen because of that FSF papers paranoid obsession, I am concerned about the forking approach.

Doing a fork implies that for any improvement made on either side, the emacs-libvterm or the Remacs community will have to do extra work to actively monitor updates and backport them, and the more changes are not backported, the more difficult it will be for other ones. All this extra work will not be spent to make either side better in other ways, where developper time is an especially scarce resource on open source projects, thus impinging the good dynamic of both projects.

Instead, Remacs could use emacs-libvterm as an untouched external dependency for instance using it as a git submodule and send improvements upstream directly.

What are the benefits to this approach that make it worth the drawbacks in your eyes ?

[brotzeit]

The goal of remacs is to replace all C code. If I wouldn't come up with a different solution we would have to port the old terminal stuff and as you know it's not worth the time. So you have to see it from a different perspective: By using libvterm and emacs-libvterm we are saving a LOT of time.

[deb0ch]

emacs-libvterm is still being developped at a fast pace, big improvements are pouring these days, and the rythm still seems steady. Will similar updates be seamless to integrate in Remacs, or will it involve some work? If you improve it in your codebase, what will it involve to Port these benefits to emacs-libvterm?

Would it make sense to keep emacs-libvterm as a submodule or similar until it reaches more maturity, and only port it to rust afterwards?

[brotzeit]

It won't be enough to copy paste improvements. It depends on the kind of changes how much work it will be to adopt them.

Would it make sense to keep emacs-libvterm as a submodule or similar until it reaches more maturity, and only port it to rust afterwards?

This work was intended to have symbolic value. As you said, it's next to impossible to get something like this merged upstream and if users see that this kind of changes take place in remacs, they are more likely to give remacs a try.

It's also kind of too late. The code has already changed significantly. I'm at a point where I want to "finish" the work and get this merged, sorry.

[TheBB]

The big difference between dynamic modules and integrated C (or rust) is that in the former case you get underprivileged access to Emacs internals. It's all good to write fast C code, but the moment you need to interface with Emacs in any way it's like pulling thread through a needle.

When writing a Python extension module you can write code on the same level as literally everything else in CPython. When writing an Emacs dynamic module you're relegated to intern and funcall for practically everything.

[brotzeit]

@TheBB Indeed. I'm not sure if I would write an emacs module.

[brotzeit]

@TheBB oh wow nice work on libegit2. I guess that would be an exception for me ;)
Do you think it would perform much better when it would be part of emacs ?

[TheBB]

Thanks. No, libgit is a perfect use case for dynamic modules. This one and something like https://github.com/TheBB/yeast is more difficult to write. I'd kill for a lower level interface to Emacs buffers, even if just read-only.

[deb0ch]

I see, I hear you both and it makes sense.

What I'm talking about is changes like #51 (enable native paste and undo), #44 (optimisation) or ec95898 (fix bugs by processing one more escape code), or even #44, which fixes visual glitches on window resize and refactors some code processing. All of these happened just in the past month, I am not even going further but you get the gist - pace is still fast, changes are big.

It might make sense to keep this untouched in your codebase and re-do this same integration in a few months or a year, when this project has gotten most of the big changes and has become stable.

The risk that I see is a state where it would be more interesting for Remacs users to use emacs-libvterm as an external package rather than the version integrated in the source, just because the former has more features that were not ported or got more stable.

[brotzeit]

@deb0ch It's really not that important that it's perfect. The main reason for doing this is to show the emacs community "hey remacs merges stuff that you will never see in emacs". I hope that from this point more users realize that emacs can move forward if we try. Remacs users still can use emacs-libvterm as a module.

[deb0ch]

Ok, I get the intention and, although it still think it would have been more optimal done at a later point, adhere to the idea - that FSF nonsense is something that has greatly held the Emacs project back and I would be delighted if a project like Remacs could prove them wrong 😺

[deb0ch]

Thank you for your explanation 👍✨

[akermu]

I'm also happy that this module is ported to remacs and rust. @brotzeit already explained, by integrating it into the core, you get way more control, how the the shell and libvterm interact with the tty and bugs like #40 could be resolved, which is not possible with the current module architecture.

[akermu]

I'll leave this open, so people can see, that there are other efforts to bring a proper terminal to an emacs.