[RDY] Builtin terminal emulation

[tarruda]

This PR uses libvterm(big thanks to @leonerd for that project) to implement a full VT220 terminal emulator using Neovim buffers and windows. ~~It will also refactor the calling of shell commands to use this facility, fixing #1044 #1386 #1496 #1716 (all caused by an issue introduced in #1365)~~~(I've decided to not touch the generic os_call_shell for now)

Still a WIP but the :terminal ex command can already be used, here's a quick rundown:

• :term [prog/args] will open a new special buffer "connected" to a pseudo terminal running prog/args
• any command that would enter insert mode will focus the terminal buffer instead.
• when focused a new mode is entered where all keys are passed directly to the program. The <c-\><c-n> combination will return to normal mode.
• new terminal-only mappings can be added with the "t" prefix(eg: tnoremap <c-a> <c-\><c-n> exit terminal focus using ctrl+a)

@justinmk An observation about the new Terminal class: It is not coupled to job control so we can use it for implementing #901 and eventually remove all blocking messages(press enter to continue) from nvim(We could have a Terminal instance created at startup and feed all messages to it, for example)

[ZyX-I]

Wondering how it deals with https://bugs.launchpad.net/libvterm/+bug/1412093. Also what is the $TERM value (see https://bugs.launchpad.net/libvterm/+bug/1412091 for why this is relevant)?

[tarruda]

@ZyX-I I've directed the URL to a fork that fixes these messages: https://github.com/neovim/libvterm/commits/cflags-override( @leonerd will probably merge them once he gets back from holidays)

As for https://bugs.launchpad.net/libvterm/+bug/1412091 , libvterm aims have xterm compliance but its not there yet.

[bfredl]

Very nice! Is it possible to somehow paste/send stdin data to the vte ? (perhaps p in "terminal normal mode" could be remapped to "paste to stdin" )

[bfredl]

Shouldn't the "real" terminal cursor follow the vte cursor when in "insert" mode ? Otherwise it looks weird when e.g. using input methods

[tarruda]

@bfredl I plan to fully integrate this with vimscript job control API(it will be possible to control debuggers, paste data, etc)

Im still not sure about the cursor behavior(perhaps it should simply become invisible when in normal mode?)

[ZyX-I]

Invisible cursor looks like neovim doing something. Always invisible cursor looks like neovim hang forever.

[tarruda]

@ZyX-I I meant that the vterm cursor should be invisible when in normal mode and also the other way around, so there will only be one visible cursor at a time

[bfredl]

@tarruda yes, and when in vterm mode the vterm cursor should be forwarded as ui cursor events. This is required to make the preview buffer of input methods to show up at the right place (that is, at the cursor)

[justinmk]

eventually remove all blocking messages(press enter to continue) from nvim

@tarruda Is this in reference to #1802? I'm not attached to that PR, but the purpose of that PR is to give plugins (and core) the option to emit non-blocking messages, not to eliminate them entirely. The next (future) step I was envisioning was for message.c to raise "notification" events which may be optionally handled by UIs instead of just directly writing to the screen (then GUIs could have a notifications area that is not constricted by cmdheight, for example).

We could offer an option to eliminate all "Press ENTER" messages, but I'm not sure this would actually be a good thing for users to do. The "pager" that Vim emulates for messages is quite handy for messages that the user wants to briefly check, and the fact that it can be dismissed by pressing any normal-mode key (which is then executed) is a feature that would be missed.

So, #1802 has a small imperfection that I still need to fix, but perhaps we should discuss whether it is worth continuing. I would really like to wrap it up this weekend one way or other.

[bfredl]

Btw, the obligatory nvim-within-nvim test already works (probably not practically useful, but reckon it is a good "stress test" ) 👍

[justinmk]

@bfredl Really cool :) Going to try this PR now

[jszakmeister]

Is this in reference to #1802? I'm not attached to that PR, but the purpose of that PR is to give plugins (and core) the option to emit non-blocking messages, not to eliminate them entirely. The next (future) step I was envisioning was for message.c to raise "notification" events which may be optionally handled by UIs instead of just directly writing to the screen (then GUIs could have a notifications area that is not constricted by cmdheight, for example).

I agree @justinmk. I don't think it's wise to remove them all, they do serve a purpose and sometimes you really do want to call attention to the user.

[cweagans]

This is awesome :)

[Tranquility]

Awesome! I really don't know how I would use this feature but it's impressive how well it works.

Some things I noticed:

• long running processes keep running after vim closes and print error messages because the lnums are invalid
• moving word wise with alt-keys does not work
• if I am in normal mode and the cursor is below the vterm cursor (someplace without text) I need capital i to focus and i only prints E21: Cannot make changes, 'modifiable' is off
• in command mode the vim cursor jumps into the buffer if something changed. it looks like it's no longer in command mode although typing gets the cursor back

[khalidchawtany]

This is SUPER AWESOME :)

[mudox]

👍 awesome feature from Neovim.

[rainerborene]

Just one thing, when I type <C-l> the whole buffer gets empty instead of clearing and show only one line like on most shells. I think this might be related with the size of the pseudo terminal when you have more than one buffer opened.

Fucking awesome feature.

[jebaum]

@rainerborene on my system I don't see either of the bugs you describe with nvim -u NONE. you should try again with that to see if something in your nvimrc is causing an issue

[tarruda]

Also when you run :terminal without arguments nvim crashes.

Whats the value of your sh option?(when running without arguments it will run your shell). But it should not crash anyway so I will investigate

[oni-link]

We need a free(line); here. ml_append seems to copy line.

[tarruda]

👍

[tarruda]

@justinmk / @jszakmeister The states where user feedback is required can be easily emulated with the new terminal module. For example, here's how I plan to enhance bang commands with this new feature while still maintaining vimscript compatibility:

• Bang commands will open a terminal buffer to run the shell
• If the current buffer is modified, run the terminal buffer in a split window, if not it will replace the current buffer.
• When the program completes, pressing enter will return the cursor to the previous buffer.

Note that the keypress sequence is still the same, the differences are:

• the bang command will no longer block the user until complete if the user escapes the window
• If this happens the bang command will simply return prematurely and the process is still running like :terminal was invoked directly.(since this is not even possible right now, there wont be compatibility problems)
• the user can access the command output after it completes

A similar concept can be applied to message output:

• When nvim starts, a hidden terminal buffer is created(without a controlling job, of course)
• When the user must be notified of something, the message is appended to the hidden terminal buffer and then different actions can be taken:
  • If the message fits in a single line and not an error(an echo command for example), the behavior can persist the same: show the message in the command line
  • If its a message that would normally require feedback from the user(a :highlight command for example), a split window is opened with the message appended and the "press RETURN" prompt which will close the window bring the user back to the previous buffer.

The advantage of using this approach may not be obvious but here are some:

• We'll get rid of a lot of code in message.c. Implementation of message paging is trivial with the new module.
• The user will be able to copy the text from previous messages from the scrollback(:messages will simply open a split with the terminal buffer)
• We'll get rid of of the HITRETURN, ASKMORE, CONFIRM and EXTERMCMD states. The new TERMINAL_FOCUS state can be used to emulate any of them.
• Neovim will be able to handle deferred events in more situations(no problem in handling vim_eval requests while in the wait_return() prompt for example.

[bfredl]

Neovim will be able to handle deferred events in more situations(no problem in handling vim_eval requests while in the wait_return() prompt for example.

👍

[tarruda]

@Tranquility

Awesome! I really don't know how I would use this feature but it's impressive how well it works.

With time, the numerous use cases for this feature will be clear, but here's one idea: This allows another level of integration between text mode programs and Neovim. For example, we'll be able to make gdb respond to nvim mappings and change file/line numbers when a breakpoint is hit(Technically this might be possible with tmux but not without numerous hacks)

long running processes keep running after vim closes and print error messages because the lnums are invalid

👍 needs to be fixed

moving word wise with alt-keys does not work

Not sure what you mean, but when you leave terminal focus(Ctrl+w, esc) you can move around the scrollback just like any other buffer)

if I am in normal mode and the cursor is below the vterm cursor (someplace without text) I need capital i to focus and i only prints E21: Cannot make changes, 'modifiable' is off

Thats expected, the terminal buffer cannot receive direct modifications from the user: The key presses are forwarded to the underlying program which outputs redraw instructions. This is how any terminal behaves(you cannot delete lines from tmux panes for example)

in command mode the vim cursor jumps into the buffer if something changed. it looks like it's no longer in command mode although typing gets the cursor back

👍 Its just a redraw glitch, should be easy to fix.

[tarruda]

Btw, the obligatory nvim-within-nvim test already works (probably not practically useful, but reckon it is a good "stress test" ) 👍

I know, this was one of the first things I tried with 5 levels of nesting 😄

[Shugyousha]

On Wed, Mar 25, 2015 at 10:14 AM, Ian Ker-Seymer
notifications@github.com wrote:

If you need a terminal other than the one nvim is running in, just open
another one

Nvim won't always be running in a terminal.

I assume that nvim will but the GUI-versions of it won't. Either case,
the point still stands. If you use gui-nvim you can open a terminal if
you need one.

The advantage of not having a terminal emulator in nvim would be

• less code/complexity
• less dependencies
• smaller footprint

@raichoo:

Proper intengration with REPLs would be one reason.

Which functionality of a REPL would not be available in a stand-alone
term emulator? Copy-pasting will be possible in both cases and code
completion can be done on the REPL side as well.

@jochen Goertler
I was actually suggesting that this should be configurable. See issue #1029

[raichoo]

@Shugyousha If you want to see how more sophisticated REPL integration could be like, take a look at the idris mode for emacs. @tarruda also wrote a nice little script to integrate gdb with neovim.
Copy-pasting is actually something I want to get rid of, so that's not really an option.

[tarruda]

@Shugyousha

First I thought that a terminal emulator would be more elegant if implemented as a plugin(possibly working better than Conque because of Neovim async capabilities), but it would never be as good or efficient as a native implementation due to the way buffers/windows currently work in Neovim.

Implementing it in the core also gives more flexibility and power to plugins, because a terminal emulator is by itself a very nice extension point for a editor(as you will see below)

The advantage of not having a terminal emulator in nvim would be

• less code/complexity

This is only an advantage for developers, not for users. In any case, I did my best to ensure the code is documented and simple. Most of the line count introduced by this PR is for functional tests.

This is the function that vim uses for calling the shell in unix systems, its has as many lines as the terminal module introduced by this PR. Would you say vim code for calling external programs is less complex because they don't implement a terminal emulator? FWIW, somewhere in that function there's code to spawn a pty and emulate a (very poor) terminal in gvim(Even though that is a bit contradictory with the information in :h design-not)

• less dependencies

Less dependencies is nice, but its infinitely worse implementing the code yourself.

• smaller footprint

That would be an advantage for embedded systems, but Neovim is supposed to run in a developer workstation, which is where its extra features makes a difference.

Which functionality of a REPL would not be available in a stand-alone
term emulator? Copy-pasting will be possible in both cases and code
completion can be done on the REPL side as well.

With a standard terminal emulator its much harder to have automatic two-way integration with the editor. This script integrates neovim with gdb and it has the following features:

• Jumps to file/line numbers when gdb hits a breakpoint
• Keps track of running/paused state
• Automatically sets/removes breakpoints in a IDE-like fashion
• Use keyboard mappings in any editor window to control the debugger and inspect expressions

The script is very basic and I plan to extract most of the logic into a common debugging interface, one that will make integrate debuggers into Neovim as simple as writing regular expressions to match debugger states.

The same concept can be used to provide automatic integration with REPLs(send lines, automatically evaluate expressions, etc)

[Shugyousha]

On Wed, Mar 25, 2015 at 03:50:00AM -0700, Thiago de Arruda wrote:

@Shugyousha

First I thought that a terminal emulator would be more elegant if
implemented as a plugin(possibly working better than Conque because of
Neovim async capabilities), but it would never be as good or efficient
as a native implementation due to the way buffers/windows currently
work in Neovim.

Implementing it in the core also gives more flexibility and power
to plugins, because a terminal emulator is by itself a very nice
extension point for a editor(as you will see below)

The advantage of not having a terminal emulator in nvim would be

• less code/complexity

This is only an advantage for developers, not for users. In any case,
I did my best to ensure the code is documented and simple. Most of the
line count introduced by this PR is for functional tests.

I would argue that keeping the code lines /complexity low is an
advantage for users too because only code that is not present == bug-free
code.

This
is the function that vim uses for calling the shell in unix systems, its
as as many lines as the terminal module introduced by this PR. Would you

Hehe, it's true that this code is very ugly but, to be fair, a lot of
the ugliness stems from the fact that there are tons of #ifdefs checking
for activated features or OSs (even for VMS...).

As far as I can tell this function is around 1100 lines of code while
we have around 3000 lines in the PR (I did not check how many of those
lines are comments though).

• less dependencies

Less dependencies is nice, but its infinitely worse than implementing
the code yourself.

Sure, but that's not what I am suggesting. I think this functionality
should be left out completely.

• smaller footprint

That would be an advantage for embedded systems, but Neovim is
supposed to run in a developer workstation, which is where its extra
features makes a difference.

Not if you don't use them... then the added lines of code and
dependencies are just additional points-of-failures and potential bugs
that make working with the software more annoying.

Which functionality of a REPL would not be available in a stand-alone
term emulator? Copy-pasting will be possible in both cases and code
completion can be done on the REPL side as well.

With a standard terminal emulator its much harder
to have automatic two-way integration with the editor. This
script
integrates neovim with gdb and it has the following features:

• Jumps to file/line numbers when gdb hits a breakpoint
• Keps track of running/paused state
• Automatically sets/removes breakpoints in a IDE-like fashion
• Use keyboard mappings in any editor window to control the debugger and inspect expressions

The script is very basic and I plan to extract most of the logic into
a common debugging interface, one that will make integrate debuggers
into Neovim as simple as writing regular expressions to match debugger
states.

The same concept can be used to provide automatic integration with
REPLs(send lines, automatically evaluate expressions, etc)

I am not sure why you would need an integrated terminal emulator for
this. Wouldn't it be just as easy to send commands to stdin of GDB/REPL
and parse the output for processing in nvim (like :make seems to do it
in vim)?

In any case, this feature seems to be wanted by a lot of people so I will
just have to live with it being there or do something about it with other
like-minded people (i. e. make it possible to disable it at compile time).

[tarruda]

In any case, this feature seems to be wanted by a lot of people so I will
just have to live with it being there or do something about it with other
like-minded people (i. e. make it possible to disable it at compile time).

Yes, simply ignore the :terminal command and termopen functions. Its not going to affect you if you don't use them 😄

[ghost]

I would definitely agree that it should be optional at compile time (because it introduces a dependency, and seems well isolated), but I'm not sure that's viable if is_call_shell() is eventually reworked to use libvterm. And as @tarruda said, it can just be ignored anyways.

[trusktr]

I'm 100% for this feature being compiled into neovim while having a (completely feasible) compile-time option to skip it entirely for those who aren't. Neovim plugins would need to check for the feature: if has('terminal').

I absolutely love this feature. I feel more productive with the convenience of all the functionality i have outside of TERMINAL mode (yanking, pasting, commands, searching, etc, etc, etc) without emulation like would be needed in tmux. I've left tmux and haven't even thought of looking back. And soon I'll be able to do this in a kickass GUI neovim? Fck yeah. Plus all the possibilities Thiago just mentioned which are now much easier to implement direvtly with neovim's architecture, not relying on (multiple) ugly hacks to get these things working with (multiple) other emulators outside of neovim? Fck yeaaaaaaah.

Another fck yeah example: with syntax highlighting, things like

//trusktr@starlancer//path/to/cwd❯ cat helloWorld.js
if (true) {
  console.log('hello world!')
}

can very easily become

//trusktr@starlancer//path/to/cwd❯ cat helloWorld.js
if (true) {
  console.log('hello world!')
}

using just the current features of neovim plus the new terminal mode (imagine the command line also colored).

[ZyX-I]

using just the current features of neovim plus the new terminal mode (imagine the command line also colored).

Command-line can easily be colored using the other means: check out zsh-syntax-highlighting or zsh-pygments-highlighting (first is slow and in many cases incorrect, second is fast, but is a technical preview, requires zsh syntax support in pygments (currently only highlights bash-compatible input) and zpython module installed).

I know you theoretically can highlight the output of cat with zsh as well, though better (== more robust/less buggy) just make cat a function that uses pygments if output is a tty.

[ghost]

@tarruda do you plan on documenting this soonish? I assume you have a lot to do already, but I can't think of a better person to document this than the person who wrote it. If I can help in any way just ask.

[tarruda]

@tarruda do you plan on documenting this soonish? I assume you have a lot to do already, but I can't think of a better person to document this than the person who wrote it. If I can help in any way just ask.

Definitely. I didnt do it yet because I will create another PR after this one that will contain changes to the job API, so I'd rather document everything in one step.

[ghost]

Definitely. I didnt do it yet because I will create another PR after this one that will contain changes to the job API, so I'd rather document everything in one step.

👍 Thanks for clarifying.

[trusktr]

@ZyX-I

Command-line can easily be colored using the other means

I wouldn't want to use all those other means because those means are already part of Neovim, and we can use them on the terminal buffer. It's so convenient.