Retain terminal processes between window reloads

[sibowilldo]

Hi

Please implement Hot Exit on Integrated Terminal.

Thanks

• VSCode Version: 1.9.0
• OS Version: Windows 10

[Tyriar]

This would be cool, I'm not sure it could be done in the sense of retaining the processes (as they should be killed once VS Code is done). Maybe the shell and environment could be restored though?

There's a PR out for hyper which is relevant vercel/hyper#945

[ppot]

@Tyriar Thanks. I'm here if you have questions!

[sibowilldo]

@Tyriar Yes restoration is exactly what i was referring to, e.g If I'm working on a Laravel Project and I run the serve artisan command, after restarting VS due to an update, it would be a great implementation if VS would automatically run the "php artisan serve" command on its own.

[Tyriar]

@fabiospampinato just published an extension which allows you to configure several terminals and launch them all at once! https://marketplace.visualstudio.com/items?itemName=fabiospampinato.vscode-terminals

[fabiospampinato]

Looks like my extension mitigates this issue almost entirely.

It would be pretty cool to retain the output of those terminals between sessions, but I guess an API for getting it as text, and another one for writing it (without executing it as a command) are necessary for this (@Tyriar do you think it would be possible to add them?) . We can leverage the fact that by using this extension no two terminals opened by it would probably have the same name, so we could safely link terminals' names with their text.

I guess it would also be nicer to have the terminals ready as soon as you open the project's folder, without the need to run the Terminals: Run command, I'll add an option for this tomorrow 👍

[Tyriar]

@fabiospampinato no plan on adding the ability to read or write right now.

Something that will help you though is that you will be able to add/remove environment variable from the environment in 1.15.

[Tyriar]

This issue is a little ambiguous at the moment, here's what I see covered by it:

1. Support reloading VS Code without terminating terminal processes

This can be accomplished by moving the terminal process to live under Electron's main process instead of the browser process, then reinstating the terminals after a load occurs. There is quite a bit of refactoring that needs to happen here, probably most significantly is splitting node-pty into 2 components; one that launches the process and exposes the file descriptorunix socket/named pipe (under the main process), and one that takes the file descriptorunix socket/named pipe and connects to it (under the browser process). We do not want the pty's data stream to just be passed through IPC as it will slow down everything.

2. Restore terminals with their cwd/environments they were created with when restoring a workspace

Note that this should only happen when 1. does not occur. The solution for this should allow for split terminals to be restored in the future #7504

[jerch]

one that launches the process and exposes the file descriptor (under the main process), and one that takes the file descriptor and connects to it (under the browser process)

@Tyriar Some OSes do not allow the pty master fd to be duped/cloned, e.g. used by another process than the original creating process. IPC is the only platform independent way imho (os pipes are pretty fast though).

[Tyriar]

@jerch good to know. The sheer amount of data that's possible to go through the terminal is the concern as it could lock up the main process which is shared between all VS Code windows 😕. Maybe I'm just overestimating the impact it would have on the main process? Any more details on the some OSes part?

[jerch]

@Tyriar It is still possible to carry the master fds around without duping it, if O_CLOEXEC is not set. It is important to reenable it in the browser process, otherwise the fds will leak to all subprocesses (assuming you have no control of electron's subprocess creation, the fds could also be closed by hand between fork and exec).

[Tyriar]

I amended my commend in #20013 (comment) to replace file descriptor with named pipe (Windows) and unix socket (Linux/macOS) as file descriptors are per-process. Implementation would use the net module https://nodejs.org/api/net.html#net_ipc_support

I'm thinking this is the way forward as node-pty is a standalone project (not just VS Code), plus plumbing terminal data streams through the standard main process IPC could block other communication if the terminal is spitting out lots of data.

[Tyriar]

@mmis1000 no configuration or tmux would be needed for my vision, it would just work magically across all platforms, probably with a setting that lets you set how long to keep the processes around without a VS Code window present.

[jerch]

@Tyriar Would be nice to have some standardized "terminal over ethernet" protocol in between (with all niftiness like replay, auth, read/write multiplexing, TTY orchestration, etc.), this way the daemons could run decoupled on several hosts and some aggregator logic in vscode just triggers those paths on reconnect. Just a wild idea 😺

[mmis1000]

Em...A standardized terminal multiplexer protocol.

Something like language server protocol but for terminal? Actually define the protocol so we can have a shared way of handling terminals instead of every single terminal multiplexer use its own protocol, tmux use a, screen use b, whatever.

[jerch]

@mmis1000 Imho a standardized terminal multiplexer protocol is a slightly different story. What I meant above is more about to specify how the data can be provided on both ends (server/client) and to transport it securely. It is more about unifying the transport interface so custom daemons or aggregators can easily plug into it. How the transported data itself is shaped (thats what a full multiplexer protocol would address), is imho up to the daemon/aggregator in use.

Or in terms of tmux - currently ppl use ssh as transport to pair/duplicate to remote sessions, which itself is not disconnect tolerant. Here a more versatile protocol could come handy, that already offers typical TTY orchestration needs. Whether it will be adopted, ofc depends on its security offers (competing with ssh it a tough one 😉) and ease of implementation/maintenance (a full stack HTTPS based protocol with rest API and JSON buckets prolly will be rejected by most multiplexer maintainers).

[mmis1000]

@jerch But connect back to the terminal without replaying everything is exactly the only thing a terminal multiplexer must do. Without such capacity. You will need to send every byte the process write to stdout over the Ethernet. And the IPC between vscode process, your network, the client is probably going to suffer.

In case of the ssh, such accident like cat xxx.mp4 is normally going to hang your client completely, the only thing you can do is close it and reopen a new one.

But as a terminal, the only thing you need is the last several lines (depend on your screen and back buffer size) that is enough to fit into your screen. You don't really care the rest because you can't see them anyway. You will want to just skip over the giant amount of data and show the last several line. Such capacity is actually impossible without some sort of terminal multiplexing.

BTW, the reason somebody complaining the scroll in tmux is due the how tmux handle back buffer. The tmux ate the back buffer completely and implement scroll on its own, resulting in the client is no longer able to mouse scroll the screen.

[jerch]

@mmcru Well the replay and the pairing ideas are just two small aspects of the much broader field of terminal multiplexing. Both prominent multiplexers screen and tmux solve that in their very own way with true emulation in between. Imho thats not the goal here. Ofc a well written protocol could go that deep into details, but it again would need screen and tmux implementing those details with a daemon and aggregator component. (Why should they - they already have their genuine abstraction for that.)

Imho the idea here is much more basic - as long as a terminal endpoint is attached, the data gets delivered right away (no to only a very small buffer on daemon side). Thats important for live sessions, as otherwise the latency will get annoying. There is no emulation in between. What I meant here as read/write multiplexing is something different - ideally the daemon knows how to create new readers or even writers (for shared read-only or read-write terminal sessions). This would allow easy "terminal collaboration", like attaching another vscode aggregator of person B to person A's session. Ofc this will need some security measures, otherwise it should never leave the testbed thinking sandbox. Why I nag about ssh above is - it is not disconnect tolerant, hard to setup right in cloud/container envs etc - thus it might be worth to think about its own security layer.

Beside that, there is no further multiplexing idea (esp. no notion about mapping multiple terminal views into one bigger one - the original multiplexing idea).

But connect back to the terminal without replaying everything is exactly the only thing a terminal multiplexer must do.

Why do you think so? A terminal recorder would have to do that, but is no multiplexer. Also a multiplexer could skip that feature (and most do, as they instead do in between emulation and just hook into that permanent background live session).

You will need to send every byte the process write to stdout over the Ethernet.

Thats always the case for attached sessions. There is no way around it, as terminal data is stateful and not layered/boxed by any means (every follow up state might depend on every previously activated state, even if it was set 10 days ago or happened 10GB of data earlier). Btw thats the real challenge of implementing a good working replay.

In case of the ssh, such accident like cat xxx.mp4 is normally going to hang your client completely, the only thing you can do is close it and reopen a new one.

If properly implemented, nope. If this happens with some stack of tooling - it is a hint, that some component does not deal well with flow control (then it might even segfault at some point).

But as a terminal, the only thing you need is the last several lines (depend on your screen and back buffer size) that is enough to fit into your screen.

Thats only what you think is needed as it is visible to you. A terminal emulator in fact holds much more state beside the last n-th buffer rows. Ofc the daemon would have to cut data in replay history somewhere to not get into storage nightmare, but it is not as simple as "save only the last n-th rows".

You will want to just skip over the giant amount of data and show the last several line. Such capacity is actually impossible without some sort of terminal multiplexing.

Not sure what you mean with that. Skipping over an existing stream of terminal data is possible without any multiplexer (in the narrow terminal multiplexer sense, in fact the daemon would be a multiplexer in the common sense yes), but it needs proper state parsing over the stream data. A replay then could be done by collecting up the initial state over the skipped parts, and start pumping stream data from that interesting point of data/time.

[mmis1000]

Ofc the daemon would have to cut data in replay history somewhere to not get into storage nightmare, but it is not as simple as "save only the last n-th rows".

That didn't work unless there is a reset control code somewhere in the response, then it can serve as a cut point.

Terminal is always stateful. Sentence after it write over the result of previous one. Cursor move relative to previous position. You can't get the correct result unless you actually replay all of it. Who replay all of does not matter, some may just split everything to client and let it do it, some do it at server and return the result of replay to client (the case of screen or tmux).

[jerch]

@mmis1000 Again not sure what you are trying to say, as it is somewhat self contra-dictionary:

That didn't work unless there is a reset control code somewhere in the response, then it can serve as a cut point.

True, but unreliable for a "skip-over" replay, as sequences like RIS or DECSTR typically dont happen during normal session run.

Still your observation here contradicts this:

You can't get the correct result unless you actually replay all of it.

If you find a state-destructive sequence (like RIS, and DECSTR to some degree), you dont have to replay it all. Note that for me there is a difference between full replay (as full emulation) vs. search accumulated data for for those points.

Again - thats not reliable, as it is likely to never happen (thus you are likely to never find a clean state entrypoint for replay beside the very beginning). But before replaying everything, there is another option you didnt consider - do some state back reporting from an active terminal endpoint, and link this to the stream data position. This way you get an unclean entrypoint (a replay from that point needs to apply the state itself first, before working on the follow-up stream data). Those unclean entrypoints can be managed by the daemon, even with discarding older stream data completely, while a healthy looking replay from there is still possible. No second full replay needed at all. (This is abit like doing terminal state snapshots, and starting from there.)

[mmis1000]

I'd consider assume you can always read data from client not reliable at all. As it always have been, client may lost/corrupt data for some reason, like a app crash, system BSOD. A recovery method that require client not encounter error at all is …weird? Even in the vscode case. If it is going to run in the browser, then things like such is very likely to happen.

[jerch]

@mmis1000 Sorry I cannot see how your last comment is related to anything above. Plz lets not jump to random conclusions without proper context, as it derails the discussion.

[mmis1000]

From the beginning, the only reason I think it is a terminal multiplexer protocol is the reconnect without resend everything', nothing more is included.

Terminal protocol is a stream of patch, and unlike video, you don't have a P frame somewhere by default to skip over to near the end of stream unless user issue a 'reset' manually (which is basically does not exist and does not matter at all because user seldom do that).

In order to able skip over without break the output completely. Somewhere in the system need to generate the P frame. So you can 'start from here' or you just can't (think of a video that trying to start from a I frame, it really can't).

It can be client (the terminal send all its last state to server, and the server response with it when reconnect), or it can be server (something like tmux but with far less function). Or probably mixed, server only do it if client is unavailable.

But I feel completely rely on client and assume the client will just do it well is a bad assumption, clients always break for variable of reasons. I am not not think of it. I am assume it is going to break.
So the only reliable option will be the server render it (always or on required).

Is there a third way can handle the reconnect correctly without show you a half broken screen?
Or show you a non broken screen on reconnect isn't one of the goal?

[jerch]

@mmis1000

There are different scenarios possible to get better fault tolerance (imho thats what you are trying to say above). But note that currently a short terminal disconnect/reconnect is simply not possible for vscode by any means, thus even a best effort approach, which has to bail out for rare edge cases, is already a huge overall win.

What you describe as "it is going to break" is currently the default case from where developing a better idea would start. Whether the final concept should aim for 99% fault tolerance or is good enough to cover most cases (80:20 rule), depends alot on the circumstances, like manpower to implement such a solution and last but not least its technical implications like additional runtime costs (more CPU/memory hungry for "play along emulation", storage needs on server side etc). Since I am not going to implement it, I wont judge about it. From a user's perspective anything into that direction will be better than now, as easy as that.

If you have some profound terminal interface experience and time to share, maybe try helping to get it shaped. I for myself have not enough time for that, thus can only give some ideas here.

[Bessonov]

Usually I close all terminals before exit because of "loss" of terminal tabs. But yesterday I didn't. Today I've noticed, that after I've connected to remote container my terminal was restored. It lost terminal tabs, but scroll history is here. It work even with multiple terminals. And it is able to processing executed commands. For example, I can execute sleep 60; echo "test", quit and connect again. It seems like it doesn't lost running command and after 60 seconds pass, it prints "test". Tested it with simulation of network failure and "reload window". It worked as expected.

For me, everything is already here. I'm very happy :D

(the only one wish - terminal tabs should be restored too or integrated)

Version: 1.51.0
Commit: fcac248
Date: 2020-11-05T18:16:10.374Z
Electron: 9.3.3
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Linux x64 5.4.0-56-generic

[Bessonov]

Unfortunately, it doesn't work every time. It seems like something dying after some time...

[Tyriar]

Done in #116449 🎉