add Json serializable API for Buffer.ts and BufferLine.ts

[JavaCS3]

Hello there, I'm working on a kind of terminal recording playback tool (similar to https://github.com/asciinema/asciinema-player) with xterm.js.
I think it would be better for xterm.js to have ability to create something like a snapshot of current buffer state. And restore to any snapshot of a state. So I create this PR.
I'm glad if we can discuss about this PR.

[Tyriar]

Hi @JavaCS3,

FYI the issue for this is #595.

I had a chat with @jerch about this and we see these problems with the current approach of including it into core:

• We would need to expose API to save/restore state but we definitely don't want to commit to a particular format so this would probably break every few versions
• It adds a bunch of bloat to the core when so few users will actually use it

Given this, we think the best way to approach this feature is to use an addon (example xterm-addon-serialize) that serializes state using the new buffer API (Terminal.buffer.*). The serialize method could look like this:

/**
 * Serializes terminal rows into a string that can be written back to the terminal
 * to restore the state. The cursor will also be positioned to the correct cell.
 * When restoring a terminal it is best to do before `Terminal.open` is called
 * to avoid wasting CPU cycles rendering incomplete frames.
 * @param rows The number of rows to serialize, starting from the bottom of the
 * terminal. This defaults to the number of rows in the viewport.
 */
serialize(rows?: number): string;

If you're unfamiliar with terminals the resulting string would look something like this "foo\n\rbar\x1b[2;4H" for a simple row (print "foo", go to next line and print "bar" then position the cursor to row 2 and column 4).

This has a bunch of benefits:

• Consumers who don't use it don't need to load the code
• Consumers who do use it can dynamically load the code
• It uses public API that is declared as stable and serializes into a standard format that could be used on any terminal
• It leverages all of @jerch's awesome work speeding up the parser
• It defaults to just the viewport which would be lightning fast to both serialize and restore
• It doesn't complicate/bloat the core architecture

Note that this would not be able to support color or other styles until we expose attributes in the API but that's something we want to do and could improve upon it at a later time.

[jerch]

@JavaCS3 Many thanks for this well coded PR and your efforts ❤️.

We still kinda have to disappoint you, as it does not suit our philosophy to cleanup the codebase from rarely used parts with v4 - many things will move into addons to keep the terminal core slim. Nevertheless feel free to come up with an addon as Tyriar already sketched up.

[JavaCS3]

@Tyriar Thanks for your advice. Let me try another PR.

[JavaCS3]

@Tyriar Is there anywhere I can find TTY control char table? I only know a little like: \033[0;30m. Many thanks

[Tyriar]

@JavaCS3 This is what I use https://invisible-island.net/xterm/ctlseqs/ctlseqs.html

I suggest at least to start with that you assume it's the first thing done in the terminal and just print lines like normal, and just using the default color for now (if this all works we can talk about how we expose colors in the API) so you would only end up using \x1b[row;colH for that for the final cursor position (provided it's not in the last position). For typical usage outside of applications like vim you wouldn't even need to set the cursor so it would end up looking like this:

line 1\n\rline 2\n\rline3\n\rline4

[JavaCS3]

@Tyriar Sounds good

[JavaCS3]

@Tyriar How to run unittest in addons folder? I have trouble doing that. API test seems working but I don't need that currently.

Please check JavaCS3@36d0354 If you have time.

I add a test-addons cmd in package.json, but when I run yarn run test-addons It failed with Error: Cannot find module 'xterm'

Many thanks

[Tyriar]

It looks like you actually want to use an integration test which are named .api.ts, See AttachAddon.api.ts for a simple example of one

[mofux]

Coming back to the original request of recording and replaying a terminal recording like asciinema - I've implemented something like that a couple of years ago using a technique similar to how asciinema does it.

Back in that time, I've been recording the data events on the pty stream, and giving them a timestamp:

// this runs in node.js
const records = [];
pty.on('data', (data) => {
  records.push({ time: Date.now(), data: data.toString() })
});

Once the pty session ends, you can serialize the records, e.g.:

fs.writeFileSync('/path/to/recording.js', JSON.stringify(records));

You will end up with a file very similar to what the asciinema recorder uses

Once you have that file, all you need to do is to parse the records, and then write the data back to xterm.js based on the recorded time:

// this code runs in the browser
// intialize xterm
const terminal = new Terminal();

// load the records from the file
const records = await fetch('/path/to/recording.js').then((res) => res.json());

// helper function to wait x ms
const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

// holds the last timestamp we've processed
let time;

// write records based on timing
for (let i=0; i < records.length; i++) {
  let record = records[i];
  if (!time) time = record.time;
  await wait(record.time - time);
  terminal.write(record.data);
  time = record.time;
}

The code above is untested and incomplete, but I hope it shows you a rough direction.

[jerch]

Btw there is a very old command in linux that already does that:

#> script

It tees the whole pty output into a file, that can later be replayed.

[Tyriar]

@jerch omg, that is unbelievably useful, why are you just telling me this now 😛

[jerch]

Lol its unix land, there is a handy tool for everything (well almost, cloning your grandma does not yet work).

[JavaCS3]

@mofux

Your approach is mostly right. But when you need to seek to any frame like a video player. There's two ways I can find to do that

1. Clear screen and write all recording data before a timestamp at one shot.
2. Cache as many key frames (like every 1/3 sec) as possible. Clear screen firstly and find the closet key frame of a timestamp and restore that frame.

The performance of the first approach is not that good when you directly jump to almost the end of a terminal "video". Because you need to flush a lot of intermediate data.

So my approach is the 2nd one. And I need an API to make a snapshot of a terminal. Maybe there's another solution.

Besides that, maybe serialize API can serialize to HTML snapshot as well if needed.

[mofux]

Yeah, the seek problem is quite hard to solve. The problem is, that beside the graphical state in the buffer, there is also a lot of other state that tells the terminal how to interpret things. There are modes for mouse handling, states that tell which buffer is currently active (a terminal has a normal and a alt buffer) and so on. If you want to jump to a certain snapshot, you would also have to serialize and deserialize that state as well. And then there's the problem of the current terminal size. What to do if you want to deserialize a state of a 30x10 terminal into a 40x20 terminal? How would you deal with terminal resizes during your recorded session?

I think a better approach here would be like this:

• Capture the pty stream data just as described above
• Use xterm.js with the DOM renderer enabled
• Write the captured pty stream to xterm.js, chunk by chunk
• After a chunk was written to xterm.js, capture the HTML content of the DOM renderer as string (term.element.innerHTML)
• Save these snapshots with the timing information to a new file
• Write a player that reads this file and outputs the saved html content (no xterm required)

[jerch]

@JavaCS3
The problem starts where you talk about "frames" - a concept with frames cannot grab all the bits needed to replay every aspect of a terminal. If you are only interested in scraping the terminal with frames and time stamps, imho the task is pretty simple:

• grab viewport content with your framerate, save with timestamp
• optional: save only diffs, thus if nothing changed, skip a frame
• learn from mpeg: do a full frame every n-th frames to lower the seeking burden
• save viewport dimensions along with the other data to spot resizes

Now your player only needs draw the content at a certain time with correct dimension. How it draws the content does not matter (let it be a gif, a html box, or even an mpeg).

I know I asked/said this several times already - imho you need to get your goals straight, to know what you want to support and what not. E.g. if you want to be able to fully replay stuff - do as @mofux pointed out - save the pty stream with timestamps and replay everything in a full xterm.js instance.