Screencast API

[ebidel]

Fixes #478.

Want to get some feedback on this before finishing tests and api docs.

The shape of the API is similar to Tracing. This PR introduces page.screencast:

page.screencast.on('frame', frame => {
  ...
});

await page.screencast.start();
await page.goto(...);
// ... other stuff ...
const frames = await page.screencast.stop();

// user can save frames themselves
frames.forEach((frame, i) => {
  fs.writeFileSync(`frame_${i}.png`, new Buffer(frame, 'base64'));
});

Passing a path creates a .webm video file when you call stop(). The solution uses no dependencies, only web platform APIs! The downside is that capturing a 'recording' is real-time, meaning longer videos will take more time to complete.

await page.screencast.start({path: 'video.webm'});

[ebidel]

There also appears to be bugs in the DTP screen api:

• Navigating to another page while a screencast is running changes the viewport. Therefore, causing screenshots to be different sizes.
• DPR always defaults to 1.
• It would be nice if there were an option to specify FPS.

e.g. Using page.setViewport({width: 1000, height: 800, deviceScaleFactor: 2}); and then navigating to a new page:

DPR, width, height

[ebidel]

Typically you scale a canvas according to the DPR of the device so the images are clear. IOW, if the screencast api captured images at 1000x800 and DPR was 2, you'd want to create a 500x400 canvas. I think we can probably remove this since 1.) we're not accounting for DPR here and 2.) The DTP screen API doesn't respect DPR.

[graingert]

can we get a stream api?

const express = require('express');
const app = express();

async function screencastSteps(page) {
  await page.goto(...);
  // ... other stuff ...
}

app.get('/goofy', function(req, res) {
  res.setHeader('Content-Type', 'video/ogg');
  page.screencast.start(screencastSteps).pipe(res);
});

app.listen(9999);

[aslushnikov]

I especially love the "no dependencies" part, impressive.

Can you write frames iteratively so that you don't accumulate them all in the memory?

[ebidel]

For producing image frames, users can do:

const frames = [];
page.screencast.on('frame', frame => {
  frames.push(frame.buffer);
});

For the video chunks, it looks like MediaRecorder.start() accepts a time slice param:

If a timeslice property was passed into the MediaRecorder.start() method that started media capture, a dataavailable event is fired every timeslice milliseconds. That means that each blob will have a specific time duration (except the last blob, which might be shorter, since it would be whatever is left over since the last event). So if the method call looked like this — recorder.start(1000); — the dataavailable event would fire after each second of media capture, and our event handler would be called every second with a blob of media data that's one second long. You can use timeslice alongside MediaRecorder.stop() and MediaRecorder.requestData() to produce multiple same-length blobs plus other shorter blobs as well.

We can send back video chunks at 1s intervals or something, but I'm not sure how useful individual small chunks would be in practice. Users would still need to put them together. That said, it's just a one liner (new Blob([chunks], {type: 'video/webm'})) and might be nice to have for larger videos.

https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/ondataavailable

Thoughts?

[graingert]

@ebidel using an async iterator would allow users to request frames whenever they need them (it's a pull based async collection)

[ebidel]

@graingert async iterators are still a ways off. http://node.green/#ESNEXT-candidate--stage-3--Asynchronous-Iterators-for-await-of-loops

[RobertoUa]

Is it possible to support fixed FPS in Puppeteer? Or should it be first implemented in DevTools Protocol?

[ebidel]

@RobertoUa DT would need to support that. Once this PR lands, you could also write a script that encodes the frames at the desired rate, using ffmpeg or something.

[ebidel]

@aslushnikov anything else we want to change before I add tests and document?

Can you write frames iteratively so that you don't accumulate them all in the memory?

Were you referring to image frames or video chunks? For the former, we could write frames as they happen but I think that would mean changing the API to look like this:

await page.screencast.start({
  path: './' // optional. If specified, writes each frame image to the directory. Required if the `file` arg is specified.
  file: './path/to/video.webm', // optional. Creates the video file when `.stop()` is called.
});`

or make users save frames themselves and call an extra method to produce the video:

const frames = [];
page.screencast.on('frame', frame => {
  frames.push(frame.buffer);
});

await page.screencast.start();
// ... later ...
await page.screencast.stop(); // wouldn't return an array of frames :(
// ... later ...
const videoFileBuffer = await page.screencast.video(frames, {path: 'screencast.webm'});

...not as nice IMO but wouldn't store any frames in mem.

[graingert]

@ebidel how about #881 (comment)

[graingert]

Then you don't need a "start/stop"

[ebidel]

@graingert looks neat, but that's a different shape than the rest of the codebase/API. We don't use streams anywhere else AFAIK. In the future, I'm imaging we could add other APIs like page.screencast.pause() to start/stop screencast from different parts of a script. Not necessarily tie the screencasting to a list of actions in a callback.

[julien-c]

I would also like to get video chunks iteratively (to stream them to ffmpeg for transcoding, for instance)

[julien-c]

Also wanted to add that this is really exciting. As someone who was generating frames one by one in Phantom.js to create videos, there are so many things that this is going to enable. Great job @ebidel 🙏

[aslushnikov]

we don't want to use this api for video streaming; it gonna be slow and low-quality. We'd need to introduce a nicer API for the protocol first to have this feature in pptr.

[aslushnikov]

Closing this in favor of upstream implementation: https://bugs.chromium.org/p/chromium/issues/detail?id=781117

[azohra]

:( seems like the upstream stalled out almost right after it started. Is anyone able to shed light of if this feature is still being pursued? Thanks for all the hard work! #478 and this were a very interesting read.