capture video question

[t-io]

I stumbled upon you tweet and this repo and was wondering how you recorded this video.
I tried something similar local where I took several screenshots and stitched them together using ffmpeg. I always had problems getting a good frame rate resulting in snatchy videos.
So I am really interested in how you tackled this problem because your video look smooth.
Also I was wondering if the whole process can really be done on lambda because of the 1 minute timeout.

[adieuadieu]

Hi @t-io,

Yes; the image I posted in that tweet was really produced on AWS Lambda! A Lambda function can run for 5 minutes. The caveat here is that, if you're invoking the Lambda function via the API Gateway, you're limited by the request timeout (which is 30 seconds—I'm not sure whether it's possible to change this in API Gateway). So, if you want longer-than-30-ish-second recordings you'd have to write them to S3 or something similar and invoke the function some other way than directly via API Gateway.

Below is the handler code for use with serverless-chrome which I used to generate the video in the tweet (it's based on this handler.) Note that it's flawed: Since writing this code, I've come to learn that the Page.startScreencast method will only trigger a Page.screencastFrame event for every rendered frame that's produced. This means that, rather than producing a frame at a specific frame-rate, it'll only produce a frame when the browser has rendered something new. In other words, a frame is only produced when there are changes on the page. This is probably why you're experiencing difficulties getting a good frame rate that doesn't result in a jumpy video. I had to fiddle with the frame-rate parameters extensively to get the video in my tweet to look right.

To implement this properly, you'd have to measure the time in between each screencastFrame (there's a timestamp in the event's metadata parameter), then when assembling the video, tell ffmpeg the duration each frame should last when rendering the video. I haven't figured out how to do this with ffmpeg. However, it might be more easily achievable with gifencoder. Take a look at this for an example of someone who's done exactly that.

---

config.js

import experimentalHandler from './experimental'

export default {
  chromeFlags: [
    '--window-size=1280x1696',
    '--ignore-certificate-errors', // Dangerous?
  ],
  logging: true,
  handler: experimentalHandler, 
}

experimental.js

import fs from 'fs'
import path from 'path'
import { spawn, execSync } from 'child_process'
import Cdp from 'chrome-remote-interface'
import config from '../config'
import { log, sleep } from '../utils'

const defaultOptions = {
  captureFrameRate: 1,
  captureQuality: 50,
  videoFrameRate: '5',
  videoFrameSize: '848x640',
}

const FFMPEG_PATH = path.resolve('./ffmpeg')

function cleanPrintOptionValue (type, value) {
  const types = { string: String, number: Number, boolean: Boolean }
  return value ? new types[type](value) : undefined
}

function makePrintOptions (options = {}) {
  return Object.entries(options).reduce(
    (printOptions, [option, value]) => ({
      ...printOptions,
      [option]: cleanPrintOptionValue(typeof defaultOptions[option], value),
    }),
    defaultOptions
  )
}

export async function makeVideo (url, options = {}, invokeid = '') {
  const LOAD_TIMEOUT = (config && config.chrome.pageLoadTimeout) || 1000 * 20
  let result
  let loaded = false
  let framesCaptured = 0
  const requestQueue = [] // @TODO: write a better queue, which waits when reaching 0 before emitting "empty"

  const loading = async (startTime = Date.now()) => {
    log('Request queue size:', requestQueue.length, requestQueue)

    if ((!loaded || requestQueue.length > 0) && Date.now() - startTime < LOAD_TIMEOUT) {
      await sleep(100)
      await loading(startTime)
    }
  }

  const tab = await Cdp.New()
  const client = await Cdp({ host: '127.0.0.1', target: tab })

  const { Network, Page, Input, DOM, Overlay } = client

  Network.requestWillBeSent((data) => {
    // only add requestIds which aren't already in the queue
    // why? if a request to http gets redirected to https, requestId remains the same
    if (!requestQueue.find(item => item === data.requestId)) {
      requestQueue.push(data.requestId)
    }

    log('Chrome is sending request for:', data.requestId, data.request.url)
  })

  Network.responseReceived(async (data) => {
    // @TODO: handle this better. sometimes images, fonts, etc aren't done loading before we think loading is finished
    // is there a better way to detect this? see if there's any pending js being executed? paints? something?
    await sleep(100) // wait here, in case this resource has triggered more resources to load.
    requestQueue.splice(requestQueue.findIndex(item => item === data.requestId), 1)
    log('Chrome received response for:', data.requestId, data.response.url)
  })

  // @TODO: check for/catch error/failures to load a resource
  // Network.loadingFailed
  // Network.loadingFinished
  // @TODO: check for results from cache, which don't trigger responseReceived (Network.requestServedFromCache instead)
  // - if the request is cached you will get a "requestServedFromCache" event instead of "responseReceived" (and no "loadingFinished" event)
  Page.loadEventFired((data) => {
    loaded = true
    log('Page.loadEventFired', data)
  })

  Page.domContentEventFired((data) => {
    log('Page.domContentEventFired', data)
  })

  Page.screencastFrame(({ sessionId, data, metadata }) => {
    const filename = `/tmp/frame-${invokeid}-${String(metadata.timestamp).replace('.', '')}.jpg`
    framesCaptured += 1

    // log('Received screencast frame', sessionId, metadata)
    Page.screencastFrameAck({ sessionId })

    fs.writeFile(filename, data, { encoding: 'base64' }, (error) => {
      // log('Page.screencastFrame writeFile:', filename, error)
    })
  })

  if (config.logging) {
    Cdp.Version((err, info) => {
      console.log('CDP version info', err, info)
    })
  }

  try {
    await Promise.all([
      Network.enable(),
      Page.enable(),
      DOM.enable(),
    ])

    const interactionStartTime = Date.now()

    await client.send('Overlay.enable') // this has to happen after DOM.enable(), cuz i don't know why
    await client.send('Overlay.setShowFPSCounter', { show: true })

    await Page.startScreencast({
      format: 'jpeg',
      quality: options.captureQuality,
      everyNthFrame: options.captureFrameRate,
    })

    await Page.navigate({ url })
    await loading()
    await sleep(2000)
    await Input.synthesizeScrollGesture({ x: 50, y: 50, yDistance: -2000 }) // Scroll the page
    await sleep(1000)

    await Page.stopScreencast()

    log('We think the page has finished doing what it do. Rendering video.')
    log(`Interaction took ${Date.now() - interactionStartTime}ms to finish.`)
  } catch (error) {
    console.error(error)
  }

  // @TODO: handle this better —
  // If you don't close the tab, and a subsequent Page.navigate() is unable to load the url,
  // you'll end up printing a PDF of whatever was loaded in the tab previously (e.g. a previous URL)
  // _unless_ you Cdp.New() each time. But still good to close to clear up memory in Chrome
  try {
    log('trying to close tab', tab)
    await Cdp.Close({ id: tab.id })
  } catch (error) {
    log('unable to close tab', tab, error)
  }

  await client.close()

  const renderVideo = async () => {
    await new Promise((resolve, reject) => {
      const args = [
        '-y',
        '-loglevel',
        'warning', // 'debug',
        '-f',
        'image2',
        '-framerate',
        `${options.videoFrameRate}`,
        '-pattern_type',
        'glob',
        '-i',
        `"/tmp/frame-${invokeid}-*.jpg"`,
        // '-r',
        '-s',
        `${options.videoFrameSize}`,
        '-c:v',
        'libx264',
        '-pix_fmt',
        'yuv420p',
        '/tmp/video.mp4',
      ]

      log('spawning ffmpeg with args', FFMPEG_PATH, args.join(' '))

      const ffmpeg = spawn(FFMPEG_PATH, args, { cwd: '/tmp', shell: true })
      ffmpeg.on('message', msg => log('ffmpeg message', msg))
      ffmpeg.on('error', msg => log('ffmpeg error', msg) && reject(msg))
      ffmpeg.on('close', (status) => {
        if (status !== 0) {
          log('ffmpeg closed with status', status)
          return reject(`ffmpeg closed with status ${status}`)
        }

        return resolve()
      })

      ffmpeg.stdout.on('data', (data) => {
        log(`ffmpeg stdout: ${data}`)
      })

      ffmpeg.stderr.on('data', (data) => {
        log(`ffmpeg stderr: ${data}`)
      })
    })

    // @TODO: no sync-y syncface sync
    return fs.readFileSync('/tmp/video.mp4', { encoding: 'base64' })
  }

  try {
    const renderStartTime = Date.now()
    result = await renderVideo()
    log(`FFmpeg took ${Date.now() - renderStartTime}ms to finish.`)
  } catch (error) {
    console.error('Error making video', error)
  }

  // @TODO: this clean up .. do it better. and not sync
  // clean up old frames
  console.log('rm', execSync('rm -Rf /tmp/frame-*').toString())
  console.log('rm', execSync('rm -Rf /tmp/video*').toString())

  return { data: result, framesCaptured }
}

export default (async function experimentalHandler (event, { invokeid }) {
  const { queryStringParameters: { url, ...printParameters } } = event
  const options = makePrintOptions(printParameters)
  let result = {}

  log('Processing Videoification for', url, options)

  const startTime = Date.now()

  try {
    result = await makeVideo(url, options, invokeid)
  } catch (error) {
    console.error('Error printing pdf for', url, error)
    throw new Error('Unable to print pdf')
  }

  // TODO: probably better to write the pdf to S3,
  // but that's a bit more complicated for this example.
  return {
    statusCode: 200,
    // it's not possible to send binary via AWS API Gateway as it expects JSON response from Lambda
    body: `
      <html>
        <body>
          <p><a href="${url}">${url}</a></p>
          <p><code>${JSON.stringify(options, null, 2)}</code></p>
          <p>It took Chromium & FFmpeg ${Date.now() - startTime}ms to load URL, interact with the age and render video. Captured ${result.framesCaptured} frames.</p>
          <embed src="data:video/mp4;base64,${result.data}" width="100%" height="80%" type='video/mp4'>
        </body>
      </html>
    `,
    headers: {
      'Content-Type': 'text/html',
    },
  }
})

After deploying with Serverless, you can invoke the function via the API Gateway:
https://XXXXXXXX.execute-api.us-west-2.amazonaws.com/dev/chrome?url=https://bbc.com&captureFrameRate=1&videoFrameRate=1&captureQuality=50&videoFrameSize=640x848

[t-io]

Thanks for your detailed answer. I will try to figure out something to solve this with ffmpeg. If I come up with a solution I will let you know

[jmealo]

@t-io: How did you make out with this?

[t-io]

Hey @jmealo unfortunately I had to drop this solution. The framerate overall was to low to get a smooth video experience. To resolve this I used xvfb framebuffer with selenium and streamed the xvfb output to ffmpeg. This worked quiet nice.

[jmealo]

@t-io: That's exactly what I've done in the past! I'm glad you found a solution, I was just hoping it was better than what I had come up with😆.

[ygokirmak]

@t-io did u shared solution on somewhere?

[t-io]

@ygokirmak I quick and dirty hacked this python script together.
Maybe it helps you.

[hbakhtiyor]

hi @adieuadieu
for scrolling you used here

    await Input.synthesizeScrollGesture({ x: 50, y: 50, yDistance: -2000 }) // Scroll the page

and in the file

     await Runtime.evaluate({
      expression: `(
        () => {
          const height = document.body.scrollHeight
          window.scrollTo(0, height)
          return { height }
        }
      )();
      `,
      returnByValue: true,
})

which of them more reliable api?