Chrome download should include proprietary codecs

[EricLiclair]

I've been trying to generate videos using the ssr feature in remotion. Now, my components contains <Video /> and <Audio /> tags from remotion. But the stitchFramesToVideo() in @remotion/renderer doesn't stich those frames.
Help me if I am doing it in a wrong way.

While using yarn build is working fine and generates the video with audio and attached video file, but not working for the api i am trying to create.

server.js

/**
 * This is an example of a server that returns dynamic video.
 * Run `npm run server` to try it out!
 * If you don't want to render videos on a server, you can safely
 * delete this file.
 */
import { bundle } from '@remotion/bundler';
import {
  getCompositions,
  renderFrames,
  stitchFramesToVideo
} from '@remotion/renderer';
import cookieParser from "cookie-parser";
import cors from "cors";
import express from 'express';
import fs from 'fs';
import os from 'os';
import path from 'path';
import { fps } from './src/utils.mjs';

const app = express();
app.use(cors());
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());

const port = process.env.PORT || 8000;
const compositionId = 'HelloWorld';

const cache = new Map();

app.post('/', async (req, res) => {
  // function to send file through the api
  const sendFile = (file) => {
    fs.createReadStream(file)
      .pipe(res)
      .on('close', () => {
        res.end();
      });
  };

  try {
    // checks if video exists in cache
    if (cache.get(JSON.stringify(req.body))) {
      sendFile(cache.get(JSON.stringify(req.body)));
      return;
    }

    const bundled = await bundle(path.join(process.cwd(), './src/index.jsx'));
    const comps = await getCompositions(bundled, { inputProps: req.body });
    const video = comps.find((c) => c.id === compositionId);
    if (!video) {
      throw new Error(`No video called ${compositionId}`);
    }

    // set response type
    res.set('content-type', 'video/mp4');

    // a temp dir
    const tmpDir = await fs.promises.mkdtemp(
      path.join(os.tmpdir(), 'remotion-')
    );

    // rendering frames
    const { assetsInfo } = await renderFrames({
      config: video,
      webpackBundle: bundled,
      onStart: () => console.log('Rendering frames...'),
      onFrameUpdate: (f) => {
        if (f % 10 === 0) {
          console.log(`Rendered frame ${f}`);
        }
      },
      parallelism: null,
      outputDir: tmpDir,
      inputProps: req.body,
      compositionId,
      imageFormat: 'jpeg',
    });

    const finalOutput = path.join(tmpDir, 'out.mp4');

    // stitching frames into a video
    await stitchFramesToVideo({
      dir: tmpDir,
      force: true,
      fps: fps,
      height: req.body.height || video.height,
      width: req.body.width || video.width,
      outputLocation: finalOutput,
      imageFormat: 'jpeg',
      assetsInfo,
      onProgress: (f) => {
        console.log(`Stitched frames ${f}`);
      }
    });
    cache.set(JSON.stringify(req.body), finalOutput);

    sendFile(finalOutput);
    console.log('Video rendered and sent!');
  } catch (err) {
    console.error(err);
    res.status(400)
    res.json({
      error: err,
    });
  }
});

app.listen(port);

console.log(
  [
    `The server has started on http://localhost:${port}!`,
    'You can render a video by making an api call',
    '',
    'If you are running Hello World, try this:',
    '',
    `http://localhost:${port}/`,
    '',
  ].join('\n')
);

HelloWorld.jsx (the component which holds the complete video sequencing)

import Box from '@mui/material/Box';
import { Sequence, Video } from 'remotion';
import jsonData from '../data.json';
import defaultaudio from './assets/aud.mp3';
import signature from './assets/signature.mp4';
import NewAudio from './Components/Audio';
import { Image } from './Components/Image';
import { Text } from './Components/Text';
import { Title } from './Components/Title';
import { createDataMap, watermark_duration } from './utils.mjs';


export const HelloWorld = ({ fps = 30, width, height, data, background, durationInFrames, audio }) => {
  console.log(data, "from helloworld.jsx")
  const propData = data ? createDataMap(data) : createDataMap(jsonData.data);
  const audio_start = 0;
  const audio_end = durationInFrames;
  const audio_duration = durationInFrames;

  return (
    <div style={{ flex: 1, backgroundColor: background || "#FFFFFF" }}>
      <Sequence from={audio_start} durationInFrames={audio_duration}>
        <NewAudio src={audio || defaultaudio} duration={audio_duration} />
      </Sequence>
      {
        propData.map((sequence, seq_id) => {
          return (
            <Box key={seq_id}>
              <Sequence
                from={sequence.sequence_start}
                durationInFrames={sequence.sequence_duration}
              >
                <Image
                  top={'50%'}
                  left={'70%'}
                  src={sequence.img.src}
                  // imgHeight={1920}
                  // imgWidth={1920}
                  duration={1 * fps}
                />
              </Sequence>
              <Sequence
                from={sequence.title.title_start}
                durationInFrames={sequence.title.title_duration}
              >
                <Title title={sequence.title.title} duration={1 * fps} />
              </Sequence>
              {sequence.texts.map((text, idx) => {
                return <Sequence key={idx} from={text.text_start} durationInFrames={text.text_duration}>
                  <Text text={text.text} duration={1 * fps} idx={idx} />
                </Sequence>
              })}
            </Box>
          );
        })
      }
      <Sequence from={durationInFrames} durationInFrames={watermark_duration}>
        <Box sx={{ width: "100%", position: 'fixed', top: "50%", left: "50%", transform: "translate(-50%, -50%)" }}>
          <Video src={signature} width="100%" />
        </Box>
      </Sequence>
    </div >
  );
};

[JonnyBurger]

Hi @EricLiclair!

Thanks for the detailed issue!
I could see a few reasons for it:

• You are using a different browser for rendering, that does not support the proprietary MP4 and MP3 codecs. I've heard this is often the case inside Docker containers
• Also check for the following scenario: The input props (that are the URL parameters req.body in this case) did change the video in a way that breaks it

If neither of those seem to be the case, can you describe more precisely how it does not work? Is there an error message, or a timeout?

[EricLiclair]

I can confirm for the input props to be working fine with the api because I've tested it on the same data in the dev environment.
Also, I receive a timeout with the following error.

TimeoutError: waiting for function failed: timeout 30000ms exceeded
    at new WaitTask (/home/shubham/dev/proj/remotion/ssr/node_modules/puppeteer-core/lib/cjs/puppeteer/common/DOMWorld.js:526:34)
    at DOMWorld.waitForFunction (/home/shubham/dev/proj/remotion/ssr/node_modules/puppeteer-core/lib/cjs/puppeteer/common/DOMWorld.js:477:26)
    at Frame.waitForFunction (/home/shubham/dev/proj/remotion/ssr/node_modules/puppeteer-core/lib/cjs/puppeteer/common/FrameManager.js:993:32)
    at Page.waitForFunction (/home/shubham/dev/proj/remotion/ssr/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Page.js:2479:33)
    at seekToFrame (/home/shubham/dev/proj/remotion/ssr/node_modules/@remotion/renderer/dist/seek-to-frame.js:9:16)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /home/shubham/dev/proj/remotion/ssr/node_modules/@remotion/renderer/dist/render.js:98:13
    at async Promise.all (index 214)
    at async renderFrames (/home/shubham/dev/proj/remotion/ssr/node_modules/@remotion/renderer/dist/render.js:86:5)

While the dev works fine, the renderer is throwing a timeout error while ssr or even when using yarn build

shubham@ericware:~/dev/proj/remotion/ssr$ yarn build
yarn run v1.22.17
$ remotion render src/index.jsx HelloWorld out/video.mp4
📦 (1/3) [====================] Bundled code 2003ms
🖼  (2/3) [==================  ] Rendering frames (4x) 2104/2250
The following error occurred when trying to render frame 2104:
Error: A delayRender was called but not cleared after 28000ms. See https://remotion.dev/docs/timeout for help. The delayRender was called: 
    at delayRender (http://localhost:3000/bundle.js:10433:32)
    at http://localhost:3000/bundle.js:12060:56
    at fk (http://localhost:3000/bundle.js:6919:359)
    at exports.unstable_runWithPriority (http://localhost:3000/bundle.js:12358:343)
    at gg (http://localhost:3000/bundle.js:6779:325)
    at Oj (http://localhost:3000/bundle.js:6918:308)
    at http://localhost:3000/bundle.js:6918:215
    at V (http://localhost:3000/bundle.js:12356:224)
    at MessagePort.F.port1.onmessage (http://localhost:3000/bundle.js:12352:346)
    at http://localhost:3000/bundle.js:10439:19

💡 Get help for this issue at https://remotion.dev/docs/timeout
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[EricLiclair]

This is the browser it downloads by default

No local browser could be found. Downloading one from the internet...
Downloading Chromium r950341 - 142 Mb [===========         ] 54% 19.5s 

I am unsure why it works using the CLI and generates the video with audio but doesn't work via the api.

[JonnyBurger]

Thanks @EricLiclair!

i can confirm this is most likely a bug that it downloads a browser that doesn’t have the codecs. We’ll try to improve that. Can you name the exact OS and CPU architecture that you are on?

In the meanwhile, you need to get a Chromium browser executable manually and tell Remotion to use it: https://www.remotion.dev/docs/cli#--browser-executable

Sorry for the inconvenience!

[EricLiclair]

Sure. Also, I found out a work-around for my use case, as mentioned in the docs here.
Because I knew my server was using a chromium browser, I converted the assets (video and audio) into webm, and then it worked all fine.

As for the system details:
I am using Windows Subsystem for Linux with Ubuntu 20.04.3 LTS, on Window 11 64-bit operating system, x64-based processor

[JonnyBurger]

Thanks! I can confirm that the automatic download For Linux downloads a version of Chrome that doesn't support media codecs. We will fix that by supplying our own version of Chrome in that case. Leaving the issue open.