Thoughts on nodegui/react as a host

[jakerr]

I was thinking there might be a performance benefit to embedding remotion into a host written in nodegui/react so that frames could be captured and manipulated natively by Qt. This might allow bundling a remotion composition into a package that can be run cross platform (cloud or wherever) without any dependency on chromium. I haven't dug so deep but I imagine Qt might even be able to do the frame / audio stitching, such that there would be no need for an ffmpeg dependency either.

I realize this would be a pretty huge project since the current remotion/core expects to be run in the context of a browser where HTMLElements and such web apis are depended on. So the trick to supporting both might be to extract that environment specific stuff into a package that can be swapped out (similar to what some react native / react.js projects do.) Easier said than done, right?

On a related note, I just sent a pull request to nodegui to enable screen grabbing of widgets which if accepted I imagine would be pretty convenient for a render host as the one I'm suggesting here.

[JonnyBurger]

Hey @jakerr!

Thanks a lot for thinking outside the box and about possible use cases for Remotion!

I find the most empowering thing about the web is that graphically, it has an insane amount of capabilities: Shapes, Video embeds, shadows, 3D capabilities, transforms, vector graphics etc.

Furthermore, the web is the most popular platform nowadays for creating UIs and the community of people who already have knowledge of it and therefore can leverage it is super big.

Another benefit is that web-based videos can be embedded into websites to create live previews of how a rendered video would look like (I'm talking about https://remotion.dev/player specifically).

In my implementation, I am trying to avoid patterns that assume that you are on the web, so that in the future, more platforms like Qt or React Native could be supported. Multi-platform is after all also great strength of React.

We are not so far yet and also, to be honest, for the reasons I mentioned I am way more excited about the web than Qt.

[jakerr]

Hi Jonny!

Thanks so much for your thoughtful response. Those are excellent points and I totally agree with you that web technologies are some of the most powerful tools for getting visually appealing results. So it totally makes sense why that should remain the focus of this project!

In my implementation, I am trying to avoid patterns that assume that you are on the web, so that in the future, more platforms like Qt or React Native could be supported. Multi-platform is after all also great strength of React.

This is great to hear that you do consider this as you implement new features. I had actually guessed that that was the case given how few the actual dependencies seemed to exist in remotion/core at this point. The API surface was small enough that I thought this crazy idea might actually be feasible.

... for the reasons I mentioned I am way more excited about the web than Qt.

Totally fair! I think you're doing the right things. I was just inspired to mention other possible hosts because there are a lot of things that remotion does well that are totally portable. The custom hooks for currentFrame, the concept of sequences automatically having their current frame offset by their parent's. Components like <Freeze> These are really interesting tools for coding a composition that make sense in so many contexts even outside of web.

All that said I guess my main motivation was really to consider anything that might improve headless render performance, but upon further thought having to re-implement a composition to use Qt to get that is probably too big of a cost for most people to pay anyhow. I have noticed a couple of discussions here around explorations into performance improvements such as embedding ffmpeg with wasm, or Chromium's latest --headless improvements. I'll keep my eye out for more in this area!

Thanks for everything you do. Remotion is awesome!