Capturing animated elements

[mkherlakian]

Hi - first off I just want to say this is a really cool project. I'm trying to use it for something we're trying to build and that involves having an audio spectrum on the final video.
I used this component: https://www.npmjs.com/package/react-audio-spectrum which shows up in the preview, but (and I kinda expected that) it doesn't show in the final render (the audio file does play though)

export const Audio: React.FC = () => {
	const frame = useCurrentFrame();
	const opacity = interpolate(frame, [0, 30], [0, 1]);

	return (
		<div
			style={{
				fontFamily: 'Helvetica, Arial',
				fontSize: 40,
				textAlign: 'center',
				position: 'absolute',
				bottom: 140,
				width: '100%',
				opacity,
			}}
		        >
			<Audio
				// autoPlay
				id="audio-element"
				src={audio}
                         />
			<AudioSpectrum
				id="audio-canvas"
				height={500}
				width={700}
				audioId="audio-element"
				capColor="red"
				capHeight={2}
				meterWidth={2}
				meterCount={512}
				meterColor={[
    {stop: 0, color: '#f00'},
    {stop: 0.5, color: '#0CD7FD'},
    {stop: 1, color: 'red'}
				]}
				gap={4}
                 />			
		</div>
	);
};

Does anyone have any suggestions on how to make the frames capture this element's visuals?

[JonnyBurger]

This component doesn't work well with Remotion because it animates itself outside of React mechanisms, aka you render it once but it keeps changing itself. Remotion requires each frame to be static, so that the animation can come together frame-by-frame.

So while this library doesn't work well in Remotion, audio visualization would be a huge usecase for Remotion! For that we probably need some new primitives in the core for Remotion. Once all the essential bugs are fixed, I'm gonna start working on Audio support, and will think about an API that will make audio visualization easy. To keep up to date, I recommend subscribing to the Audio issue #28 so you know when it gets released.

[mkherlakian]

Thanks for the reply @JonnyBurger!

That makes sense. I was going through the code for the video component and trying to see if a similar approach could be implemented for that particular problem. Some other techniques use OfflineAudioContext to process the audio which might be a clue.

We might want to do contribute on some of the development work for the audio issue (I'm subscribed to issue #28) - If you can share some high-level guidance for that and audio visualization, we might take a crack at it (Specifically visualization if audio rendering is not a pre-requisite)...

[JonnyBurger]

Thanks for pointing to OfflineAudioContext! This looks like the right API to create audio visualizations to me. And you can probably make some visualizations with it from today on. The question is more if Remotion can provide some abstractions to make it easier for the end user.

If you would like to work on it or research it a bit, it would be awesome! I'd be up to chat and give guidance, for example on the new Discord.

For audio visualization to work in Remotion, we have to, based on the frame number, determine the audio 'waveform' at that timepoint. This has to happen asynchronously, so we have to use the delayRender API from Remotion for sure as the first step.

Then we can, for example inside an useEffect, use the right Web API to statically get the waveform data as an array of values over time and store it in a state. I found an awesome example of an open source audio editor which can get the waveform statically without having to play the audio! https://audiomass.co/ We should have a look and see which Web API it uses.

If we have the array and the frame number, we can determine the audio properties at a certain point in time. And render it with Remotion 😃