Add audio support

[yuta0801]

This PR adds audio support!

It might not work fine with many audios by limitation of ffmpeg and pattern detecting from audio data of every frame, so we need more tests.

[tomByrer]

Cool!
How can this be used? I don't see any README edits, or did I miss them?

[JonnyBurger]

Thanks a lot for this amazing PR!

I'm currently wrapping up some other features and bugs and trying to make a 1.4 release soon. So it will take a few days until I can take a look, but nonetheless I'm hyped!

[JonnyBurger]

@tomByrer: From what I understand, you simply place <Audio/> tags in the video and now they'll be included in the output!

@yuta0801: In the meanwhile, could you give me access to push on this branch? Maybe you have to invite me to your repo. This would be helpful if I need to reorganize, cleanup or add docs, would be nice to have everything in one branch.

[yuta0801]

@JonnyBurger
I checked "Allow edits by maintainers", so I think you can access to this branch.
If you can't or want other changes, I'll invite you to the repo.

[yuta0801]

It seems the stitching is broken after the merge, I don't know if there was any breaking changes or if the merge was wrong. I'm tracking it down...

[JonnyBurger]

@yuta0801 already figured it out, pushed a fix!

[JonnyBurger]

Seems to crash when there are no audio stream:

npx remotion render src/index.tsx shadow-circles --codec=h264 out.mp4

[ljbc1994]

Hey guys, great work so far! Could we add support for muted on the Video so it doesn't pick up audio? Thinking it should be as simple as checking for muted when registering the asset?

More than happy to add it if you need me to!

[JonnyBurger]

Great idea @ljbc1994! 💯
You are more than welcome to add it (can send a PR to this branch), and if not I'll ensure that it gets added before release.

[ljbc1994]

Great idea @ljbc1994! 💯
You are more than welcome to add it (can send a PR to this branch), and if not I'll ensure that it gets added before release.

On it now! Added the code to support it... but output is buggy so looking into it.

EDIT: PR ready here: yuta0801#1

[yuta0801]

I've investigated about the all flag of adelay support in ffmpeg

the adelay filter's all flag is not supported in ffmpeg 4.2.4 at least (since the feature was added at Oct 2019, and release note say the version "was cut from master on 2019-07-21")

The problem is there's no way to install compatible ffmpeg in ubuntu LTS (I cannot find it even though spent hours)

• apt repository only has ffmpeg 4.2.x for current ubuntu LTS, so we cannot just use apt install ffmpeg
• the PPA jonathonf/ffmpeg-4 is not supporting current ubuntu LTS
• snap repository only up to 3.4.1, and it seems be buggy, cannot input sequense images using %03d.jpeg
  • but it worked with ffmpeg 3.4.1 that I download from here. if that version is not wrong, ffmpeg binaries behavior might depends the building process, so it have no consistency

I've only success manually installed static build binary ffmpeg 3.4.2 from here...

[ljbc1994]

Thanks for investigating that @yuta0801, that's a shame that LTS Ubuntu doesn't have that functionality ..

In that case, we would have to find out the number of audio channels (through ffprobe) then use that to build up the adelay option (so essentially mimicking what the filter is doing)?

i.e.

`adelay=${new Array(channels).fill(startInVideo).join("|")}`

EDIT: @yuta0801, pushed new changes to remove filter and manually build up adelay from audio channel info for each asset. Also using defined concurrency so it's speedy if a lot of assets are used.

[JonnyBurger]

Woah awesome work @ljbc1994!! this will be awesome to support a broader range of versions of ffmpeg.

[JonnyBurger]

I committed sample video and audio, and worked a bit on the PR, audio testing and audio testing mute works well in preview and render! Still lots of things to do and test, but this is now my top priority.

I also merged main branch which fixes some embarrassing bugs in video playback in development. It's getting solid!

[JonnyBurger]

Today I added a bunch of tests, including one that takes JSX input and simulates a render, and compares it with the media output it gets. Will be very helpful to figure out the edge cases.

Also, I fixed the audio being low if you have multiple audio tracks, so now the volume, at least in early testing, is just like in the preview!

[ljbc1994]

@JonnyBurger Ah that multiple audio tracks is perfect! Also moving the filter generation makes absolute sense!

Be interesting to expound on how we could add audio filters (i.e. fade in / out audio), but guess that would be difficult as it'd require ffmpeg to process it first.

[JonnyBurger]

@ljbc1994 I am working on it as you say it 😂 hope to have support for it in a few hours! I'll give an update today.