Replace LIBAV with original FFMPEG

[sgothel]

Feature Request

While working on NVENC support #259
I have compared FFMPEG with LIBAV a little, backporting FFMPEG's NVENC implementation
to LIBAV.

Here I have discovered a few artifacts:

1. FFMPEGs NVENC version is more up-to-date and closer to the Nvenc spec,
   LIBAVs version is sort of outdated.

2. Most enhancements are being produced for FFMPEG first, then merely backported to LIBAV

• FFMPEG is more up-to-date regarding error handling and security
• FFMPEG is also more quick to adopt state of the art features
• Backports to LIBAV are performed by 3rd party developer and may introduce new hard to find
  issues bugs due to increasingly diverting API and lack of internal knowledge or time at hand.
• See Nvenc encoder is not supported yet! #259 (comment)
• See a libav maintainer https://lists.alioth.debian.org/pipermail/pkg-multimedia-maintainers/2015-April/043942.html

3. Above is also visible with the backporting, i.e. the format writer doesn't work properly using NVENC

• See cf7c027
• See 'academic' pull request Nvenc encoder libav #973 of trial branch https://github.com/sgothel/HandBrake/tree/nvenc-encoder-libav

4. Reasons for Debian (and hence Ubuntu) to revert to the original FFMPEG

• See https://lists.debian.org/debian-devel-announce/2015/07/msg00001.html
• See https://wiki.debian.org/Debate/libav-provider/ffmpeg
• See https://en.wikipedia.org/wiki/Libav#Fork_from_FFmpeg

HandBrake version (e.g., 1.0.0)

Master branch

Example

Commit sgothel@0466852
as used for the NVENC feature in branch https://github.com/sgothel/HandBrake/tree/nvenc-encoder
(pull request #971)

I could easily replace LIBAV 12.2 with FFMPEG 3.4, however, one would need to
re-validated the many LIBAV patches for FFMPEG 3.4 - if they still apply.
A first attempt disclosed multiple rejections and this task surely needs some effort.

Work Assignment

I would volunteer performing this work, if the transition to FFMPEG is supported.

[galad87]

I have nothing against switching to FFmpeg. Other developers opinions may not be the same as mine.

[sgothel]

• See a libav maintainer https://lists.alioth.debian.org/pipermail/pkg-multimedia-maintainers/2015-April/043942.html

[sr55]

We'll want to push 1.1 out the door (hopefully sooner rather than later) then we can look at what's involved in switching over in 1.2. We don't really want to rock the boat right now quite close to finishing up work that will allow us to release.

1.2 can focus on decode/encode improvements as a key point.

While personally, I'd rather stay with libav, they are falling behind in certain areas that interest us so I'm not going to object to a switch over. If we had more time ourselves to maintain and backport what we need, it might be a different story but that's not going to happen realistically.

[jstebbins]

I currently know of only one incompatibility between libav and ffmpeg. libav uses AV_CODEC_ID_SRT for SRT subtitles. ffmpeg uses AV_CODEC_ID_SUBRIP, even though they also have a define for AV_CODEC_ID_SRT. Since they have the define, HandBrake builds without error against ffmpeg. But it fails to recognize SRT subtitles.

I wouldn't be surprised if there are more such problems waiting for us to discover.

Some of our current libav patches already made it into libav git. And many of those have gotten into ffmpeg due to the libav merge effort ffmpeg continues. But we'll have to inspect the patches carefully because ffmpeg often decides to do things a little differently.

[sgothel]

@jstebbins yes ofc, that would be one important task for the ffmpeg move.
@sr55 how about creating a 1.2 branch and moving to ffmpeg on master .. or within its own branch?
Then you could merge my branch and we could work from there.

[sgothel]

Maybe you like to have a clean FFMPEG branch of mine to merge w/ NVENC changes on top?

[jstebbins]

An ffmpeg switch should be done as it's own PR. Any PR that needs to be built on top of an ffmpeg switch can rebase or merge with the ffmpeg switch PR as necessary.