Upgrade to ffmpeg v4.4 and use new detector pre-loading API

[jailuthra]

What does this pull request do? Explain your changes. (required)

Upgrade ffmpeg and LPMS to use v4.4, a better detector API and misc. transcoding fixes.

Specific updates (required)

81a4e06
In this commit we switch to the new session pre-loading API introduced in livepeer/lpms#246 (comment)

Earlier we used to initialize FFmpeg at the time of node startup using InitFFmpegWithDetectorProfile - which would internally load a tensorflow model and share it across instances of our scene classification filter. We did that as it was an expensive (~3s) operation that couldn't be done when a new stream came in.

The earlier approach was problematic because we shared (a non-thread-safe) Tensorflow graph across concurrent go-routines, and it was inflexible as we could only initialize one detection model per node.

In this commit, I've mostly sticked close to the legacy idea of "pre-loading a single model" at node startup, but using the newer transcode-session API. Whenever a new stream comes in, we now initialize a new detector session, which has an impact of (~500ms) on the transcode time of the first segment.

This meets the real-time threshold on my local testing, but the plan is to improve the load balancer to have a constant MinPreinitializedDetectorSessions always ready at hand before a stream comes in. Doing that will reduce the additional overhead introduced by the new API, while also preventing any thread-safety issues in the Tensorflow models.

See the commit history for rest

How did you test each of these updates (required)

• Local benchmarks and B-O-T Testing
• Test in staging
• Test in a single production region

Does this pull request close any open issues?

Fixes #1962

Checklist:

• Read the contribution guide
• make runs successfully
• All tests in ./test.sh pass
• README and other documentation updated
• Pending changelog updated

[yondonfu]

Just a few non-blocking questions, but LGTM

[yondonfu]

Just a note that can be addressed outside of this PR:

I think this API is a bit confusing. While I know that the device ID arg is only relevant for initializing the DNN filtergraph, when I first read this function my first thought is that the device ID would be used for both the transcoder and the DNN filtergraph.

[yondonfu]

Mind also adding an entry to PENDING_CHANGELOG?

[jailuthra]

Tested the last fix in a multi-GPU setup and it worked as expected. Rebased and added a PENDING_CHANGELOG entry.

Holding on a merge until the new go-livepeer release is officially out.

[yondonfu]

Tested the last fix in a multi-GPU setup and it worked as expected.

🥳 🎉

Holding on a merge until the new go-livepeer release is officially out.

The release was just published so good to merge after resolving the conflict 👍 .

[yondonfu]

Btw I forgot to clear CHANGELOG_PENDING in #1986, but looks like the commit with the CHANGELOG_PENDING entry will clear all the old entries anyway 👍 .