Add detection filter

[oscar-davids]

Add dnn filter for detection(livepeer/lpms#181).

To use this filter, need to install the TensorFlow for C library.
See https://www.tensorflow.org/install/install_c and configure FFmpeg with --enable-libtensorflow.
We can write the result of the detection filter in a file.
Added "lavfi.lvpdnn.text" metadata.

filter name

lvpdnn

parameter

• model: trained tensorflow model path.(file extention is .pb)

• input: tensorflow input layer name(in our model is input_1)

• output: tensorflow output layer name(in our model is reshape_3/Reshape)

• sample: detect every sample frames.

• threshold: threshold for inference confidence.

• log: path for save inference results.

Benchmark and example

1. Software transcoding:

ffmpeg -i input.mp4 -vf lvpdnn=model=tmodel.pb:input=input:output=outshape:sample=1:threshold=0.5:log=outlog.txt out.mp4

2. Hardware transcoding:

ffmpeg -vsync 0 -hwaccel cuvid -c:v h264_cuvid -i input.mp4 -c:a copy -vf lvpdnn=model=tmodel.pb:input=input:output=outshape:sample=1:threshold=0.5:log=outlog.txt -c:v h264_nvenc -b:v 5M out.mp4

3. Mixed transcoding:

ffmpeg -vsync 0 -c:v h264_cuvid -i input.mp4 -vf lvpdnn=model=tmodel.pb:input=input:output=outshape:sample=1:threshold=0.5:log=outlog.txt -c:v h264_nvenc out.mp4

4. ffprobe example:

ffprobe -show_entries frame_tags=lavfi.lvpdnn.text -f lavfi -i "movie=input.mp4,lvpdnn=model=tmodel.pb:input=input:output=outshape:sample=1:threshold=5.0" > lvptext.txt

[j0sh]

Is the blob at 62164c7 a compile time or a run time requirement?

Rather than checking a 25+ MB binary blob into the tree (git does not handle such blobs well), can we make it a temporary requirement to have that blob locally available? Then later on we can explore alternative solutions, perhaps using git-lfs or even downloading a default model from somewhere if necessary.

[oscar-davids]

Not need in compile time. This blob is run time requirement and can't replace it with a temporary.
Maybe we should use git-lfs.

[j0sh]

This blob is run time requirement

If it's runtime, then just make a note that the user needs to be sure to download the blob prior to running the filter. Checking into the ffmpeg tree doesn't seem to be a strict requirement either, since it'd have to be shipped over to the user (LPMS) somehow.

For now, we can store the model in a GCS storage bucket and add a link to that somewhere in the documentation.

[jailuthra]

LGTM after fixing/answering minor comments.

Needs to be rebased on top of latest livepeer branch before merge.

[yondonfu]

Mostly small nit comments and a few clarifying questions.

I think it would be helpful to describe in a comment what the additional steps in the filter that currently involve sws_scale are doing and why they are necessary (as already mentioned in another comment).

[yondonfu]

@oscar-davids I see that 3be1a59 sets the gpu_options.visible_device_list for TF in the set_input_output_tf function of dnn_backend_tf.c. Did the suggestion in this comment of setting the GPU options when the model is loaded via ff_dnn_load_model_tf (which seems to accept an options arg) not work out? If not, could you explain why it didn't work?

[oscar-davids]

As we know there were many changes in the original FFmpeg DNN module in 2~3 months, I reviewed the code and thought that have to update a lot of parts of our codebase now It may cause safety issues.

The method they are using is to create a sess_config(i.e 0x10022805320e09cdccccccccccec3f20012a01303801) with an external python code(tf_sess_config.py) and use it as an argument. Overall, our goal is to use multiple GPUs.
For this purpose, I thought it would be more advantageous for the user to select the GPU id as a filter argument.

[yondonfu]

The go-livepeer install_ffmpeg.sh script currently disables the majority of features and then selectively enables only the features that are currently required. I believe the motivation for this is to avoid unnecessarily increasing the size of the resulting binary by including code for additional features that are not used.

When I tried compiling go-livepeer using the code from this branch, I ran into compilation errors indicating missing references for ff_get_dnn_module. I was able to workaround this compilation error by also enabling the SR filter.

My best guess as to why enabling the SR filter addressed the compilation error is that enabling the SR filter pulled in the required DNN dependencies required by the lvpdnn filter.

In the configure file, _select seems to indicate that something is a "selected dependency". For example, the SR filter has the value of sr_filter_select set to dnn. The derain filter has the value of derain_filter_select set to dnn. Perhaps this means that each filter needs to specify what its selected dependencies are. If this is the case, then we'd want to specify the "dnn" dependency for the lvpdnn filter as well so that if it is enabled on its own it will pull in the required DNN dependencies.

[jailuthra]

LGTM for merge after adding the dnn dependency in configure script as mentioned by Yondon.

[yondonfu]

@oscar-davids Saw that I was tagged for a follow up review, but I can't seem to find the changes relevant to this comment. Could you point to the changes that address that comment?

[oscar-davids]

I think that there is nothing to change in the code because it is ffmpeg build and config options.

[jailuthra]

I think that there is nothing to change in the code because it is ffmpeg build and config options.

We can add a line within this PR

FFmpeg/configure

Line 3483 in b34f2e1

derain_filter_select="dnn"

lvpdnn_filter_select="dnn"

[jailuthra]

Confirmed that after changes in a10074d we no longer need to enable sr-filter in go-livepeer livepeer/go-livepeer@a121c1e 👍

[yondonfu]

LGTM

[jailuthra]

LGTM for merge after -

1. Rebasing over latest livepeer branch (so that unused commit 1da1787 is dropped)
2. Cleaning up the log by squashing temp commits + editing commit log messages to match ffmpeg style guidelines

[jailuthra]

Merged ✔️ 🚢