avfilter: Added text2bitmap subtitle filter #1

tcoza · 2022-06-07T02:53:33Z

Added a text2bitmap subtitle filter which converts text-based subtitle tracks
to bitmap-based subtitle tracks. The filter uses libass to render the subtitles.
It takes as parameters an output height and width, as well as a number of colors
in the output palette as well as sources of fonts. All its arguments are optional.

softworkz

This was from a quick look, I didn't look at the code closely yet.
Thanks

configure

libavfilter/sf_text2bitmap.c

softworkz

Some more comments

libavfilter/sf_text2bitmap.c

configure

libavfilter/sf_text2bitmap.c

tcoza · 2022-06-08T13:28:15Z

Ok, I made the modifications suggested. I forced pushed to the same commit, I'm not sure this is how it works, I just am used to adding to the same commit because that's how ffmpeg wants it.

softworkz

Thanks for the updates. Getting better and better :-)

One missing bit is a signed-off line at the bottom of the commit message.
You don't need to really sign the commits, just add this line,

libavfilter/sf_text2graphicsub.c

doc/filters.texi

softworkz · 2022-06-11T05:08:07Z

I have pushed a new version which:

has all entries in allfilters and makefile in correct alphabetical order
adds snull and strim filters
supports simple filter strings for subtitles from the command line

Once you have rebased and updated, I will apply your code locally and do some testing.
Then we can submit a new patchset version with your filter included.

tcoza · 2022-06-14T03:54:47Z

This rebase is driving me insane. There are about 100 merge conflicts and I have no idea how to solve them. I am just solving at random.

softworkz · 2022-06-14T03:59:54Z

I know what you mean, I've been there.

Shall I do it for you?

softworkz · 2022-06-14T04:01:44Z

But you have a single commit only - that's not even close to the core of rebasing hell... :-)

tcoza · 2022-06-14T04:02:53Z

Sure, if you can. It seems like I should keep the commits from your update, but that seems to be deleting a lot of documentation. I don't understand it.

softworkz · 2022-06-14T04:03:38Z

Just

duplicate your branch to a backup branch
then reset your working branch to latest upstream (my branch) state
then cherry-pick your commit from the backup branch

You'll get a simple conflict resolution scenario then

tcoza · 2022-06-14T04:04:52Z

Cherry-pick? Is that a technical term? What does it mean?

softworkz · 2022-06-14T04:05:21Z

Sure, if you can. It seems like I should keep the commits from your update, but that seems to be deleting a lot of documentation. I don't understand it.

Ah, you merged:

You must not merge, you need to rebase when pulling

tcoza · 2022-06-14T04:09:11Z

I know, but github only does merge, so I merge, and then I just do the rebasing on my computer. Unless there's a way to just do rebase fom Github, but I'm unaware of it.

softworkz · 2022-06-14T04:09:14Z

Cherry-pick? Is that a technical term? What does it mean?

Yes, it's a GIT term, but in fact you just need to rebase instead of merge when pulling.

So, forget my previous instructions. Instead, reset you branch to your original commit.

Then pull with rebase. Which GIT client do you use?

tcoza · 2022-06-14T04:09:41Z

I just use git for linux

softworkz · 2022-06-14T04:19:47Z

I've sent you a rebased version of your path.

Now, you need to reset your working branch to my most recent commit: 251582a

Then you can apply your patch that I just setn on top of that via

git am <xx.patch>

tcoza · 2022-06-14T04:21:56Z

I did the cherry-pick thing, do you think that's ok? It was very easy..

softworkz · 2022-06-14T04:26:06Z

I did the cherry-pick thing, do you think that's ok? It was very easy..

When you had exactly two conflicts (in allfilters and makefile) to solve, then you did it right and you can now force-push your branch from local to your GitHub repo

tcoza · 2022-06-14T04:27:29Z

Okay, yes I just had 2 conflicts. Thanks anyway. How come rebase is so complicated compared to cherry-pick? Isn't it basically the same thing? At least in my case.

softworkz · 2022-06-14T04:27:57Z

For testing, you can still try to apply the patch I sent you with git am. When it says "nothing to commit", it's all ok.

tcoza · 2022-06-14T04:28:26Z

I'll try that.

softworkz · 2022-06-14T04:31:37Z

Okay, yes I just had 2 conflicts. Thanks anyway. How come rebase is so complicated compared to cherry-pick? Isn't it basically the same thing? At least in my case.

rebase is not complicated, it's just git rebase ....

The difference is that with git rebase, it tries to apply your commits on top of the updated base branch, while with merge, it tries to integrate the updated branch with commits that come AFTER your changes.
But you when you do so in this case, you already have my original commits, and now you start applying all my commits again.
That means that the first commit already creates chaos because it's being applied on the end result of my previous completion state.

tcoza · 2022-06-23T16:55:15Z

Ok it's on the repo.
Not sure. I saw some screenshots. Are you talking about SubsTest_2_H264_8bit.mkv?

softworkz · 2022-06-23T16:56:05Z

No, the 1drv.ms link I sent via e-mail

softworkz · 2022-06-23T16:57:03Z

Watch!

tcoza · 2022-06-23T17:03:50Z

Yeah, looks dope.

softworkz · 2022-06-23T17:04:49Z

Done with a single ffmpeg command.
This demo is what I wanted to do for a long time. I just hadn't time to do a t2gs filter..

tcoza · 2022-06-23T17:10:48Z

Wow, that's pretty good. Glad I could be of help. By the way, big thanks for helping me through this, you've been very kind.

softworkz · 2022-06-23T17:29:47Z

I hate like many of the ffmpeg devs behave. It's all based on negativism. The best you can get is silence, which means nobody objects, and below that level, there's a huge bandwidth from justified objections to biased objections - rarely objections are coming with alternate suggestions - to "talking-to-death" behavior and finally even down to personal insults.

It's nothing for the faint-of-heart, but I'm standing strong as well - for and behind the "Subtitle Filtering" addition, which is surely a major one (and it's not coming from one of their core developers, which is an irritating factor for them). This has already gone through about 30 iterations by now, and most objections have been ruled out already.
For the finale now, I am very glad to have you on board which increases the impact, importance and standing of this submission, so thanks for joining and trusting me!

softworkz · 2022-06-24T04:15:49Z

So, for that final issue, I don't have immediate good news.

But first of all - let's put all concerns in terms of making sense or being reasonable aside and see what we get in all possible cases.

[E01_a.MKV] With stripstyles=0

This is probably what you mean to have seen before:

[E01_b.MKV] With stripstyles=1 but not dropping text subject to animation

(I have temporarily added an option of this)

[E01_c.MKV] Normal Operation with stripstyles=1

=> No lyrics visible

softworkz · 2022-06-24T04:41:22Z

Now let's compare the output files from all three cases

Format

Video Streams

All identical information

Subtitle Streams

All identical information as well

but....

Frames Counts

softworkz · 2022-06-24T04:51:54Z

The B-file is larger than the A-file because the bitmaps created are larger.

Bottom Line 1

Neither A nor B is an acceptable result file.

2k subtitle bitmaps for 60k video frames is insane. VLC may play it, but normal DVD or DVB devices won't do it.

Bottom Line 2

If you still want to produce this kind of output, you are better off by totally disabling stripstyles. A combination of stripstyles but keeping the text doesn't make it any better - actually even worse: larger bitmaps + worse appearance

Possible Solution

would be to de-couple input and output of the filter and have a fixed-framerate output query on the filter (like textsub2video does). But for showing animations, you would still need a certain frame rate (maybe 5 fps) and when an animation is ongoing, you will have a different bitmap(-set) every 200ms. And bitmap subs are just not made for this.

So, IMO, it's not worth the effort. And when you want to have it for fun working in VLC, you can still disable stripstyles.

softworkz · 2022-06-24T04:58:02Z

BTW, it will not be like this for all videos. Some have a regular base text for the lyrics and the animations are decorating these on top.

In your specific video, every single letter is a separate ASS event with animation tags - hence there is no easy way to get a useful output.

softworkz · 2022-06-24T05:02:18Z

I noticed a tiny parsing bug which I have fixed and pushed, so you would need to rebase once again and push the complete commit.
I'd be ready to submit it - are you?

Signed-off-by: softworkz <softworkz@hotmail.com>

and provide a compatibility shim for the legacy api Signed-off-by: softworkz <softworkz@hotmail.com>

…itle encoding This commit actually enables subtitle filtering in ffmpeg by sending and receiving subtitle frames to and from a filtergraph. The heartbeat functionality from the previous sub2video implementation is removed and now provided by the 'subfeed' filter. The other part of sub2video functionality is retained by auto-insertion of the new graphicsub2video filter. Justification for changed test refs: - sub2video The previous results were incorrect. The command line for this test specifies -r 5 (5 frames per second), which is now fulfilled by the additional frames in the reference output. Example: The first subtitle time is 499000, the second is 15355000, which means 0.5s and 15.35s with a difference of 14.85s. 15s * 5fps = 75 frames and that's now the exact number of video frames between these two subtitle events. - sub2video_basic The previous results had some incorrect output because multiple frames had the same dts The non-empty content frames are visually identical, the different CRC is due to the different blending algorithm that is being used. - sub2video_time_limited Subtitle frames are emitted to the filter graphs at a 5 fps rate by default. The time limit for this test is 15s * 5fps = 75 frames which matches the count in the new ref. - sub-dvb Running ffprobe -show_frames on the source file shows that there are 7 subtitle frames with 0 rects in the source at the start and 2 at the end. This translates to the 14 and 4 additional entries in the new test results. - filter-overlay-dvdsub-2397 Overlay results have slightly different CRCs due to different blending implementation - sub-scc The first entry is no longer in the output because it is before the actual start time and the strim filter removes such entries now (like for video and audio) Signed-off-by: softworkz <softworkz@hotmail.com>

The previous code expected a segment of type CLUT definition to exist in order to accept a set of segments to be complete. This was an incorrect assumption as the presence of a CLUT segment is not mandatory. (version 1.6.1 of the spec is probably a bit more clear about this than earlier versions: https://www.etsi.org/deliver/etsi_en/ 300700_300799/300743/01.06.01_20/en_300743v010601a.pdf) The flawed condition prevented proper fallback to using the default resolution for the decoding context. Signed-off-by: softworkz <softworkz@hotmail.com>

Added a text2graphicsub subtitle filter which converts text-based subtitle tracks to bitmap-based subtitle tracks. The filter uses libass to render the subtitles. It takes as parameters an output height and width, as well as a number of colors in the output palette as well as sources of fonts. All its arguments are optional. Signed-off-by: tcoza <traian.coza@gmail.com>

tcoza · 2022-06-24T14:11:20Z

I don't care that much about the opening subtitles anyway, so it's fine. I just thought it was weird how it used to work, but not anymore.

Sure, let's submit.

softworkz · 2022-06-25T10:11:05Z

Done!

I'm closing as I've cherry-picked your commit directly from your branch.

davs2_decoder_close doesn't free those on the fly frames which don't get output yet. It's a design bug, but easy to workaround. Before the patch: Direct leak of 1198606 byte(s) in 2 object(s) allocated from: #0 0x563af5e1e5f0 in malloc (ffmpeg+0x6675f0) #1 0x563af9765ef3 in davs2_malloc davs2/source/common/common.h:1240 FFmpeg#2 0x563af9765ef3 in davs2_alloc_picture davs2/source/common/header.cc:815 Indirect leak of 3595818 byte(s) in 6 object(s) allocated from: #0 0x563af5e1e5f0 in malloc (ffmpeg+0x6675f0) #1 0x563af9765ef3 in davs2_malloc davs2/source/common/common.h:1240 FFmpeg#2 0x563af9765ef3 in davs2_alloc_picture davs2/source/common/header.cc:815 Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>

Exmaple command: ffmpeg -h filters=overlay Output: Filter overlay Overlay a video source on top of the input. slice threading supported Inputs: #0: main (video), Formats: Dynamic, Default: [yuv420p, yuvj420p, yuva420p, nv12, nv21] #1: overlay (video), Formats: Dynamic, Default: [yuva420p] Outputs: #0: default (video), Formats: Dynamic, Default: [yuv420p, yuvj420p, yuva420p, nv12, nv21] overlay AVOptions: [...] Signed-off-by: softworkz <softworkz@hotmail.com>

softworkz requested changes Jun 7, 2022

View reviewed changes

libavfilter/sf_text2bitmap.c Outdated Show resolved Hide resolved

libavfilter/sf_text2bitmap.c Outdated Show resolved Hide resolved

libavfilter/sf_text2bitmap.c Outdated Show resolved Hide resolved

softworkz reviewed Jun 7, 2022

View reviewed changes

libavfilter/sf_text2bitmap.c Outdated Show resolved Hide resolved

softworkz reviewed Jun 7, 2022

View reviewed changes

libavfilter/sf_text2bitmap.c Outdated Show resolved Hide resolved

softworkz reviewed Jun 7, 2022

View reviewed changes

libavfilter/sf_text2bitmap.c Outdated Show resolved Hide resolved

configure Outdated Show resolved Hide resolved

libavfilter/sf_text2bitmap.c Outdated Show resolved Hide resolved

tcoza force-pushed the submit_subfiltering branch 2 times, most recently from 3556936 to 4475081 Compare June 8, 2022 13:25

tcoza requested a review from softworkz June 8, 2022 16:59

softworkz force-pushed the submit_subfiltering branch from a66debd to 251582a Compare June 11, 2022 05:00

softworkz requested changes Jun 11, 2022

View reviewed changes

libavfilter/sf_text2graphicsub.c Outdated Show resolved Hide resolved

doc/filters.texi Show resolved Hide resolved

tcoza force-pushed the submit_subfiltering branch from ac3cf5e to cfff09a Compare June 14, 2022 05:26

softworkz force-pushed the submit_subfiltering branch from 3b9550c to 5007eba Compare June 24, 2022 04:56

softworkz added 4 commits June 24, 2022 07:09

avfilter/snull,strim: Add snull and strim filters

959604e

Signed-off-by: softworkz <softworkz@hotmail.com>

avcodec/subtitles: Migrate subtitle encoders to frame-based API

bbdab9f

and provide a compatibility shim for the legacy api Signed-off-by: softworkz <softworkz@hotmail.com>

softworkz force-pushed the submit_subfiltering branch from 5007eba to 5978631 Compare June 24, 2022 05:10

tcoza force-pushed the submit_subfiltering branch from 3043511 to 5186ac0 Compare June 24, 2022 14:07

softworkz force-pushed the submit_subfiltering branch from dd38501 to a90a6e1 Compare June 25, 2022 09:40

softworkz closed this Jun 25, 2022

avfilter: Added text2bitmap subtitle filter #1

avfilter: Added text2bitmap subtitle filter #1

Conversation

tcoza commented Jun 7, 2022

softworkz left a comment

Choose a reason for hiding this comment

softworkz left a comment

Choose a reason for hiding this comment

tcoza commented Jun 8, 2022

softworkz left a comment

Choose a reason for hiding this comment

softworkz commented Jun 11, 2022

tcoza commented Jun 14, 2022

softworkz commented Jun 14, 2022

softworkz commented Jun 14, 2022

tcoza commented Jun 14, 2022

softworkz commented Jun 14, 2022

tcoza commented Jun 14, 2022

softworkz commented Jun 14, 2022

tcoza commented Jun 14, 2022

softworkz commented Jun 14, 2022

tcoza commented Jun 14, 2022

softworkz commented Jun 14, 2022

tcoza commented Jun 14, 2022

softworkz commented Jun 14, 2022

tcoza commented Jun 14, 2022

softworkz commented Jun 14, 2022

tcoza commented Jun 14, 2022

softworkz commented Jun 14, 2022

tcoza commented Jun 23, 2022

softworkz commented Jun 23, 2022

softworkz commented Jun 23, 2022

tcoza commented Jun 23, 2022

softworkz commented Jun 23, 2022

tcoza commented Jun 23, 2022

softworkz commented Jun 23, 2022

softworkz commented Jun 24, 2022 • edited Loading

[E01_a.MKV] With stripstyles=0

[E01_b.MKV] With stripstyles=1 but not dropping text subject to animation

[E01_c.MKV] Normal Operation with stripstyles=1

softworkz commented Jun 24, 2022

Now let's compare the output files from all three cases

Format

Video Streams

Subtitle Streams

Frames Counts

softworkz commented Jun 24, 2022

Bottom Line 1

Bottom Line 2

Possible Solution

softworkz commented Jun 24, 2022

softworkz commented Jun 24, 2022

tcoza commented Jun 24, 2022

softworkz commented Jun 25, 2022

softworkz commented Jun 24, 2022 •

edited

Loading