wishlist: vtt: support styling

[pan93412]

mpv version and platform

mpv 0.30.0 Copyright © 2000-2019 mpv/MPlayer/mplayer2 projects
 built on Tue Nov 12 00:20:16 CET 2019
ffmpeg library versions:
   libavutil       56.31.100
   libavcodec      58.54.100
   libavformat     58.29.100
   libswscale      5.5.100
   libavfilter     7.57.100
   libswresample   3.5.100
ffmpeg version: n4.2.1

• System: Arch Linux (testing / kde-unstable) x86-64
• System Language: zh_TW.UTF-8
• Desktop Environment: Plasma 5.17.3 / KDE Frameworks 5.64.0 / Qt 5.14
• Font: Noto Sans CJK TC
• youtube-dl installed

Reproduction steps

1. open terminal (in my case is konsole)
2. mpv https://youtu.be/S8dmq5YIUoc
3. switch the subtitle to zh_TW

Expected behavior

When playing this video with zh_TW subtitle on YouTube Web. (0:14)

Actual behavior

When playing this video with zh_TW subtitle on mpv + youtube-dl. (0:14)

(For developers) some reference of styling

• Text Color: https://www.w3.org/TR/webvtt1/#default-text-color
• Cue settings, like alignment, position and etc.: https://www.w3.org/TR/webvtt1/#cue-settings

Log file

Omitted.

Sample files

• The zh_TW subtitle of that video

[jeeb]

Now, I haven't looked at the sample yet, but I do know that we have two possible parts here:

1. I know some youtube VTT things do things in a non-standard way.
2. FFmpeg handles the WebVTT parsing and converts it to ASS internally.

---

1. Is unfortunate, but the last time someone brought up stuff like karaoke etc that didn't seem to work in a standard way.
2. If the VTT file seems standard, you can check with ffmpeg.c (the FFmpeg command line app) whether doing something like

ffmpeg -v verbose -i INPUT.vtt -c:s ass out.ass

has the styling.

If not, then this is an FFmpeg issue.

If it has, then mpv is dropping the styles. I think there was some argument before regarding mpv dropping the generated styles that FFmpeg provides.

[ghost]

It drops the ASS style section. Not doing so in unacceptable, because for most formats, the ffmpeg default style is just arbitrary, and it makes no sense to force that on users.

[pan93412]

ffmpeg -v verbose -i INPUT.vtt -c:s ass out.ass

$ mpv --sub-file=out.ass AIAIAI.mkv # the filename has been shorted

So looks like ffmpeg lost those styles...

[jeeb]

Alright, feel free to create a ticket on https://trac.ffmpeg.org/ regarding it.

[pan93412]

Ok :) I'll do it later

[Akemi]

let us know when you reported it, so i can close this issue.

[pan93412]

@Akemi @jeeb @wm4 Ok I reported this to upstream: https://trac.ffmpeg.org/ticket/8410#ticket

Could you guys take a look? thanks! :)

[Akemi]

closed since it was reported upstream.

[Slider-Whistle]

Hope they can do it, ASS is just such a terrible, unreadable subtitle format.

[ghost]

Hope they can do it, ASS is just such a terrible, unreadable subtitle format.

What does that help us? Besides webvtt is more terrible than ASS.

Anyway, if anyone wants to implement a webvtt parser/renderer in mpv, you're welcome.

[Totorrr]

Hi everyone,

vlc has vtt files displaying with styling working. Can mpv copy/integrate vlc's parser/renderer in mpv's code?

Thanks,

[sergeevabc]

@ghost, could you elaborate why you believe VTT is a terrible format?
Compared to SRT, it looks cleaner to me and surely takes up less space.

@Totorrr, came here to say the same after a few attempts to output colored text.
Actually, for the first time in my memory it seems that VLC outperforms MPV.

[pan93412]

@ghost, could you elaborate why you believe VTT is a terrible format?
Compared to SRT, it looks cleaner to me and surely takes up less space.

@Totorrr, came here to say the same after a few attempts to output colored text.
Actually, for the first time in my memory it seems that VLC outperforms MPV.

@ghost = deleted users. Tagging it takes no effect.

[CounterPillow]

VTT is a terrible format because it requires a full browser engine to render properly. mpv will not ship a full web browser to render subtitles.

[Slider-Whistle]

It's just CSS and a bit of XML, isn't it? Little details aside, what makes that so different from the ASS subtitle format? An XML interpreter is hardly a "full web browser".

[CounterPillow]

It's a full DOM, with JS scriptability, and a whole CSS layout engine. (EDIT: after a quick glance at the W3C spec, the JS or CSS of the HTML page the <media> tag is found in can affect the WebVTT styling as well, which is just grand, because mpv does not have this information at all. There's of course WebVTT specific CSS pseudo-selectors, which do insane things like "style all cues temporally before/after this cue", not to forget all the other fun basic CSS stuff like "make the background of this an image fetched from this URL" (which means implementing cross-origin stuff I guess? Aside from some way for a subtitle renderer to make HTTP requests) and I bet it even supports CSS animations and transitions)

ASS, on the other hand, is basically just timed vectors with some transforms.

FYI, VLC did pull in a full CSS layout engine to get even partial support for styled WebVTT from what I recall, and if I remember their statement at VDD about this they were less than happy about having to do this. WebVTT is a format specifically written to make life easier for web browsers, and incidentally will make life harder for everyone who is not yet a web browser.

[layercak3]

A workaround for YouTube subtitles (which is SRV3/YTT, I think it's non-standard WebVTT?) is to use YTSubConverter to convert it to ass which can be played in mpv. It should largely resemble the subtitles you'll see in the YouTube player.

[Nicryc]

So if I understand correctly WebVTT colouring will never be supported by MPV because it needs too much to work (a complete web browser engine)

[dhouck]

The W3C spec says

For the purpose of resolving URLs in STYLE blocks of a WebVTT file, or any URLs in resources referenced from STYLE blocks of a WebVTT file, if the URL’s scheme is not "data", then the user agent must act as if the URL failed to resolve.

So, thatʼs still an almost-full CSS renderer and most of the problems that come with it, and I understand not wanting to include that, but no HTTP requests are necessary, and none of the security or blocking problems associated with those.

That can still be a lot of extra effort, of course, but not nearly as bad as implied.

[Disonantemus]

Maybe a limited WebVTT spec support? no full karaoke, just basic tags like: color/line/position, that it's already there, because ass. Can be a 5 line bash script, just to convert vtt2ass, retaining that.

[dhouck]

I think that would take more than 5 lines of bash, and I am in no way a maintainer or anything, but I bet if you did write that script people would be willing to use it.

[Tama47]

A workaround for YouTube subtitles (which is SRV3/YTT, I think it's non-standard WebVTT?) is to use YTSubConverter to convert it to ass which can be played in mpv. It should largely resemble the subtitles you'll see in the YouTube player.

How to convert VTT to ASS? From their description:

YTSubConverter is primarily an .ass → .ytt converter. You can create .ass subtitles using Aegisub.

[Bobo1239]

@Tama47 See the Reverse conversion section of the README.

[MasterInQuestion]

    Too many things wrong from the very beginning.
    The concept of the whole presentation scheme is initially mal-defined:
    Scenes (slicing) and overlays (filtering) are important elements of raw media.
    Not only those straight captured (e.g. directly from camera).

    Somewhat related:
    https://github.com/MasterInQuestion/talk/discussions/3#discussioncomment-9376594
    https://trac.ffmpeg.org/ticket/5309#comment:16
    https://trac.ffmpeg.org/ticket/11055#comment:201
    (more complete review to be done)