VSFilter has kerning & ligatures sometimes

[Seeder101]

Hi =D
how are you ?
Recently I have been using MPV/VLC as my daily players, and I downloaded an Episode which has 2 subtitle lines at the same time, one which is supposed to go over the other, to highlight some words. in VSFilter it is correct, but Libass the positioning for the colored word is messed up. Examples:
LIBASS

VSFILTER

as you can see the position in vsfilter is correct and libass is a big-mess =D

[astiob]

@Seeder101 Could you please post the relevant lines from the ASS stream/file? We’ll need the two Dialogue lines that contain the white and the grey text and the corresponding Style line(s). You can use MKVToolNix to extract the ASS stream from the MKV file.

@wm4 I don’t think this is related to the shaper. The shaping seems to be the same; just the positioning is different. I’m guessing the grey line contains just the grey text and has a \pos tag while the white line has \an2 and the exact coordinates of its rightmost glyphs don’t match the \pos.

I’m not sure if these coordinates are supposed to be well-defined at all. It seems like they should be: just add up the glyph advances as specified in the font. The advances don’t depend on the font renderer implementation, so that’s well-defined. Oh wait… hinting? Is this affected by hinting?

Assuming this is supposed to be well-defined, it follows that we somehow screw it up (or VSFilter somehow screws it up in a hopefully predictable manner and we fail to account for that). Could this be a known issue? We have #32 with a similar case where two lines had the same text and appeared slightly out of sync, so to say, but I think (the screenshots are gone) it had one of the lines appear stretched relative to the other, which doesn’t seem to be the case here. We also have #21, which is about ⅛-pixel rounding for each glyph run’s position and size, but the screenshots here seem to show a larger difference of several whole pixels.

[Seeder101]

@astiob hi sir, yes...I can...this is the Dialogue from the ass file

`Dialogue: 0,0:07:16.70,0:07:21.99,Main,,0,0,0,,.نامي- سينباي كانت لصّة تسرق الكنوز من القراصنة
Dialogue: 0,0:07:16.70,0:07:21.99,Main Z,,0,0,0,,{\pos(1024.5,705)}نامي
Dialogue: 0,0:07:16.70,0:07:21.99,Main Z,,0,0,0,,{\pos(900.7,705)}سينباي
Style: Main,One_Dice,85,&H00FFFFFF,&H0000FFFF,&H28000000,&H913B3B3B,0,0,0,0,83,95,0,0,1,2,0.8,2,10,10,15,1
Style: MainZ,One_Dice,85,&H00C0C0C1,&H0000FFFF,&H00000000,&H69000000,0,0,0,0,83,95,0,0,1,1.3,0,2,10,10,15,1

`

Weird thing, sometimes it works fine in some lines
some other lines it is out of place
this is the Dialogue
Dialogue: 0,0:11:09.79,0:11:11.91,Past,,0,0,0,,هل سمعت بالبحر المدعو أول بلو؟
Dialogue: 0,0:11:09.79,0:11:11.91,Past Z,,0,0,0,,{\pos(443.5,705)}أول بلو

this is the Dialogue
Dialogue: 0,0:13:55.98,0:14:00.62,Main,,0,0,0,,.روبين- سينباي وفرانكي- سينباي كانا عدوين في بادئ الأمر
Dialogue: 0,0:13:55.98,0:14:00.62,Main Z,,0,0,0,,{\pos(945.8,705)}سينباي
Dialogue: 0,0:13:55.98,0:14:00.62,Main Z,,0,0,0,,{\pos(652,705)}سينباي
Dialogue: 0,0:13:55.98,0:14:00.62,Main Z,,0,0,0,,{\pos(1080.3,705)}روبين
Dialogue: 0,0:13:55.98,0:14:00.62,Main Z,,0,0,0,,{\pos(796.5,705)}فرانكي

the sync is inconsistent, while it is perfectly fine in VSfilter

[ghost]

Does vlc set any margins?

[Seeder101]

@wm4 same thing in both players, mpv and vlc

[amayra]

can you send SUB file ?
and how about assfiltermod ?

[astiob]

Thanks!

So my guess was right: the grey text has its own event and a \pos.

Huh, that last one looks particularly curious. So there’s multiple word overlays at the same time, and out of the four, one is positioned almost correctly and the others appear as if they’re stretched horizontally centred around the correct one.

I’mma download that file and play around with it for a bit.

Edit: Now that I think about it, this method of using \pos seems to be the only portable (modulo this very issue) way to highlight right-to-left text, as using any override tags in the middle of a right-to-left line will cause libass and VSFilter to render the line differently due to different handling of bidirectional text. Hmm, would it be possible to force VSFilter’s ordering upon libass using some Unicode bidi override characters?

[astiob]

So, at least for the specific file these screenshots are from, adding Kerning: yes to the ASS headers makes libass’ output almost perfectly match VSFilter’s. (Note: VSFilter doesn’t understand the Kerning header; only libass does.)

libass with kerning:

VSFilter:

libass with kerning:

Are we wrong in disabling kerning by default? Or does GDI, like, use kerning for Arabic but not for Latin? Or does it depend on the font?

[grigorig]

GDI has a lot of special case handling, so it wouldn't surprise me...

[astiob]

Okay, so, for starters, while not directly relevant to this issue, it seems GDI keeps a per-process cache of kerning values indexed by some sort of font name. When I removed the kerning data from the font, reattached it to the MKV, closed and reopened the MKV in MPC-HC, the kerning was still present. To see the change, I had to either rename the font or close and relaunch the whole MPC-HC.

Here are my findings so far:

• GDI applies kerning to punctuation regardless of the surrounding script, while HarfBuzz seems to act as if the punctuation has inherited script and only applies kerning defined for that script.

  In this font, “, is kerned in a kern GPOS feature that is listed for latn{dflt} and nothing else. libass with HarfBuzz:
  
  VSFilter:
  

• When Latin text or punctuation is right-to-left (naturally or because of an override character), HarfBuzz applies kerning in visual order, while GDI consistently applies all kerning in logical order. If I understand correctly, GDI is technically right here, as OpenType kerning data is specified in logical order and has no flag limiting its use to a particular writing direction. Who ever thought this was a good idea?!

  In this font, Q, is kerned so that the comma overlaps with the letter, and with HarfBuzz, <rtl override>,Q looks the same as Q, while <rtl override>Q, is not kerned:
  
  
  In VSFilter, <rtl override>,Q is not kerned while <rtl override>Q, looks like ,Q<negative space>:
  
  

• GDI, specifically the TextOut function that VSFilter uses (I hear there’s also a DrawText function that always applies kerning), applies kerning if anywhere in the string there is at least one Arabic character or directional formatting character (even just a left-to-right mark). There are probably other characters that trigger this too. In the context of ASS rendering, remember that VSFilter calls TextOut for each text run and passes it the entire run as a single string, so kerning ends up enabled or disabled for each text run separately depending on whether anywhere within that run there is any trigger character.

• GDI’s GetTextExtentPoint32 function, which VSFilter uses to calculate the width of each text run, never applies kerning to Latin text and punctuation but does apply kerning to Arabic. (This is actually a VSFilter problem; see below.)

  The following two lines are right-aligned, and they’re identical except that the second line contains a left-to-right mark. VSFilter rendering:
  
  Replacing the LTR mark with some Arabic makes no difference. If the line contains only Arabic though, kerning works and there’s no blank space at either end of the line.

[astiob]

By the way, when \fsp is nonzero, VSFilter renders text character-by-character. We have already disabled ligatures in this case in 50c53a1, but actually, this means all contextual and glyph-pair OpenType features should be disabled (except when one character gets substituted by multiple glyphs which are then substituted further, I guess), along with bidi. Yep, everything should be forced left-to-right.

[astiob]

Ah, I’ve figured it out. GetTextExtentPoint32 is fine; the width/display discrepancy is VSFilter’s own fault. I’d forgotten that it computes the width of each “word” separately, where a “word” is either a sequence of non-override-tag/comment characters other than U+0020 SPACE and U+000A LINE FEED (LF) or a single such character (after \n, \N and \h have been replaced by U+000A LINE FEED (LF) and U+00A0 NO-BREAK SPACE). Then it merges consecutive “words” that have the same style, adding together the previously computed widths (some of which may have used kerning and some may have not) and then rasterizing each merged run as a whole (which causes the whole run to have kerning if any of the words did).

[astiob]

Meanwhile, the way TextOutW and GetTextExtentPoint32 decide whether to be stupid or to honour OpenType is by asking Uniscribe’s ScriptIsComplex and (if it says no) checking whether the font contains a glyph for each character in the 4-character string dMr". If the script is complex or if the font doesn’t contain these glyphs, they delegate their job to Uniscribe’s ScriptStringAnalyse and ScriptStringOut functions, which are OpenType-aware. Otherwise, they delegate to the old in-kernel GDI functions, which presumably aren’t. There are actually even more branches in the code, but I don’t understand them and just hope they don’t matter.

Of course, the Uniscribe path probably only exists on Windows 2000 and newer. And ScriptIsComplex learns to recognize new complex scripts in each version of Windows, so ASS files using newly added complex scripts presumably look different on different versions of Windows. But then again, if Uniscribe doesn’t even recognize the script, then it probably looks like garbage anyway, so I think we can safely ignore this and focus on compatibility with the newest versions of Windows and Uniscribe.

[khaledhosny]

So what the conclusion, is there a HarfBuzz/Uniscribe incompatibility here or just VSFilter bugs? cc @behdad.

[grigorig]

It's just GDI being GDI, i.e. piles and piles of hacks on top of hacks. Nothing to see here, move along.

[astiob]

Mostly that as far as ASS is concerned, but there are also a couple HarfBuzz/Uniscribe incompatibilities: see my first and second bullet points in the list above. But I’m not sure which way is right and whether HarfBuzz wants to change its behaviour.

For what it’s worth, I’ve just checked what (my old) OS X does (specifically TextEdit.app), and it seems to agree with Uniscribe on both matters, not with HarfBuzz. But when the base direction is right-to-left, then it doesn’t apply the “, kerning. But then, when I try to edit the document, TextEdit insists on wrapping each new comma that I add inside an LTR embedding, and then it suddenly does apply the kerning once again.

[astiob]

Some test strings for the HarfBuzz/Uniscribe/CoreText discrepancies, in C string format to make the overrides visible:

L",\u201C,\u201C,\u201C,\u201C\u0645\u0631\u062D\u0628\u0627"
L",“,“,“,“مرحبا"
/* seems to be rendered differently by CoreText depending on base direction */

L"\u202A,\u202C\u201C\u202A,\u202C\u201C,\u201C,\u201C\u0645\u0631\u062D\u0628\u0627"
/* the same with some LTR-embedded commas */

L"rtl \u202E|,Q\u202C  Q,| ltr"

L"rtl \u202E|Q,\u202C  ,Q| ltr"

[astiob]

Oh, you also need the font. I couldn’t find any licensing information, so here’s the file.

[grigorig]

Mostly that as far as ASS is concerned, but there are also a couple HarfBuzz/Uniscribe incompatibilities: see my first and second bullet points in the list above. But I’m not sure which way is right and whether HarfBuzz wants to change its behaviour.

I don't think these are HarfBuzz issues. HarfBuzz is concerned with shaping runs of a single script and single direction at a time. Text with mixed scripts or direction is handled by some layout system at a higher level. So whether and how we want to e.g. apply kerning at script run boundaries is up to us.

[astiob]

Oh, right!

That probably explains why libass doesn’t kern “,<arabic> (my first bullet point). I see now in ass_shaper_determine_script that we treat HB_SCRIPT_COMMON as HB_SCRIPT_INHERITED. It seems this is wrong, and features (including kerning) designated for all scripts should be applied to common characters (right?), whereas with the way we do it, we restrict common characters to the script of the surrounding text. How can we fix this?

The other issue still remains though, where libass-with-HarfBuzz matches kerning pairs in visual order while Uniscribe and CoreText both match them in logical order. Is this intentional? (It sure seems sensible, but it seems to contradict the OpenType spec—as much as I haven’t read the actual spec, just some documentation on the Web.) Or is this possibly an inevitable unfortunate consequence of HarfBuzz’s API? Does HarfBuzz get text from us in logical order or does it get it in display order and have to guess as to what the logical order might have been? We tell HarfBuzz the writing direction anyway, so it can uniquely determine the logical order.

[astiob]

For reference, I played my sample file in VSFilter in Windows XP out of curiosity, and: the Arabic text is kerned, BWB is kerned, but neither Q, nor “, is kerned. I don’t know why.

[realfinder]

So, at least for the specific file these screenshots are from, adding Kerning: yes to the ASS headers makes libass’ output almost perfectly match VSFilter’s. (Note: VSFilter doesn’t understand the Kerning header; only libass does.)

is there are a way to do Kerning: yes for some ssa4+/ass specific styles/style not all them? or better by naming the fonts

[TheOneric]

is there are a way to do Kerning: yes for some ssa4+/ass specific styles/style not all them?

No. The Kerning header applies to the whole file.

[realfinder]

let say that 1 font need Kerning in libass but another font don't need it in order to make the subtitle work same on both VSFilter and libass, the Current Kerning option in this case will fix one font compatibility for libass but will case another! or there will be another new solution later?

[astiob]

Apparently, Uniscribe doesn’t apply TrueType kern tables—only OpenType GPOS tables. Or at least in some cases doesn’t apply them. See #706 (comment) and the font in that issue’s initial post.

(It’s not clear to me when Windows applies TrueType kern at all, then. Microsoft clearly says it does, and FontForge has options for it. But if GDI doesn’t do it and Uniscribe doesn’t do it, then what does? Maybe GDI has a separate option toggle with which an application can enable kerning? But then why does it allow Uniscribe to apply kerning regardless of the toggle? Weeird.)

[astiob]

GDI applies kerning to punctuation regardless of the surrounding script, while HarfBuzz seems to act as if the punctuation has inherited script and only applies kerning defined for that script.

In this font, “, is kerned in a kern GPOS feature that is listed for latn{dflt} and nothing else. libass with HarfBuzz:

VSFilter:

And now, on Windows 11 23H2, with a different font, I’m seeing it apply kerning to T', 's, ’' and ’’ but not to T’ or ’s (the font’s original kerning pair; the rest were added for testing). It seems as though it treats U+2019 RIGHT SINGLE QUOTATION MARK ’ as a distinct non-Latin script and splits the shaping runs around it 🤯 but treats U+0027 APOSTROPHE ' as an inherited-script code point (despite UCD tagging both as Common). The font has a single kern GPOS lookup tagged DFLT{dflt} and latn{dflt}.

Good time as any to repeat that I’m baffled by how OpenType kerning (and in general all GPOS/GSUB features, but especially kerning) is seemingly forbidden from straddling script/language boundaries. Either that or OpenType implementation APIs (including HarfBuzz) are seemingly at fault for not allowing it.