ASS/SSA Format: Introduce LayoutRes{X,Y} script headers to allow easy multi-resolution releases

[TheOneric]

@Cyberbeing, @pinterf, @clsid2:
We would like to propose a retroactive addition to the ASS/SSA format, but in order to not fracture ASS even more we will only move forward with this if this extension will also be accepted and implemented in (most) active VSFilters.
This retroactive addition should be able to provide tangible benefits, while avoiding undue issues.

Some background first:
Rendering of the ASS/SSA format depends on the video’s storage resolution for several effects. This means subtitle releases are bound to a specific video (storage) resolution and in general cannot be reused for e.g. later BD-source releases of a higher resolution or in case of DVD-sources, different encodes which did/didn't undo the anamorphic stretching. (Simple, sparingly styled subs may happen to avoid this by not using the affected effects, but this isn't possible for more complex subs.) Affected tags are \be, \blur, \frx, \frz and if ScaledBorderAndShadow isn't set to yes also all border and shadow tags.

Thus we propose adding two new headers LayoutResX and LayoutResY, which if both present and both set to legal (> 0) values will be used in all places instead of the storage dimensions of the video file.
This way, subtitle authors are able to create subs once and then reuse the very same file for future release with arbitrary video resolutions as long as the display aspect ratio matches the original (and timing + colours didn't change, but we can't do much about that and they are easier to semi-automatically adjust than the scaling-affected tags).
If the headers are not or only partially present or set to illegal values, rendering continues to work exactly as it always did before.

It seems fairly unlikely any existing subs put non-functional LayoutRes{X,Y} headers into their [Script Info] section, so this change will not break existing subtitle files. By encouraging authors to set the layout resolution equal to the actual video storage resolution, the initial/main subtitle releases would also continue to work in older renderers, giving users some time to update their renderers instead of suddenly be confronted with bizarre rendering results.
If this extensions is accepted, the plan is to also get common Aegisub forks to implement support for it and automatically initialise the headers as the actual storage resolution (and possibly, for the beginning at least, nudge users to keep it in sync with the actual resolution as currently happens for PlayRes{X,Y})

---

@Cyberbeing, @pinterf:
I wanted to draft up a patch implementing the headers for xy-VSFilter, but unfortunately I don't have access to any Windows dev setup and no familiarity with VS, so I can only offer this completely untested patch:
see here, or xy-VSFilter_layout-res_patches.tar.gz.
The patch is on top of Cyberbeing’s xy_sub_filter_rc5 branch, specifically Cyberbeing/xy-VSFilter@fc01a8d though it likely applies to pinterf/xy-VSFilter as well after the incompatible scaling changes are reverted (see: pinterf/xy-VSFilter#34).
I tried getting it to build in GitHub Actions, but this didn't work out either with the build failing to locate yasm no matter where I put yasm.exe. I'm willing to test and work on this patch, but would require assistance in getting a working build setup. Certainly welcome and ideal though, would be someone more familiar with both Microsoft VS and xy-VSFilter taking over the patches.

@clsid2:
For MPC-HC’s Internal Subtitle Renderer the situation is actually a bit different from original guliverkli(2)-VSFilter, xy-VSFilter and libass. MPC-HC’s ISR is already incompatible with many existing files and other renderers for a long time (in our experience, this resulted in barely any complex subs targeting ISR). Among others, MPC-HC's ISR completely stopped considering the original video resolution during rendering. While this deviation has been around long enough for ISR to probably better keep it when no LayoutRes{X,Y} was set, it would be nice if the headers could be taken into account when they are present to reduce the incompatibilities to other softsub renderers.

---

(Replaces #597 to get a new, backlog-free, discussion now that more people are involved and the previous points re. LayoutRes{X,Y} have all been decided.)

[Masaiki]

I tried getting it to build in GitHub Actions, but this didn't work out either with the build failing to locate yasm no matter where I put yasm.exe. I'm willing to test and work on this patch, but would require assistance in getting a working build setup.

@TheOneric You can refer to Masaiki/xy-VSFilter@30afd3b to write GHA workflow, the key point is VSYASM and ExecutablePath

[clsid2]

Why not just add a function to Aegisub to adjust an existing sub for a new video? Just input the old and new res and let it do its magic to rescale the values in the script.

If ISR is doing things wrong you need to open new tickets on my issue tracker for each individual issue.

And a proper specification of how PlayResX/Y should behave when it differs from the stored/displayed video size would be useful as well. With practival examples. Same for LayoutRes.

[TheOneric]

You can refer to Masaiki/xy-VSFilter@30afd3b to write GHA workflow, the key point is VSYASM and ExecutablePath

Thanks! I tried VSYASM before to no avail, but copying the ExecutablePath bits from your project files now allowed everythin to work after I fixed some small errors in the patch (links in OP updated).
I can confirm the xy-VSF patch has the intended effect when used via VSFilter, I’d assume the same holds true for the xy-SubFilter interface, but am unable to test as MPC-HC’s rendering modes compatible with xy-SubFilter don't work with the emulated GPU of my VM. If someone wants to test, here are the binaries as a GHA build artefact: https://github.com/TheOneric/xy-VSFilter/actions/runs/2941169396 (updated link after refactoring the patch)

[TheOneric]

Sorry for the delay, I didn't manage to finish this yesterday.

Why not just add a function to Aegisub to adjust an existing sub for a new video? Just input the old and new res and let it do its magic to rescale the values in the script.

This would have the advantage of not requiring format additions, but means it will still not be possible to create self-contained ASS subs and still require author interaction and possible imprecision after one or multiple conversions.If I'm not mixing things up, writing such a script also isn't straightforward and no complete script (yet) exists.

One place where ASS not being self-contained is an issue is e.g. streaming subs and videos, where the video stream may be downscaled according to available bandwidth. The received video storage size may not match the storage size the subs were authored with, so a naïve approach would result in incorrect rendering. It's possible (with libass and xy-SubFilter at least) to signal the original storage resolution via an additional channel, but that requires special handling not only on the server- but also client-side. With self-contained ASS regular streaming clients can be used (and only the server needs to e.g. on the fly insert appropriate LayoutRes* headers if none are already present).

So using new headers, avoids precision problems, requires no author-interaction, editors can even automatically add headers if none yet exists and allows for self-contained ASS subs simplifying other things. Since I believe issues from the addition to be minor to barely noticeable, I think these advantage justify the retroactive addition.

And a proper specification of how PlayResX/Y should behave when it differs from the stored/displayed video size would be useful as well. With practival examples. Same for LayoutRes.

Sure, if you decide to implement compatible scaling, we can work out the transform matrices etc. Tbh, since it started quite a while ago, I couldn't give you the exact transformations otoh right now.

To give some background and a brief overview though, there are three resolutions relevant to rendering conforming ASS:

1. PlayRes{X,Y} determined by the sub author and included in the sub file itself.
2. Rendering resolution, determined by the player (possibly based on viewer input). It is always relayed to renderers by necessity.
3. The video’s storage resolution, determined by the video the subs are paired with. Note, storage resolution is the resolution the video is encoded in (sans codec-level trims), which is different from the video’s native display size for anamorphic content.

Note, PlayRes{X,Y} has no inherent connection to any other size. Often but not always it is an isotropic scaling of either the video’s display or storage resolution, but that's not reliable.

For classic guliverkli(2)-VSFilter and xy-VSFilter, rendering resolution is always equal to the video’s storage resolution, as they are only capable of rendering directly onto the directly decoded video frame, effectively reducing it to only two resolutions.
However for historic reasons, rendering does indeed depend on the exact values of both, not just one resolution + a aspect ratio rate. PlayRes* gives a coordinate grid for placement and its height determines font-scaling, but effects are partially (without ScaledBorderAndShadow mostly) calculated in the storage resolution space.

xy-SubFilter and libass can render at arbitrary resolutions and because the exact storage resolution is required to keep rendering content the same, they receive the exact storage resolution from API-users.
I believe MPC-HC (and -BE?)'s ISR can also render at arbitrary resolutions, but afaik do not consider the storage resolution, contributing to incompatibilities. (Iirc, there were already several other incompatibilities in ISR since at least 2010 iinm.)

Rendering resolution usually is an isotropic scaling of the video’s native display resolution. Any choice of rendering resolution must result in an image visually quasi-identical to what the output would have been when rendering at storage resolution, just scaled accordingly.

A valid pair of LayoutRes{X,Y} headers would simply replace every instance of the video’s storage resolution making ASS subs with LayoutRes* (and YCbCr Matrix: None) self-contained, independent from external properties.
If no valid LayoutRes{X,Y} pair is present, the video’s storage resolution continues to be used.

---

Here are some example files: layoutres_demo_01.tar.gz
The original_* samples do not have LayoutRes* headers and will render correctly in xy-(VS|Sub)Filter, guliverkli(2)-VSFilter and libass, but probably not MPC-HC’s ISR, showing the current difference. There are also screenshots showing the correct rendering.
The reused+LayoutRes_* samples correspond to a original_* sample, but the video is swapped out for a different resolution while the subs are identical excpet for the addition of LayoutRes* headers. They'll render correctly in patched renderers and incorrectly in any of the current renderers.

[astiob]

If I'm not mixing things up, writing such a script also isn't straightforward and no complete script (yet) exists.

Indeed. No such tool exists yet, and when I worked on one for my personal use some years ago, it involved matrix decompositions, error-prone case-by-case logic, and the output required ugly ASS markup to achieve arbitrary horizontal offsets (which even seemed impossible at the time due to glitches in VSFilter’s rasterizer, but we’ve [re]discovered the n command since then, which should do the trick). And I don’t remember if I ever got to the point of handling all possible input cases.

(That’s for \frx and \fry. Rescaling \blur, shadows and borders is trivial. Meanwhile, rescaling \be in the source is literally impossible in the general case, although cases where it matters are rare or nonexistent. Indeed, while it’s decently possible during rendering, no current renderer does it that well.)

---

I may add that for some years already, some people have been making multi-resolution video releases with the exact same subtitle track attached, even though they would look bad in the most popular subtitle renderers, precisely because it would take effort to manually adjust the subtitles. Others do it because their entire workflow is automated (and no automatic rescaler exists). Ironically, some of those files look best in MPC-HC’s current internal subtitle renderer and in older VLC, which ignore(d) video storage resolution, and in fact some files accidentally produce heavy lag in the other renderers.

So it might seem that making all renderers follow suit and ignore video resolution could be better. However, other files do rely on this video resolution dependence. (Indeed, that’s why libass ever started emulating it in the first place.) And among those multi-resolution releases I mentioned, I hear a big chunk actually need LayoutRes different from PlayRes to render properly (due to sloppy authoring); that is, they don’t look right in any of the current renderers, but they could be trivially fixed by adding LayoutRes headers.

Thus, LayoutRes is a compromise tool that would make it very easy both to author new files and to fix/reuse old files of all kinds.

[TheOneric]

Ironically, some of those files look best in MPC-HC’s current internal subtitle renderer and in older VLC, which ignore(d) video storage resolution

or mpv with the non-default --sub-ass-vsfilter-blur-compat=no setting to withhold storage resolution info from libass (which breaks other, more responsibly authored files).

[clsid2]

Can you give a link to the ASS/SSA spec? The documents I find are not very detailed.

Do I understand correctly that (for existing files) scaling should happen based on PlayRes vs. StorageRes, ignoring the anamorphic stretch of video (if any)? To be clean, I mean for rendering a sub that will be overlayed on top of the stretched video, not before stretching.

And that you want to add LayoutRes to override StorageRes, so that same script will work no matter if the video is encoded anamorpic or not? Do encoders still use anamorphic in this day and age?

Simple example: suppose that PlayRes is 360p, LayoutRes is 720, and I am watching/rendering at 1440p. Does everything need to be scaled by 4x, or are some there things that need to be scaled 2x?

[astiob]

Can you give a link to the ASS/SSA spec? The documents I find are not very detailed.

There isn’t one. The documents you’ve found are most likely the same ones that we have. We just have to emulate what older (more well-established) renderers do in practice.

---

This isn’t just about anamorphic video, although that is certainly included.

1. PlayRes to StorageRes:

   • Most things are scaled from PlayRes to StorageRes: horizontal measures from PlayResX to StorageResX, vertical measures from PlayResY to StorageResY.

   • Border and shadow are scaled in the same manner, but only if the ScaledBorderAndShadow header is present and set to a true value.

   • Font size (\fs) is scaled from PlayResY to StorageResY. (This is the pain point for anamorphic video in particular, as this scales font glyphs along both axes.)

   • \be, \blur and the 3D distance to the screen plane (relevant for the perspective transform involved in \frx and \fry) are not scaled from PlayRes at all. (This is the bigger pain point that truly prompts the introduction of LayoutRes.)

2. StorageRes to final window/fullscreen resolution: everything is stretched as though rasterized at StorageRes and rescaled as a bitmap. This includes any anamorphic stretching (in which case the scale factors are different along the two axes) as well as full-screen playback or simply a resized player window.

LayoutRes would replace StorageRes in these calculations.

Simple example: suppose that PlayRes is 360p, LayoutRes is 720, and I am watching/rendering at 1440p. Does everything need to be scaled by 4x, or are some there things that need to be scaled 2x?

Assuming for the sake of the simple example that there’s no anamorphic stretching and ScaledBorderAndShadow is enabled, most things would be scaled 4x, but blur and the screen plane distance would be scaled only 2x.

[TheOneric]

EDIT: Disregard this, astiob already tested XySubFilter. Sorry for the ping.
pinterf: I assume you have a working xySubFilter setup. Could you perhaps, if it doesn't bother you, verify that my xy-* patch works when the xySubFIlter interface is used? Unfortunately I'm only able to test the xy-VSFilter interface myself (for which everything works as expected). I don't really expect any difference, but let's make sure.

Prebuilt DLLs can be found here as a GHA artificat: https://github.com/TheOneric/xy-VSFilter/actions/runs/2941169396 or if you prefer you can use the source code to build it yourself.
To test simply download layoutres_demo_01.tar.gz and check if the reused+LayoutRes_from1080pToAnamorphic.mkv matches the original_1080p__correct.png screenshot in everything but background colour. And the same for reused+LayoutRes_fromAnamorphic576pTo1980p.mkv and original_s720x576_d1024x576.mkv.

[astiob]

Oh, I have a free minute and with a prebuilt DLL I can test.

OK, yep, that works:

(Just to be sure I also verified that stock XySubFilter and MPC-HC ISR fail this test.)

Nice work!

[clsid2]

Suppose LayoutRes is 1920x1080 bundled with original 1920x1080 video. This script is then bundled with a version of the video where black bars are cropped away, giving for example 1920x800 video. Is that an allowed scenario? And if so, how should it be handled? If done as astiob says above it would mean a simple stretch of the rendered bitmap (LayoutRes -> DisplayRes), which would obviously give wrong AR in this scenario.

[TheOneric]

If the display aspect ratio changes, it won't work and there is no general foolproof way this could be made to work. That's why the original post said (emphasis added):

create subs once and then reuse the very same file for future release with arbitrary video resolutions as long as the display aspect ratio matches the original (and timing + colours didn't change, but we can't do much about that

What using LayoutRes also for PAR handling allows however, is to reuse scripts across different storage ratios and resolutions as long as the display aspect ratio stays the same. Consider e.g. subs originally typeset against a 16:9 PAL DVD release. DVDs must use anamorphic for 16:9 and in this case the next-best allowed storage resolution would be 720×576 which will be stretched to 1024×576 for display. Now LayoutRes allows the subs to be ported easily to a non-anamorphic reëncode or a later BluRay release. The demofiles I linked earlier have examples for reuse from this anamorphic PAL DVD resolution to non-anamorphic 1920×1080 and vice versa.

[astiob]

If you crop part of the video, you get a different video content. For a different video, you need different/adjusted subtitles. Put simply: when cropping the video, you must also crop the subtitles. There’s no way for a renderer to know that the video has specifically had black bars cropped (not to mention of some particular size). And the original subtitles on the original video may have had items positioned within those black bars; again there wouldn’t be any universally correct way for a renderer to decide what to do with them: a human must decide and edit the subtitles.

This is the same regardless of LayoutRes, anyway.

[clsid2]

That is what I assumed. I just wanted to get it clarified so that it gets explicitly mentioned in the specification of the new header.

[clsid2]

Hi guys, I am working on LayoutRes implementation for MPC-HC ISR.

Can you also make a conformance checking sample that uses LayoutRes, but without anamorpic stuff? And one with ScaledBorderAndShadow off. Thanks.

[clsid2]

I got the two 1080p samples rendering correctly already.

The other two almost:

Text is rotated too much. It is related to the PAR compensation. But I am struggling to fix it.

[astiob]

Just a note: besides rotations, this is also known to affect

• Gaussian \blur, which MPC-HC’s internal renderer currently never scales at all,
• (and \be the non-Gaussian blur, but do feel free to disregard that at the moment, as it can’t be done by simply changing the parameter value and no renderer does it correctly yet),
• ScaledBorderAndShadow: no (as you’ve already seen),
• and the Banner and Scroll effects, mainly the delay parameter: when it’s rounded in CRenderedTextSubtitle::ParseEffect, this rounding should happen in the layout-resolution space, and the scale from layout to final rendering resolution should happen in floating point here or later, as any difference in the rounding can add up quickly to significant deviations visible to the naked eye. Not only does it look different from other renderers, but it also causes the subtitles to jump across the screen in MPC-HC’s internal renderer itself when the user just changes the window size.

[clsid2]

Some good simple demo files for those issues would be greatly appreciated.

[TheOneric]

Some good simple demo files for those issues would be greatly appreciated.

Sorry for the delay, I currently don’t have much time.
For Gaussian \blur I have a sample and images showing the unrelated non-scaling issue in MPC-HC 1.9.23 ISR. (I intended to file a report about it, but haven’t got to it). See: mpchc_layoutres_samples_blur.tar.gz
The subs don’t use LayoutRes*, but there are two 16:9 video files, 1080p and 360p, and screenshots for either. Without LayoutRes \blur30 in 1080p should look visually identical to blur10 in 360p. If you add LayoutRes* headers it should always look the same and match the LayoutRes-less rendering for a video of the same resolution.

For delay you might find the samples from here: #645 (comment) useful.
Note: libass needs to treat delay parameters smaller than 1 * StorageRes / PlayRes differently to emulate VSFilter’s output. If there were significant changes to MPC-HC’s Effect code since guliverkli, this might become relevant too, but if not this effect will happen automatically. See this commit for an explanation of the difference and also an example for when the general rounding error by int truncation becomes noticeable.

For ScaledBorderAndShadow, just take any subs with a noticeable thick border and see how the border shrinks going from 360p to 1080p without LayoutRes* and should stay the same with LayoutRes*. Same for a large shadow offset.

[astiob]

Note: delay parameters smaller than 1 * StorageRes / PlayRes are treated differently from larger values

Not in VSFilter. Or at least, I think this wording is more confusing than useful for VSFilter developers. We’re emulating code that VSFilter already has; it just needs to keep that same code but use a different scaling factor, moving the remaining scaling to a later time.

But this reminds me that the lround here (for Scroll) and above it (for Banner) need to be turned back into (int).

[TheOneric]

Not in VSFilter. Or at least, I think this wording is more confusing than useful for VSFilter developers.

I wasn’t sure if MPC-HC ISR still kept this behaviour, since other Effect differences were mentioned. But indeed the wording wasn't great and potentially confusing. Updated my previous post.