MediaPlayerElement/MediaElement: Closed Caption and Subtitle Styling got Broken by Recent Update

[softworkz]

This is a tough one, because it spans wide and it's interdisciplinary - a kind of RACE condition between multiple

Teams/Areas Involved

• WinUI2, WinUI3 Media Elements
  Microsoft.UI.Xaml
• Windows.Media
  Windows.Media.ClosedCaptioning.ClosedCaptionProperties
• Chrome/Windows and Edge/Windows
• Windows Media Player
• Windows Settings
  Accessibility Section
• New Live Captions Feature
  (probably originated by)

I'm not sure whether this is the best place for this, but for Windows.Media and UWP, there's a cluttered forum only and over at MS.UI.Xaml, an earlier report has been ignored since (precisely) one year. The App Sdk seems to have the widest scope and - lastly - has the most welcoming text 😄
Feel free to move it, though.

Symptoms

WinUI3: Subtitles are not visible

There's no way for an application to change that through the media apis.

WinUI2: Subtitle Styles cannot be controlled

Certain properties of TimedTextStyle, most importantly the FontFamily.

Impact

This is showstopping for us. We are migrating a media player application to UWP (xbox) and WinUI3 and are blocked by this.
I guess the "invisible" subtitles effect doesn't need further explanation, and regarding the font setting, one might easily think that this is a minor problem - but it's not like that:

• Videos often use different fonts and styles for different situations, which aren't shown due to the current situation
• Videos often have special fonts embedded, especially in Asian titles where subtitles have even become a kind of an art form
  Not being able to properly display them won't be acceptable for many users
• We have our own settings for overriding subtitle styling and this can't be driven from two different places with conflicting settings
• Many subtitles have their own rich styling, which would never become visible when overriding settings exist which cannot be deactivated

But it's not just about us. It probably affects Millions of systems and not only custom WinUI apps - it also includes Media Player and Chrome + Edge browsers.

It's not invisible on the latter, but they're still affected. To understand the issue, we need to step back a little first.

Background

After intensive research, I have come to the following picture of what has happened:

Attempting to Reconstruct

In 2015, a new class was introduced in the Windows.Media namespace: ClosedCaptionProperties.
The static readonly properties are corresponding to these values in the registry:

(also mirrored under HKCU)

The purpose of this is obviously a system-wide (or rather "user-session-wide") override of caption/subtitle styles to provide an adjustable experience which fits everybody's needs. A very good thing without doubt!

The controls on Windows 10 looked like this:

Pattern: Default + Optional Override

Here we get to the core of the subject:

• When looking at the settings, all are set to default
• Looking at the registry values, all are set to zero
• All the enums of the properties are starting with a "Default" entry valued zero

The pattern is clear and makes sense: Those settings do nothing until the user deliberately changes a value.

Implementations followed

Probably there are many more cases where implementations were done to make the presentation of captions adhere to those settings. These are the ones I know so far:

• Windows Media Player
  (the new one art least)
• WinUI2 MediaElement
• WinUI3 MediaPlayerElement
  (in CueStyler.cpp)
• Chrome and Edge Browsers
  (caption_style_win.cc)

In all cases this has been implemented consequently and without further ado: When an override is set, then it will be applied.
Again, this makes sense, because nothing happens until a user makes such a setting change. And when he does that, he should be able to rely on getting the result as configured.

The Fatal Mistake

I can only speculate about why it happened - but at least it's quite plausible that during the process of bringing those captions settings into the new Win11 settings UI, someone (or multiple) started to wonder and doubt about all the settings having the value "Default" - which raises the question "what does 'default' even mean'?"
Clearly, such things as audio transcriptions or live captions do not have any default text style - but seemingly it wasn't considered or known that subtitles can very well have their own ("default") styles and even their own embedded fonts.
Even ATSC captions (EIA-708 and CEA-708)) can have some style information. Or maybe it wasn't known that it would affect all the media players and browsers.

No matter why, the new UI for caption styles was introduced:

You can choose between five presets and you can also create new ones - but three mistakes were made:

• There is no default 'empty'' preset like all values at 0)
  => You can't turn off all overrides
• The default entries are not included in the individual settings
  => It is impossible to reset individual setting so that the content defaults are used
• And the worst of all:

The shipment of the update has set override values on - probably - many, most or even all Windows machines in the world

Remember - these are override values, which were meant to be applied explicitly only by users.

And even worse: Users cannot get out of this situation, because it is impossible set all values back to zero (= defaults, no override).
People are forced to view subtitles with the Windows styles settings applied and are unable to set all back to default, to have captions being presented in the original way and style.

What you get from the update is this:

The preset is called "Default", but it's just a preset like all the others and it brings explicit values for all properties.

Individual Remedy

As an individual, you can fix this for your own machine, for example by merging a REG file like this: WindowsCaptionsUser.zip

It adds a custom preset named "Controlled by Application" which has all zeros and sets it as default.
But it's not something an application can do. For once, you can't mess around with a user's settings and as a store or UWP app, you don't even have the permissions to do so at runtime.

Point of Return?

For the MS side, it's a similar problem: It has been rolled out already. Can it be reverted in another update? Just adding a "real default" preset wouldn't be sufficient, it would also need to be set active. I'm not sure whether that's feasible...

If it cannot be done like that, then a different method would be needed. For example a way to prevent those styles from being applied. But all other video playback implementations that I mentioned plus those that I don't know about would all need to implement it on their side to get everything right again.

[softworkz]

Is there anybody in charge or do I need to escalate otherwise?
Thanks

[softworkz]

Thanks for transferring.

The offending element in this context is the ClosedCaptionProperties class which conveys the data from the registry key (mentioned above) to CueStyler.cpp (in case of WinUI3).
Unfortunately there is no mechanism implemented to prevent this from happening.

Besides changing the implementation (which only MS can do), I had identified the following ways as potential methods for working around this bug:

1 Unregister the WinRT Class ClosedCaptionProperties

WinUI3

CueStyler.cpp expects a CLASS_E_NOT_REGISTERED error, in which case it is just resorting to apply the settings from TimedTextStyle.

This might work with a WinUI3 app with runFullTrust, but it would require to manipulate the general registration of this class, which would be a really dirty way.

WinUI2

Yet, runFullTrust is not the goal and - most importantly - we're also targeting Xbox/UWP where permissions don't allow any such manipulation.

2 Ship a Custom Replacement for ClosedCaptionProperties

WinUI2

I made some attempts to include a custom implementation of the IClosedCaptionProperties interface in the app, exposed via CSWinRT, but at some point during debugging I ended up at a place where some guards were implemented to prevent an app from providing types in the Windows.x namespace.

What I haven't tried is to do the same but as a C++/WinRT component. I'm not sure whether this might have better chances to inject a runtime replacement for ClosedCaptionProperties...?

3 Manipulate at the Registry Level

WinUI3

Without runFullTrust, there's no way for the application to change those registry values, but eventually I discovered a (rarely documented) way to include certain registry settings in the app package for apps that are running containerized.
This is the only little success I had. It allowed me to override all those ClosedCaptioning values from that registry key, setting them all to 0. It's still not possible to change the values at runtime (like for setting the text effect for which no setting exists in TimedTextStyle), but at least it's an install-time solution to eliminate those caption style settings from Control Pane => accessibility => Caption Themes).

WinUI2

UWP uses a different containerization model and doesn't seem to support those install-time registry settings (no matter what I tried). Even though the tooling for viewing data/registry in the context of the running application was showing the override values, in the registry, the UWP app itself got the actual registry data instead of the overridden ones from the container.

Windows.Media.dll

I "analyzed" the core implementation of Windows.Media, specifically looking for some kind of mechanism to disable or intercept the behavior of ClosedCaptionProperties. I even found one. there's an undocumented way to redirect the registry key from which the configuration is read. But that's rather for achieving a system-wide override for caption settings and requires changing keys under HKLM.
So, it's again not doable for a UWP application package.

Summing Up

There's a half-way reasonable workaround for WinUI3 - but nothing for UWP.
So, the focus and major concern is on WinUI2/UWP for Windows and Xbox

What can be done to resolve or workaround this on UWP? We want to get this ready and ship it to the store rather sooner thatn later.

Thanks again
sw

[brabebhin]

I wouldn't call those workarounds. This is the kind of headache that disqualifies UWP/winui 3 for any project.

[softworkz]

I think in this case, it's been about an action taken on one side (caption styles in sysctl dropping defaults) after it had previously been mandated that all playback components have to obey to those settings (while relying on the premise that defaults will always be neutral).

But what's really not understandable is that it has been reported a year ago already (even when the full impact might not have been visible) and no action has been taken since then. A new framework will always have bugs, but what makes the difference between a solid and relatable platform on which want to build and rely on - or something whether rather like to turn around quickly is whether and how quickly such issues are getting addressed.

It appears that the subtitle rendering has been completely re-built for WinUI3 (presumably due to its disabilities in composition), and when looking at the code and all the (admittedly nice and funny) ASCII paintings there, it surely wasn't done quickly and under time pressure - easily a 6-figure total. With all the effort, time and money having been invested in just this part, it should actually be a red alert for every program manager, when it turns out that the result of that effort gets nullified by the fact that nobody can use it.

Hopefully ASCII art can be suspended for a moment for getting this resolved.

[brabebhin]

This kind of bug in this kind of endeavour shouldn't have seen the outsides of Microsoft offices. Makes you question how did they test the implementation in the first place.

[softworkz]

I believe that the captions settings change came afterwards.

It's also this change alone which is to blame for the mess that it has created.

What goes on this team, is "just" the fact that it hasn't been addressed.

[softworkz]

@codendone - Do you have any updates, what's the status?

Thanks

[softworkz]

Any news?

[brabebhin]

Yes, they made it worse. Some of the built-in fonts that you can select in settings don't work anymore (subtitles render blank lol).

[softworkz]

@codendone - What's the status?

[codendone]

I passed the status request on to the person who this issue is assigned to (internally).

[softworkz]

Oh, that's great news. Thank you very much.

[softworkz]

I passed the status request on to the person who this issue is assigned to (internally).

@codendone - Was there any response so far?

[softworkz]

@ranjeshj

• What does this labelling mean?
• Why did you do it silently without comment?
• And who would be in charge then?

In fact, this is very well actionable in this repository!

It can be fixed in the same file for which I have already submitted two PRs (#9248, #9224).

I just didn't submit a PR because there are several ways to do it. And the PRs are ignored same like most other issues here.

What's actually happening here?
How many developers are working on WinUI3?
Is WinUI3 abandoned and in maintenance mode only?

[softworkz]

Related: Great News: All Bugs Will be Fixed In the Year 2041

[DmitriyKomin]

@softworkz - thank you for the detailed investigation here… it all makes sense, as you mentioned, custom CC settings would get overridden when the default profile no longer had 0 values throughout. Trying this out on public 23H2 OS build 22621.1.amd64fre.ni_release.220506-1250, I was pleasantly surprised to see that (on a fresh install VM), the default is again filled with 0s:

(The HKCU did not have a matching ClosedCaptioning entry).

It appears the regression in Settings you identified has been fixed, and WinUI MPE should again be applying custom CC attributes (specified in .ttm file, etc). Can you confirm the issue is resolved for you on updated OS?

[DmitriyKomin]

@softworkz, closing this issue per above findings. If you can still repro, please comment here and we'll investigate further.

[softworkz]

@DmitriyKomin Thanks a lot for confirming the issue and my findings.

Though - Can you please re-open?

Because the problem is this: The change has already happened, and this means that more than a Billion of Windows devices have these settings applied and these don't get reverted automatically.

The fact that new installations don't have those values is interesting but doesn't change the situation in any way.
(btw, it's still possible that those settings are getting applied at some time later during the personalisation procedures for new users - but anyway, it doesn't matter, the issue exists as is)

[DmitriyKomin]

@softworkz - apologies, you are right... I rushed the conclusion on this :-(. Experimenting more on the 23H2 VM, I see the HKCU key gets populated with nonzero values once Caption style is touched in Settings (including the 'Default' value). So indeed the problem is mostly still present (it does seem the OS default is at least fixed).

The problem is clearly outside of XAML, we'll need help from Windows Settings/Captions area owners. I'm working on getting the right people looped in internally, and post updates here as they come. Thank you for your persistence in getting this important issue resolved.

[softworkz]

@DmitriyKomin - thanks a lot for re-opening.

I have thought a lot about possible remedies since I opened this issue and I see things a bit more clear meanwhile:

• The accident has happened already, quite some time ago. I don't think that there's a chance for turning back in any way
• At some point there must have been an effort in the context of accessibility to make those system settings apply to all cases where media is being played. This goes that far that even the Chromium source code adheres to those registry values on Windows.
• I don't think we can expect any help from the Accessibility/Captions owners
• I'm also not sure whether we can expect much help from the media playback people (assuming these are not owning Windows.Media,Core as well)

And here's why I think so:

In their world, subtitles don't have styles and colors. Subtitle format in Windows.Media has always been very limited and windows media based players needed extra codecs and renderers (like the DirectShow filter 'VSFilter' for SSA/ASS subtitles).
But what's supported out of the box is just formats with no or minimal intrinsic styles (like a word in bold etc.).
This has led to a way of understanding subtitles to be mostly just plain text and it's up to the user to choose color, size, background and outline effect.
From that logic, it made even sense to "centralize" those choices in the system settings app, where a user can set his preferences accordingly and those will be reflected in all applications that do media playback.

The "misfit": Windows.Media.Core

This is what I'm unable to fit into the picture: Windows.Media.Core has introduced an incredibly detailed and comprehensive object model for subtitles with types like TimedTextCue, TimedTextStyle TimedTextLine, TimedTextSubformat, etc.
Windows Media player (old or new) or MediaFoundation do not support subtitle formats that can have such rich styling capabilities like those APIs are supporting, which makes me wonder which team is driving this API and for which use cases?
It's not even old stuff, the TimedTextBouten class has been added not that long ago.

Hence I think the best place to get answers might be the Windows.Media.Core owners.

[softworkz]

Hence I think the best place to get answers might be the Windows.Media.Core owners.

I just fear that these people don't exist anymore (in that position), because otherwise they would have noticed this mess already and taken action, because the current situation essentially renders their designed API unusable.

[softworkz]

Possible Solutions

Let's analyze this a bit to understand the range of possible solutions that even exist on the technical side, so we don't try to rely on some who aren't even able to help in this regard.

We want to prevent the registry settings from overriding our own style settings in the application and the flow goes over 3 "hops" only:

graph LR
   A["1. Registry"] --> B["2. ClosedCaptionProperties"]
    B --> C["3. CueStyler.cpp"]

Loading

1. Registry

This is where the override settings are stored - even when a user has never set any.

• If you are sitting in front of a machine, you can of course make the change manually (see my .reg file above) and your done
• But if you are rolling out an application through the store (UWP or WinUI3), you can't make that change and even when you could (WinUI3+runFullTrus), you still shouldn't change user settings without notice
• A mass-rollout of a Windows update which would revert the mistake is very unlikely to happen, so there's not much to expect in that regard

2. ClosedCaptionProperties

The XAML text cue renderer reads the user's CC settings by using the static WinRT class ClosedCaptionProperties.

For the developer side, I had seen two possible strategies:

2.1 Replace ClosedCaptionProperties [Developer Side]

I had created a class which is mocking ClosedCaptionProperties but returning null/default values instead of the registry values, giving it the same namespace and class name, but I wasn't able to convince the runtime to use it instead of thhe real one.
(maybe I didn't do it right, but probably that's just not allowed)

2.2 Make ClosedCaptionProperties appear unregistered [Developer Side]

CueStyler.cpp gracefully handles the case when the ClosedCaptionProperties WinRT class is not registered. I was able to leverage this by manually removing the class registration from the registry, but I didn't find a way to achieve this within the context of the application.

2.3 Add a``DisableUserSettings` bool property [API Side]

If such kind of flag-property (get/set) could be added to the ClosedCaptionPropertiesstatic class, it would return null/default for all properties if that property is set to true.
Would be a rather simple soution.

3. CueStyler.cpp

This is where the override is actually happening. Same like 2.3, the goal would be to have a kind of condition or flag or behavior to disable the overriding values from ClosedCaptionProperties.
Many ways are possible to achieve this.

[DmitriyKomin]

@softworkz, thank you for the insightful response. I am now engaged internally on this issue with folks SettingsUI (owning Captions page), and also from MediaFoundation (owning ClosedCaptionProperties), and we have an active discussion around possible ways to address this. I don't have anything to report yet, but personally I am hoping for a global solution in SettingsUI/regkey, presumably a new knob of some kind to indicate user preference for app/content-provided caption style. This change definitely requires some design and would take more time, but we're now having the needed conversation.

Regarding your proposed workarounds, #2.3 is closest in sprit (or maybe rather in scope) to what I want to see, but it's questionable giving apps a way to bypass User Settings (without user interaction)... I expect this question will get clarified in the Settings owner I mentioned above. Fixes in WASDK also seem problematic as another "rogue behavior" against user settings, and additionally create inconsistency in behavior of various media players, I am hoping we can avoid this.

I will post an update here with results of the cross-team conversation, and then we pick up this discussion as appropriate. Thanks again for your helpful insights and engagement!