Kneeboard flickers in DCS MT

[Wyverex42]

Hi,

I'm the author of IvyVR (https://ivyvr.net) and first of all let me thank your for writing up https://fredemmott.com/blog/2022/05/31/in-game-overlays.html. This has been an invaluable tool for me while figuring out how to render textures into VR using shared memory!

I'm currently converting IvyVR to use OpenXR instead of OpenVR and I've roughly followed what you're doing to render quads using OpenXR. I got a prototype working now but I have the same issue that OpenKneeboard has when I run DCS: Quads flicker a lot. This has already been an issue with OpenKneeboard before I even started my own API layer, so I assume it must have something to do with how you render the quads in the first place.

At the moment I'm pretty stumped as to why this happens. Did you see this on your own end as well?

I'm using a Valve Index, I'm running OpenKneeboard 1.4.2 and I have a solid 90 FPS in the menu.

2023-05-29.19-22-21-1.mp4

[Wyverex42]

Some more observations:

• There's no flicker at all while loading into a mission (e.g. while the SteamVR grid is showing) or while DCS starting up (before entering the menu)
• Interestingly the flicker seems to disappear for my API layer when I'm in a mission while it's still present in OpenKneeboard
• Once I go back to the menu, the flicker is present again (for both my layer and OpenKneeboard)
• I tried doing the texture rendering in xrBeginFrame instead (and only pushing the quad in xrEndFrame) but that doesn't make any difference

I can live with that result for my layer but I still don't understand why it happens. There's definitely some weird interaction with DCS here.

[fredemmott]

One possibility: Make sure that OpenKneeboard has SteamVR or OpenXR enabled, not both

I’ve not seen flicker with steamvr and either a quest or G2, but those should have direct openvr drivers that bypass the steamvr compositor, while an index or vive will use the steamvr compositor

[fredemmott]

Another possibility: openkneeboard only updates the swap chain when it actually has new content. This is more efficient, but some software assumes that the swap chain is updated every frame. This used to be an issue with openxr toolkit and the obs mirror api layer, but both have now fixed.

as a side note, once you’re on openxr, you’re able to use quest hand tracking via oculus link in developer mode; this isn’t currently supported via virtual desktop (it only does controller emulation)

[fredemmott]

(Unlike openkneeboard, https://github.com/fredemmott/HTCC is licensed in a way that allows reuse under other terms, including commercial)

[Wyverex42]

One possibility: Make sure that OpenKneeboard has SteamVR or OpenXR enabled, not both

I've disabled OpenVR in OpenKneeboard and, for good measure, also set both DCS executables to use OpenXR

Another possibility: openkneeboard only updates the swap chain when it actually has new content. This is more efficient, but some software assumes that the swap chain is updated every frame. This used to be an issue with openxr toolkit and the obs mirror api layer, but both have now fixed.

Possible. My own layer doesn't have such an optimization, at the moment I rerender the texture each frame (although the content stays the same). So if that's the reason, it would explain why it doesn't flicker in a mission but then I'd also expect my layer not to flicker in the menu. So there must be something else going on here.

Maybe I'll stumble on a solution by accident when moving the prototype to production

[Wyverex42]

(Unlike openkneeboard, https://github.com/fredemmott/HTCC is licensed in a way that allows reuse under other terms, including commercial)

Interesting, I didn't know this existed. If I understand this correctly, you're translating finger tracking (+ some input buttons) into the equivalent of click events, right? That's a different approach to mine, I detect connectors in the cockpit and translate gestures on them to device commands using the DCS export API. Your approach is much cheaper to implement while mine offers the added immersion of reaching out and performing a gesture matching the control type. But I have to put way more effort in to make it feel good. Both have their merits!

As for the license, I should clarify that I'm not actually copying any of your code. I just read it as an inspiration to understand how you implemented the overlay concept in OpenXR. I also prefer to properly learn how to use OpenXR rather than just copying code.

[fredemmott]

If I understand this correctly, you're translating finger tracking (+ some input buttons)

Input buttons are required except on Quest Link developer mode - for that, just hand tracking is an option: I use the gesture bits from XR_FB_hand_tracking_aim

I tend to use the pointctrl buttons even with a quest though - no false positives/negatives with those, unlike gestures.

into the equivalent of click events, right?

Depends on the configuration.

It can 'point' either by emulating an Oculus controller via openxr (this doesn't require an Oculus headset), or by emulating a tablet (like a mouse, but absolute coordinates rather than relative)

It can click either by sending mouse clicks, or by emulating openxr controller inputs. The controller inputs it emulates are game-specific - for example, 'left click' is 'move thumbstick left' on a quest controller in DCS, not the trigger or a/x buttons, but generally it's a trigger in MSFS.

These can be mixed - for example, for MSFS, point with openxr controller, click with mouse works best.

Your approach is much cheaper to implement while mine offers the added immersion of reaching out and performing a gesture matching the control type

Definitely advantages to both - my personal ideal would either be yours with hand tracking, or HTCC but with 'controls are magnetic' to reduce the need to be precise - I think this is similar to mouse in BMS, or your lock feature. But I don't currently have the time to make and maintain per-aircraft code.

As for the license, I should clarify that I'm not actually copying any of your code. I just read it as an inspiration to understand how you implemented the overlay concept in OpenXR.

No problem - I meant it more of a 'you can grab HTCC if you want' - I only mentioned OpenKneeboard's license given where this issue is :)

[fredemmott]

Another possibility: openkneeboard only updates the swap chain when it actually has new content. This is more efficient, but some software assumes that the swap chain is updated every frame. This used to be an issue with openxr toolkit and the obs mirror api layer, but both have now fixed.

Possible. My own layer doesn't have such an optimization, at the moment I rerender the texture each frame (although the content stays the same). So if that's the reason, it would explain why it doesn't flicker in a mission but then I'd also expect my layer not to flicker in the menu. So there must be something else going on here.

Maybe I'll stumble on a solution by accident when moving the prototype to production

If you're willing to test a little, there should be a new MSI at the bottom of https://github.com/OpenKneeboard/OpenKneeboard/actions/runs/5124207707 in ~ 45 minutes

This adds this option at the bottom of 'advanced settings':

This has absolutely no effect on a Quest, even when using SteamVR-via-OpenXR, but it'd be great to know if it helps SteamVR-native headsets.

You can see the difference in render calls here - I'm running hello_xr, and the option in OpenKneeboard takes effect instantly. I switch it off and on a few times, and neither hello_xr nor OpenKneeboard need restarting:

... though the 'render' here is just a texture copy. The once-per-second update with the option off is to update the clock in the footer.

[fredemmott]

On another tangent, you might find https://github.com/fredemmott/openxr-tracing useful :)

[fredemmott]

There's no flicker at all while loading into a mission (e.g. while the SteamVR grid is showing) or while DCS starting up (before entering the menu)

DCS does seconds-per-frame rather than frames-per-second at these points :)

[Wyverex42]

Definitely advantages to both - my personal ideal would either be yours with hand tracking, or HTCC but with 'controls are magnetic' to reduce the need to be precise - I think this is similar to mouse in BMS, or your lock feature. But I don't currently have the time to make and maintain per-aircraft code.

Making controls magnetic is a huge chunk of work on its own (at least in DCS, I haven't looked into MSFS yet). You have to parse the EDM file to read connector positions and their transformation rules, so that you're always in the right spot. Once that's automated (and bug-free), it doesn't need any maintenance.

The per-aircraft bit is making sure that automatic detection of control types is correct and that the results are intuitive. It's a pain in the arse for sure, so I built a lot of tooling to make supporting an aircraft as easy as possible. When I started out I wanted to make everything automatic but for some things it's unfortunately not possible.

Once I'm on OpenXR I'm definitely going to look into hand tracking, especially since the Pimax Crystal I've preorderd is supposed to get finger tracking soon.

[Wyverex42]

If you're willing to test a little, there should be a new MSI at the bottom of https://github.com/OpenKneeboard/OpenKneeboard/actions/runs/5124207707 in ~ 45 minutes

This adds this option at the bottom of 'advanced settings':

Unfortunately, this version crashes when I click on "Advanced Settings". Crash dump here: https://drive.google.com/file/d/1sip7ejdVwxDikku6Y2s5q4hhlLsQEVei/view?usp=share_link

[Wyverex42]

On another tangent, you might find https://github.com/fredemmott/openxr-tracing useful :)

Ah that looks useful. I've seen the Tracing calls in your code but I didn't have the additional context for how to use them. Does this library trace all OpenXR functions? Atm I'm just writing a plain log file for what I'm interested in specifically

[fredemmott]

On another tangent, you might find https://github.com/fredemmott/openxr-tracing useful :)

Ah that looks useful. I've seen the Tracing calls in your code but I didn't have the additional context for how to use them. Does this library trace all OpenXR functions? Atm I'm just writing a plain log file for what I'm interested in specifically

Yep, all OpenXR functions that are defined in OpenXR v1.0.27, excluding debug-only and not-on-windows ones, but including most other vendor-specific ones; it's generated from the XML version of the specification.

Unfortunately, this version crashes when I click on "Advanced Settings".

Well this is fun, of course it only crashes in release builds, not debug builds. And seems to be a winui internal error. Thanks for trying :)

[fredemmott]

Re debugging the crash: this doesn't make sense to me. Nothing's changed that should lead to WinUI attempting to load another image processing DLL.

I'm trying https://github.com/OpenKneeboard/OpenKneeboard/actions/runs/5126750757 which is a re-run of build #1515 , which works, to narrow down if this is an environment change (visual studio update?) or a code change.

[fredemmott]

Updated to latest MSVC, does not crash, also does not crash with locally-made release builds, or on-CI rebuilds of previous good versions. Combo of environment and code apparently :/

[fredemmott]

This is an installer bug; clean install of 1.4.3 + upgrade to linked version crashes. repair via installer fixes it.

[fredemmott]

This is an installer bug

In particular, the installer doesn't install WinUIEdit.dll when upgrading - only on fresh installs or repairs

Install log when upgrading:

1040:MSI (s) (B0:3C) [23:14:25:961]: Component: CM_CP_Default.bin.WinUIEdit.dll; Installed: Absent;   Request: Local;   Action: Null
1745:MSI (s) (B0:EC) [23:14:26:151]: Component: CM_CP_Default.bin.WinUIEdit.dll; Installed: Local;   Request: Absent;   Action: Absent
3890:MSI (s) (B0:EC) [23:14:27:148]: Executing op: FileRemove(,FileName=WinUIEdit.dll,,ComponentId={C988FB96-0987-59AF-ACD3-80D0F3A091B6})
3891:RemoveFiles: File: WinUIEdit.dll, Directory: C:\Program Files\OpenKneeboard\bin\
3892:MSI (s) (B0:EC) [23:14:27:148]: Verifying accessibility of file: WinUIEdit.dll
5027:MSI (s) (B0:3C) [23:14:27:487]: Executing op: ComponentRegister(ComponentId={C988FB96-0987-59AF-ACD3-80D0F3A091B6},KeyPath=C:\Program Files\OpenKneeboard\bin\WinUIEdit.dll,State=3,,Disk=1,SharedDllRefCount=0,BinaryType=1)
5028:1: {5C75091F-2AD9-4D92-8FF8-DBE7ECC1047C} 2: {C988FB96-0987-59AF-ACD3-80D0F3A091B6} 3: C:\Program Files\OpenKneeboard\bin\WinUIEdit.dll

[Wyverex42]

Good news!

The installer repair fixed the crash and I can confirm that the flickering (in a mission) is gone with "Always update swapchains" enabled. If I disable the option, the flickering returns.

It's still there in the menu every few seconds but I can live with that.

That's valuable information for me as well because I was also considering to do some optimizations there given that some of my overlays like the crosshairs don't really change. I'll have to make that configurable too.

Thanks for your help!

[fredemmott]

Thanks; I'll flip the default to on for the next version then.

Hoping MS will fix the versioning on WinUIEdit.dll - there's workarounds I can do, but the Windows Installer really doesn't like 'downgrading' DLLs when upgrading. Probably would need to entirely stop using cmake to generate the installer to do the 'right' workaround here.

Old version: WinUIEdit.dll follows Windows SDK versioning, so is 10.x
New version: WinUIEdit.dll follows Windows app SDK versioning, so is 1.3.x

[fredemmott]

Discussed with some other OpenXR developers, reported bug here (pending moderation): https://steamcommunity.com/app/250820/discussions/0/3843305519472175943/

[fredemmott]

Leaving open for now; depending on Valve's response (or lack of), I'll either flip the default when using SteamVR, or remove the option before the next release.

[fredemmott]

also filed #360 to separately track the Windows App SDK DLL versioning issue

[Wyverex42]

@fredemmott I've figured out where the flickering is coming from, at least on my side.

Turns out it was a threading issue. Using your trace layer I noticed that DCS runs the rendering OpenXR commands in a different thread than the action syncing commands. Curiously, xrWaitFrame also runs on the action thread (presumably the main thread). That's where I generate my input data for the frame which is then sent to IvyVR. The output data is processed in xrEndFrame. However, new output data is only generated if new input data was generated and I didn't create a copy of the output data for optimization reasons.

So what seems to happen then is that sometimes the rendering thread calls xrEndFrame before new output data is available and then I wouldn't render my quads in that frame. I tried copying my output data and using the local copy instead, and now my quads stopped flickering. The flickering was even more pronounced in BeatSaber (although there commands run a bit differently, but it was ultimately still the same reason).

I haven't delved deeply into how your shared memory protocol works but I have a hunch that OpenKneeboard also assumes single-threaded input and output processing and thus flickers for the same reason?

[fredemmott]

I do everything in xrEndFrame(), and a snapshot is created and stored in-process. ID3D11Fence's are used to guard reads, so it should be thread-safe

[Wyverex42]

Good point about doing everything in xrEndFrame.

I've also changed my code to only acquire and render a swapchain image if the content has actually changed. For my crosshairs this happens very rarely, so I basically never render the swapchains again. And there's still no flicker. OpenKneeboard stopped flickering after your fix above which made it render each frame though. So interestingly, what introduces flicker in OpenKneeboard doesn't do it for me.

[fredemmott]

This is reportedly fixed in the latest SteamVR beta; can anyone here confirm?

[fredemmott]

Closing: Valve claim it's fixed, and no further feedback.