Render from PV Camera Causes Loss of Holograms in MRC

[jbienzms]

Describe the bug

I was struggling with not being able to capture holograms in either Photo or Video mode of MRC. I knew this was application specific because it did not happen in the shell or other apps.

I finally remembered that I enabled the setting "Render from PV Camera". Currently I'm using Unity 2019.2.21f1. Though the UI says "is supported on Unity 2018.4.13f1 or newer", this feature does not appear to work correctly on Unity 2019.2.x builds.

I feel this warning should include more information about which specific Unity builds are supported. I would also like this warning to be more specific about what is meant by "incorrect capture behavior".

Thanks

Screenshots

Your setup (please complete the following information)

• Unity Version: 2019.2.21f1
• MRTK Version: 2.4

Target platform (please complete the following information)

• HoloLens 2

[jbienzms]

Just to include some extra context, this may not be entirely due to a Unity issue. As noted in HoloLens Known Issues updated April 2020:

If the application sets the focus point behind the user or the normal to camera.forward, holograms will not appear in Mixed Reality Capture photos or videos. Until this bug is fixed in Windows, if applications actively set the focus point they should ensure the plane normal is set opposite camera-forward (for example, normal = -camera.forward).

[keveleigh]

if applications actively set the focus point they should ensure the plane normal is set opposite camera-forward (for example, normal = -camera.forward).

Looks like everywhere MRTK sets the focus point makes sure the normal is pointing back at the camera. For example:

MixedRealityToolkit-Unity/Assets/MRTK/SDK/Experimental/Features/Utilities/StabilizationPlaneModifier.cs

Line 358 in 18fe32f

HolographicSettings.SetFocusPointForFrame(planePosition, -gazeDirection, Vector3.zero);

Though the UI says "is supported on Unity 2018.4.13f1 or newer", this feature does not appear to work correctly on Unity 2019.2.x builds.

Additionally, yeah, this text is misleading. It meant "or newer [Unity 2018.4 builds]", not just newer Unity builds. Who's to know which specific 2019 builds are "newer" than 2018.4.13, since they were being released simultaneously! We should call out a specific 2019 build to clear that up.

[Strepto]

I had this same bug, but it only happens for some of my users (possibly OS version dependent)

When recording video and then starting the app, the HoloLens user could not see any holograms. But the holograms were visible on the video afterwards. Disabling "Render from PV Camera" setting resolves the issue. The same happened while using Remote Assist, but then the holograms were not visible for any user.

Unity 2019.4.8, MRTK 2.4 Release, UWP 10.0.18362.0, Legacy XR Pipeline (Windows Mixed Reality v4.2.3 Package)

[jbienz]

Wow, that's really interesting. I had the opposite. I could see the holograms on device but not see them in the MRC video. Both would be good to describe in the docs update.

@keveleigh or @wiwei any thoughts on who we could assign this to? Looks like both a docs update and potentially a code fix too.

Urgency isn't critical, but it is difficult to figure out when it happens.

[fast-slow-still]

I can get 100% repro on this, using Unity 2019.4.2f1. It appears the MRC is ignoring the MRTKPlayspace transform and only using the camera's local transform.

Repro steps:

1. Open HandInteractionSamples scene.
2. Set "Render from PV Camera" checkbox (will need to clone some profiles to do this).
3. Set the MixedRealityPlayspace position to (1, 0, 0).
   3.a) Note that something intermittently sets the camera's transform to counteract the playspace offset. Make sure camera transform is still identity.
4. Observe in game view that camera is displaced to the right, as expected.
5. Build, deploy, and run.
6. Observe in HL2 that camera is displaced to the right, as expected.
7. Open MRC in Windows Device Portal
8. Start Live Preview
9. Observe that camera is still centered, unlike what user sees in HL2 (or Unity game view).

Note that if "Render from PV Camera" is not enabled, then there is no disparity, the MRC and the user view are both displaced to the right as expected.

[jbienz]

@keveleigh @davidkline-ms This seems to be affecting more customers. Any chance we can bump this up? It appears this may be a Unity issue and not something we can fix. There is still the question @keveleigh posted above about whether there's anything we need to be careful about with camera normals. If nothing else, could we figure out which Unity 2019 and 2020 versions this is fixed in and update the docs for the feature?

@polar-kev This is an example of one of the bugs I would like to see a version table for. "If you need Render from PV camera, here are the Unity versions where this feature works as designed."

[keveleigh]

This seems to be affecting more customers.

My read of this is that this thread has come to describe multiple issues:

1. The original report on Unity 2019.2.21f1, which had plenty of issues with this feature. These should all be fixed on legacy on the latest 2019.4 releases. Holograms disappearing should be fixed in XR SDK on the latest 2019.4 releases, but holograms aren't yet properly aligned in the resulting screenshots/videos. A fix for this is in progress.
   1. I haven't been able to repro any "holograms not appearing in screenshots/videos under 'normal' (not case 3 below) circumstances" issues in a few months now. If you're still seeing them on recent releases, then we'll definitely need to report a new bug on that.
2. A second issue, where holograms are reported to stop rendering on-device while the recording is happening. This is a new report for me, and we'll have to raise this to Unity.
3. Then there's a different issue, where the additional holographic camera isn't offset in the same way the main camera is (for example, if the main camera is under an MRTK Playspace which has been teleported). The additional camera appears to still be treated as though the main camera is relative to the world origin. This issue is newly reported and hasn't been fully investigated yet.

[soshogoh]

The customer is using MRTK HandMenuExample that can reproduce this issue.
The hologram is not showing up on the LivePreview, not on the device.

The customer told us that Unityの2019.4 2019.4.15f1 could fix this problem.

The customer environment which reproduce this issue:
・Hololens2 10.0.19041.1391
・Unity 2019.4.13f1
・MRTK 2.5.1

[polar-kev]

if applications actively set the focus point they should ensure the plane normal is set opposite camera-forward (for example, normal = -camera.forward).

Looks like everywhere MRTK sets the focus point makes sure the normal is pointing back at the camera. For example:

MixedRealityToolkit-Unity/Assets/MRTK/SDK/Experimental/Features/Utilities/StabilizationPlaneModifier.cs

Line 358 in 18fe32f

HolographicSettings.SetFocusPointForFrame(planePosition, -gazeDirection, Vector3.zero);

@keveleigh, is there any further investigation that needs to be done on this?

[JBrentJ]

@polar-kev Do you know if this setting is true in MRTK version 2.3 as well?

[polar-kev]

Hi @soshogoh, is it just the hand menu that's disappearing or is it all holograms that are disappearing?

[BillingsAm3]

@polar-kev Do you have an estimated timeline for this fix? This is our second oldest partner issue, so we want to reach resolution for our partner ASAP.

[polar-kev]

Status update on not being able to record holograms during mixed reality capture,

Fixed in

• Unity's legacy XR with Unity 2018.4.13f1 and newer if using Unity 2018,
• Unity's legacy XR with Unity 2019.4. @elbuhofantasma has tested and confirmed this is working is Unity 2019.4.9. Please use 2019.4.9 or later if using Unity 2019.
• Unity's XR SDK with OpenXR and Unity 2020.2 or newer if using Unity 2020. Please note that OpenXR is still in preview and Unity 2020 is not officially recommended by the MRTK as yet.

Not Yet Fixed in

• Unity's XR SDK with Unity 2019. This is not an MRTK issue, it's a Unity bug. @JBrentJ is tracking. I'll leave this MRTK issue open for visibility.

[polar-kev]

@RogPodge, can you update the Mixed Reality Capture profile "information" caption (see the screenshot in the issue description) with the following, "Render from PC camera is supported in Unity 2018.4.13 and newer if using Unity 2018, and Unity 2019.4.9f1 and newer if using Unity 2019".

[keveleigh]

@polar-kev There's also the issue about

the additional holographic camera isn't offset in the same way the main camera is (for example, if the main camera is under an MRTK Playspace which has been teleported). The additional camera appears to still be treated as though the main camera is relative to the world origin.

which seems to be fixed in Unity 2020 but is still present across the board in 2018 and 2019.

[polar-kev]

Not Yet Fixed in

*** Unity's XR SDK with Unity 2019. This is not an MRTK issue, it's a Unity bug. @JBrentJ is tracking. I'll leave this MRTK issue open for visibility.**

This is now fixed as of Unity 2019.4.25

---

@polar-kev There's also the issue about

the additional holographic camera isn't offset in the same way the main camera is (for example, if the main camera is under an MRTK Playspace which has been teleported). The additional camera appears to still be treated as though the main camera is relative to the world origin.

which seems to be fixed in Unity 2020 but is still present across the board in 2018 and 2019.

This is now fixed as of Unity 2019.4.25 if using 2019 and Unity 2018.4.34 if using 2018.