Does turning ARCameraBackground on/off change the camera FOV?

[michelepanegrossi]

Unity 2020.2.1f1
ARFoundation 4.0.9

Testing on Android (Pixel 4) and iOS (iPhone XS).

I am experiencing an issue where my canvases set in world space change size according to whether ARCameraBackground is enabled or disable on the AR camera. Why is that? It looks like the camera turns into a Physical camera but the FOV on the actual camera component doesn't change at all, yet the UI shrinks.

I am trying to have a floating UI with a 3D cube in the background to interact with as if it was an AR layer, but I don't want to see the camera feed until I enable it manually. That's why I am enabling/disabling the ARCameraBackground.

Any idea about how enabling / disabling ARCameraBackground affects the camera FOV or other parameters?

[tdmowrer]

The ARCameraBackground sets the projection matrix, which can change the FOV. If you want to turn off the ARCameraBackground, you can still set the projection matrix yourself by subscribing to the ARCameraManager's frameReceived event.

[michelepanegrossi]

Thanks @tdmowrer for pointing me in the right direction. The projection matrix indeed seems to be the issue.

I fixed the projection matrix shift with this code

private void OnEnable()
{
        _ARcameraManager.frameReceived += OnFrameReceived;
}

private void OnDisable()
{
        _ARcameraManager.frameReceived -= OnFrameReceived;
}

private void OnFrameReceived(ARCameraFrameEventArgs eventArgs)
{
        Camera.main.projectionMatrix = _UIManager.standardPM;
}

using the cached projection matrix from before activating ARCameraBackground.

However this causes an issue where the AR layers don't look as good as they did before. They are slightly enlarged and not as well 'glued' to the markers as they are when the projection matrix is not changed.

I am going to try the opposite, applying the ARCameraBacground matrix to the rest of the scene when the ARCameraBackground is disabled.

[tdmowrer]

You need to use the projection matrix provided by the ARCameraFrameEventArgs, i.e.:

if (eventArgs.projectionMatrix.HasValue)
{
    Camera.main.projectionMatrix = eventArgs.projectionMatrix.Value;
}

(assuming Camera.main is the camera used by the ARSession).

[michelepanegrossi]

You need to use the projection matrix provided by the ARCameraFrameEventArgs, i.e.:

if (eventArgs.projectionMatrix.HasValue)
{
    Camera.main.projectionMatrix = eventArgs.projectionMatrix.Value;
}

(assuming Camera.main is the camera used by the ARSession).

Thanks @tdmowrer I figured that out when I tried to fix the AR tracking issues I was experiencing. Thank you again for pointing me in the right direction!

Unfortunately this solution doesn't work for my use case either, as eventArgs.projectionMatrix.HasValue returns false for about 12-15 frames every time the ARCameraBackground is disabled.

It seems like the camera 'resets' the projection matrix during that time. The result is a glitch where my UI is larger for those 12-15 frames and then shrinks, every time the ARCameraBackground is disabled.

The workaround I found at the moment is to use a special scene when the app is launched, to cache the ARCameraManager projection matrix as soon as it becomes available. Then I load the home of the app.

However in this case I apply the projection matrix to the AR Camera constantly in LateUpdate, apart from when the ARCameraBackground is enabled. Like so:

private void LateUpdate()
{
        if (_isToUpdateMatrix) //This is true when ARCameraBackground is disabled, false when it is enabled
        {
            Camera.main.projectionMatrix = _myARMatrix;
        }
}

where _myARMatrix is the projection matrix I had saved from eventArgs.projectionMatrix.Value when the app started.

This seems to work. Can you think of a reason why this is a bad idea or a better way of doing this?

Thanks a lot!

[tdmowrer]

eventArgs.projectionMatrix.HasValue returns false for about 12-15 frames every time the ARCameraBackground is disabled.

The ARCameraBackground has no effect on the camera frame event, so this should not be the case. Are you doing anything else (i.e., affecting enabled AR features) at the same time?

[michelepanegrossi]

eventArgs.projectionMatrix.HasValue returns false for about 12-15 frames every time the ARCameraBackground is disabled.

The ARCameraBackground has no effect on the camera frame event, so this should not be the case. Are you doing anything else (i.e., affecting enabled AR features) at the same time?

@tdmowrer
Apologies, you are right. eventArgs.projectionMatrix.HasValue always returns True - EDIT - [but my feeling is that it actually returns False for a few frames when the ARCameraBackground is disabled]. However, I experience the behaviour I told you before, where the UI (world space canvases) gets larger for a few frames and then shrinks again, only when the ARCameraBackground is disabled (but not when it is enabled). This makes me think that disabling the ARCameraBackground must cause the projection matrix to reset for a few frames or something similar. What do you think?

See this screecapture video (better to download it as it will be clearer). You can see that everytime the bottom right toggle is deactivated (which disables the ARCameraBackground) the UI 'jumps'

This also happens when the scene first loads, as I disable the ARCameraBackground in Awake.

//Disable the camera feed
_ARCameraBackground.enabled = false;

This is the script I have on the AR Camera to apply the eventArgs.projectionMatrix

using UnityEngine;
using UnityEngine.XR.ARFoundation;

public class FOVReportHasValue : MonoBehaviour
{
    private ARCameraManager _ARcameraManager;

    private void Awake()
    {
        _ARcameraManager = GetComponent<ARCameraManager>();
    }

    private void OnEnable()
    {
        _ARcameraManager.frameReceived += OnFrameReceived;
    }

    private void OnDisable()
    {
        _ARcameraManager.frameReceived -= OnFrameReceived;
    }

    private void OnFrameReceived(ARCameraFrameEventArgs eventArgs)
    {

        Debug.LogWarning($"===== HAS VALUE?? \n {eventArgs.projectionMatrix.HasValue}");
        if (eventArgs.projectionMatrix.HasValue)
        {
            Camera.main.projectionMatrix = eventArgs.projectionMatrix.Value;
        }

    }

}

and this is how I am disabling the ARCameraBackground with the toggle in my UI manager script

    public void OnTogglesChanged(Toggle _whichToggle)
    {
        //Play only when it's on? Maybe a different sound when it's off?
        if (_whichToggle.isOn)
        {
            _toggleSoundFX.Play();
        }

        if (_allToggles.IndexOf(_whichToggle) != 5) //NO Scan toggle
        {
            _TheMissionsPanel.SetActive(!_whichToggle.isOn);
        }

        if (_allToggles.IndexOf(_whichToggle) == 5) //Scan toggle
        {
            _ARCameraBackground.enabled = _whichToggle.isOn;
            _TheMissionsPanel.SetActive(!_whichToggle.isOn);
            _theARCube.SetActive(!_whichToggle.isOn);

            //Tell the tracking manager whether we want to display the prefabs when images are tracked
            _imageTrackingResultsManager.isScanToggleActive = _whichToggle.isOn;

        }
        _allCanvases[_allToggles.IndexOf(_whichToggle)].SetActive(_whichToggle.isOn);
    }

I am not doing much else, apart from SetActive(true/false) on a few UI elements in the world space canvases, no more than two at a time.

[michelepanegrossi]

Hi @tdmowrer
I did a bit more testing and I discovered that although eventargs.projectionMatrix.HasValue always returns true, the camera projection matrix definitely changes when I disable the ARCameraBackground.

I have modified my script to print the camera projection matrix on Update, like this:

using UnityEngine;
using UnityEngine.XR.ARFoundation;

public class FOVReportHasValue : MonoBehaviour
{
    private ARCameraManager _ARcameraManager;

    private void Awake()
    {
        _ARcameraManager = GetComponent<ARCameraManager>();
    }

    private void Update()
    {
        Debug.Log(Camera.main.projectionMatrix);
    }

    private void OnEnable()
    {
        _ARcameraManager.frameReceived += OnFrameReceived;
    }

    private void OnDisable()
    {
        _ARcameraManager.frameReceived -= OnFrameReceived;
    }

    private void OnFrameReceived(ARCameraFrameEventArgs eventArgs)
    {
        Debug.LogWarning($"===== HAS VALUE?? \n {eventArgs.projectionMatrix.HasValue}");

        if (eventArgs.projectionMatrix.HasValue)
        {
            Camera.main.projectionMatrix = eventArgs.projectionMatrix.Value;
        }
    }
}

and I found that the projection matrix resets for one frame when I disable the ARCameraBackground. This is on the Samsung S8:

and this is on the Pixel 4:

I also noticed that when the scene is first loaded, it takes 8 frames on the pixel 4 and 5 frames on the samsung s8 for eventArgs.projectionMatrix.HasValue to return True and for the correct projection matrix to be applied to the camera.

The workaround I described before (caching eventArgs.projectionMatrix.Value on a DontDestroyOnLoad gameobject and then force it in LateUpdate) it is still the only way I found to avoid this issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.