[Bug] AR Foundation TryAcquireLatestCPUImage broken (IOS 16.2)

[VanIseghemThomas]

Describe the bug
The function TryAcquireLatestCpuImage causes the preview to freeze on IOS.

To Reproduce
Steps to reproduce the behavior:
Put the following MonoBehaviour on a GameObject in the scene:

using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
 
public class TestXRCpuImage : MonoBehaviour
{
    public ARCameraManager cameraManager;
   
    void OnEnable()
    {
        cameraManager.frameReceived += FrameChanged;
    }
 
    void OnDisable()
    {
        cameraManager.frameReceived -= FrameChanged;
    }
 
    private void FrameChanged(ARCameraFrameEventArgs args){
        if (!cameraManager.TryAcquireLatestCpuImage(out XRCpuImage image))
            return;
    }
}

Then build the app for IOS.

Expected behavior
The image is returned and nothing happens.

Actual behavior
The image is returned and the preview freezes, making the application unusable. No errors are logged to the console about this.
The application itself isn't locking up since all other functionallities still work like expected, it's just the camera preview.
It is important to note that this was working without problems before updating to IOS 16.2.

Platform

• Device: iPad Pro 3th gen
• OS: IOS 16.2
• Unity version: 2021.3.14f
• ARFoundation version: 5.0.2 & 5.0.3
• ARKit 5.0.2 & 5.0.3
• Xcode 14.1

[Unity3D-Hardik]

Add logic and get "TryAcquireLatestCpuImage" every X second (0.5f or 1) in place of every frame that will resolve your problem...

[VanIseghemThomas]

Hi @Unity3D-Hardik, what do you mean with "login" exactly?

[andyb-unity]

Hi @VanIseghemThomas,

The code you have provided will always cause the camera to freeze. XRCpuImage implements the IDisposable interface, meaning that you are responsible to dispose the XRCpuImage instance if TryAcquireLatestCpuImage returns true.

Failing to dispose the XRCpuImage is a memory leak, and repeated failures to dispose will cause ARKit to run out of memory. When ARKit runs out of memory, it can no longer allocate new textures for the camera, causing the camera to freeze.

The solution here should be to check the result of TryAcquireLatestCpuImage and always Dispose() the XRCpuImage instance if TryAcquire was successful.

For more information see our Camera Image docs: https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.2/manual/cpu-camera-image.html. (I've linked the 4.2 version here because we are currently in the process of releasing 5.0.4, and the new release has changed the structure of the Camera docs which will cause URL's to break)

[Unity3D-Hardik]

Sorry, that was a typo. In place of every frame try to get TryAcquireLatestCpuImage on every 0.5f or 1 second.. If you want i can share logic.

[VanIseghemThomas]

Hi @andyb-unity thanks for reaching out,

Seems that I indeed forgot to dispose the image and this was the cause for the preview locking up. I did however still have problems I couldn't figure out causing it to freeze even when dispose was called. I was able to fix this by creating a seperate class that just fires an event whenever an image is available:

using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

public class XRCpuImageFetcher : MonoBehaviour
{
    public ARCameraManager cameraManager;

    void OnEnable()
    {
        cameraManager.frameReceived += FrameChanged;
    }

    void OnDisable()
    {
        cameraManager.frameReceived -= FrameChanged;
    }

    private void FrameChanged(ARCameraFrameEventArgs args){
        if (cameraManager.TryAcquireLatestCpuImage(out XRCpuImage image)){
            ImageAvailable?.Invoke(image);
            image.Dispose();
        }
    }

    public delegate void ImageAvailableDelegate(XRCpuImage image);
    public event ImageAvailableDelegate ImageAvailable;
}

Seems that it's really finicky.
Also I've found an error in the documentation. In the example code, cameraManager.cameraFrameReceived += OnCameraFrameReceived; should be cameraManager.frameReceived += OnCameraFrameReceived;

@Unity3D-Hardik Seems that if you've had the same problem, you might want to try this.

[romanvanderaa]

Hii, I'm encountering a similar issue with TryAcquireLatestCpuImage, but before trying to figure out what causes my errors, I want to investigate if I'm trying the right method. I want to capture the camera feed, without GUI and instantiated augmented 3D objects. There's not much I can find about this method, do you guys know if this is a proper solution for my case? Does TryAcquireLatestCpuImage even return a clean camera feed? I know this question was closed, but maybe someone can answer me anyways :)

[ankur-unity]

@romanvanderaa - see #1058 (comment)

[andyb-unity]

@VanIseghemThomas

Also I've found an error in the documentation

Thanks for letting us know. As I previously mentioned, we have made a significant update to the Camera docs for the 5.0.4 patch. One of the changes is that all the sample code is now compiled as part of our package tests, so this error has been fixed and future similar errors on this page are highly unlikely.

[VanIseghemThomas]

Alright, thanks for the help @andyb-unity !