Add CloudAnchorNode

[morhenny]

I have been having a very strange issue in my project that occurs after resolving AR Core Cloud Anchors, where I am not entirely sure if this error is related to sceneview, but since I am unable to find the root cause of this error, I figured you might know something or have had a similar problem like this before.

So my app does not crash with any "normal" exceptions or stacktrace, but when navigating away from my AR fragment, the app will just close with the following error message that doesn't really help too much:

I/native: I0304 19:19:53.140083   16305 session_lite_c_api.cc:37] Deleting ArSession...
A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x6c00000018 in tid 25038 (Thread-20), pid 16305 (nn.arsamplecode)

While trying to find the cause, I could pinpoint it on callling session.resolveCloudAnchor() with an existing anchor ID.
Turns out, while AR Core is still trying to resolve a cloud anchor and hasn't found it yet, it will crash when trying to navigate away from the fragment. As soon as the anchor is successfully resolved, navigating works perfectly fine again.

I created a small sample to hopefully reproduce the issue, though for that you will have to use your own Cloud Anchor API Key and an existing cloud anchor ID.
In the sample, tapping on the AR plane will call the resolveCloudAnchor, and then simply navigating back will crash the app.

Since I really am out of approaches of how to fix this, I was hoping that maybe y'all have some experience with these kind of errors or potentially have an idea how I can solve this issue.
Thanks alot in advance

[ThomasGorisse]

Hi,

Your issues sound like an ArCore recent known issue which doesn't release the session until you still have unresolved cloud anchors.

Could you please add a breakpoint inside the `ArSceneView.onDestroy()' to first see if it's called at the same time as 'Fragment.onDestroy()' ?

@RGregat @grassydragon Does it remind you something and what was the ArCore original issue to check if the 1.30.0 might resolved it.

[grassydragon]

Hi!
Yes, here is the original issue and a workaround is to detach the anchor before the ARCore session is destroyed: SceneView/sceneform-android#209

[ThomasGorisse]

@morhenny Could you please update the ARCore dependency to 1.30.0 without touching anything else so we see if it has been solved?

[morhenny]

So I just tried to use 1.30.0, but that doesn't seem to fix it. For that I only changed the arcore_version in the ArSceneView build.gradle, not sure if there wouldn't been another place to change it.

I also just realized that I was smart enough to forget to link my sample, so even tho you might not need it, here it is: sample

I'll try the detach workaround now and see if that works

[morhenny]

Awesome, calling detach() on the anchor did actually resolve my issue, but I have to call it in onStop(), because the cloudAnchor reference already seems to be null when onDestroy() is called.

I appreciate the quick help alot!

[ThomasGorisse]

Your code looks good but you could remove all this part : https://github.com/morhenny/sceneview_issue_sample/blob/master/app/src/main/java/de/morhenn/arsamplecode/SecondFragment.kt#L59-L82

and use the arNode.loadModel() instead of loading the renderable the old way.

Could you create a quick Pull Request which destroys the whole session anchors just before the session.close() is called?
It should be placed directly inside the ArSession class. Use the session.anchors field to get all of them.
This will help us a lot to not forget it one more time and it should help release memory a lot for other use cases if ARCore have other issues on it.
Don't worry if you do something wrong, we will validate it.

[morhenny]

Not sure if that's already everything, but I think just changing the onDestroy() of the ArSession to this should be it:

override fun onDestroy(owner: LifecycleOwner) {
        allAnchors.forEach {
            it.destroy()
        }
        close()
    }

I am unable to test if this actually fixes the issue though because I cannot locally publish the project at the moment due to some compiler errors. Did I break something on my end, or is the current master branch not able to build?

[ThomasGorisse]

My fault sorry about that. I forgot to push some commits.

[morhenny]

No worries, I'll test it tomorrow then, or whenever you get around pushing those, to make sure that it actually does fix it

[morhenny]

I seem to still run into the same or a very similar SIGSEGV error that only happens very rarely now, and is not related to closing the ar fragment.
It seems to only happen after having used the app for a bit, and this crash occurred right when I opened my AR Fragment again and tracking started. The crashlog now also includes a warning about maxImages, so this might be a different but similar problem that is happening. This is the crash log I got this time:

I/native: I0306 18:44:52.660849   19333 model_inference_cpu.cc:82] Successfully started the ArDepthCalculator graph.
W/native: W0306 18:44:52.668274   19333 plane_estimator.cc:661] Got a new plane, id: 0, with a invalid statistics.
I/native: I0306 18:44:52.677370   19470 jni_util.cc:41] GetEnv: not attached
E/native: E0306 18:44:52.696742   18212 hit_test.cc:428] generic::internal: No point hit.
I/tflite: Replacing 112 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 12 partitions.
I/n.ar_navigatio: NativeAlloc concurrent copying GC freed 838907(26MB) AllocSpace objects, 147(9052KB) LOS objects, 75% free, 18MB/73MB, paused 93us,224us total 205.126ms
A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x10 in tid 18212 (n.ar_navigation), pid 18212 (n.ar_navigation)
W/NdkImageReader: Unable to acquire a lockedBuffer, very likely client tries to lock more than maxImages buffers
W/System: A resource failed to call release. 
I/native: I0306 18:44:53.159575   18224 session_lite_c_api.cc:37] Deleting ArSession...
I/native: I0306 18:44:53.159980   18224 session_lite_c_api.cc:39] Deleted ArSession.
W/System: A resource failed to call release. 
W/System: A resource failed to call release. 
W/System: A resource failed to call release. 
I/native: I0306 18:44:53.168793   18224 session_lite_c_api.cc:37] Deleting ArSession...
I/native: I0306 18:44:53.168838   18224 session_lite_c_api.cc:39] Deleted ArSession.
W/System: A resource failed to call release. 
W/ImageReader_JNI: Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers

A few days ago I had a random other crash of my app, where I don't quite remember how it happened and I never seemed to figure out what exactly caused it, but it also referenced something with maxImages, so since it might be related, here is that stacktrace as well

I/native: I0305 00:19:53.155586   14268 session_lite_c_api.cc:37] Deleting ArSession...
I/native: I0305 00:19:53.155686   14268 session_lite_c_api.cc:39] Deleted ArSession.
W/ImageReader_JNI: Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers
E/AndroidRuntime: FATAL EXCEPTION: ImageStreamHandler
    Process: de.morhenn.ar_navigation, PID: 14256
    java.lang.IllegalStateException: maxImages (16) has already been acquired, call #close before acquiring more.
        at android.media.ImageReader.acquireNextImage(ImageReader.java:560)
        at android.media.ImageReader.acquireLatestImage(ImageReader.java:441)
        at dlu.onImageAvailable(PG:9)
        at android.media.ImageReader$1.run(ImageReader.java:837)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loopOnce(Looper.java:201)
        at android.os.Looper.loop(Looper.java:288)
        at android.os.HandlerThread.run(HandlerThread.java:67)
E/FATAL: A fatal crash caused the app to stop
    java.lang.IllegalStateException: maxImages (16) has already been acquired, call #close before acquiring more.
        at android.media.ImageReader.acquireNextImage(ImageReader.java:560)
        at android.media.ImageReader.acquireLatestImage(ImageReader.java:441)
        at dlu.onImageAvailable(PG:9)
        at android.media.ImageReader$1.run(ImageReader.java:837)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loopOnce(Looper.java:201)
        at android.os.Looper.loop(Looper.java:288)
        at android.os.HandlerThread.run(HandlerThread.java:67)
I/n.ar_navigatio: System.exit called, status: 1
I/AndroidRuntime: VM exiting with result code 1, cleanup skipped.

[grassydragon]

I think it is related to this method call: https://developers.google.com/ar/reference/java/com/google/ar/core/Frame#acquireCameraImage()

[ThomasGorisse]

Looks like the issue we had with Depth Texture.
Have you enabled depth occlusion inside your app?
Are you using any acquire functions mentioned above?
Those ones are pretty sensitive and need to be carefully released and not called to much.

[morhenny]

As far as I know I did not enable anything related to depth occlusion, these are my only configuration apart from the default:

config.cloudAnchorMode = Config.CloudAnchorMode.ENABLED
config.planeFindingMode = Config.PlaneFindingMode.HORIZONTAL
config.updateMode = Config.UpdateMode.LATEST_CAMERA_IMAGE
sceneView.planeRenderer.planeRendererMode = PlaneRenderer.PlaneRendererMode.RENDER_ALL

I also am not using any of those acquire functions either, the only things I do in my ar fragment are hosting and resolving cloud anchors and placing quite a few ArNodes

[ThomasGorisse]

If you are not using depth occlusion, the only place where we acquire images is within the LightEstimation and they are closed after using them:

sceneview-android/arsceneview/src/main/kotlin/io/github/sceneview/ar/arcore/LightEstimation.kt

Line 445 in c7d482d

image.close()

Could you just disable light estimation to see if it could come form here?

[morhenny]

Sure, I added config.lightEstimationMode = Config.LightEstimationMode.DISABLED to my configurations, but since this was a rare bug, I can only try to reproduce it again, but that might take some time.

Regarding the original issue and that Pull Request you asked about @ThomasGorisse
I did test to detach all the anchors onDestroy, but I found a few problems there:

1. allAnchors list ist already empty onDestroy, so this would have to be done latest in onStop
2. Seems like each in allAnchors are already detached when I call it onPause() (When I have normal nodes as anchors)
3. Not-yet-fully resolved cloud anchors are never in that allAnchors list to begin with, so they can't be detached this way. Seems like them never getting added to the sceneView list of anchors might be the cause of this then.

Since my call of sceneView.session?.resolveCloudanchor() will return that anchor in my code tho, I can just manually keep a reference and then detach it onStop

[ThomasGorisse]

Thanks for the PR answer.
Could you try to override some functions like this one inside ArSession class and see if they are called even when you create an anchor from another function:

override fun createAnchor(pose: Pose?): Anchor {
    return super.createAnchor(pose)
}

The goal is to find the lowest place where anchors are created in order to keep a ref to it.
The problem is that we must not detach anchors at onPause() since they need to be here at onResume()

[morhenny]

I think the problem is that the cloudAnchor are never actually added to the sceneview list, when they are not yet resolved. All the normal anchors that get added to the scene are being detached just fine already onStop, as far as I could tell.
If there would be an overwritten version of the session.resolveCloudAnchor() within sceneView, that reference could be added to the sceneView anchorList there.

When developing with CloudAnchors, you have to implement your own CloudAnchorManager to handle the callbacks and keep track of those anchors in progress. I feel something like this could have a good place in sceneView to reduce a lot
of boilerplate code for someone working cloudAnchors and is where I am keeping track of those in-progess anchors in my application now and where I call detach. Google ArCore Sample CloudAnchorManager

[ThomasGorisse]

Thanks for the explanation.
It would be great to have a CloudAnchorNode within SceneView.
It could try to resolve the cloud anchor inside the overrided ArNode.onArFrame(arFrame : ArFrame)
I still got a lot of subjects to threat before the Cloud one. Would you make it?
We will check it and merge it directly within ArSceneView module.

[morhenny]

Yea I wouldn't mind implementing that, but I will definitely need some help and guidance on how this CloudAnchorNode should be set up and what functions and logic it will need, besides the actual cloud anchor stuff. I also am not quite sure yet how soon I will be able to do that, as I have a big university project at the moment and I'll have to see how much time I have on the side

[grassydragon]

I'm interested in working on the CloudAnchorNode too since I use Cloud Anchors in my application.

[morhenny]

If you are not using depth occlusion, the only place where we acquire images is within the LightEstimation and they are closed after using them:

sceneview-android/arsceneview/src/main/kotlin/io/github/sceneview/ar/arcore/LightEstimation.kt

Line 445 in c7d482d

image.close()

Could you just disable light estimation to see if it could come form here?

I haven't had the original crash after disabling that, but I did have a strange bug today, where the planes and my anchor stopped being tracked all of a sudden. All rendered planes were not displayed anymore and wouldn't reappear until restarting and my objects were directly in front of the camera and moving with it instead of staying anchored.

Since the stacktrace included parts about max images from the ImageReader, I figured I'd put it here in case it really is related:

W/native: W0316 17:46:54.443201   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
A/libc: Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 10487 (n.ar_navigation), pid 10487 (n.ar_navigation)
W/native: W0316 17:46:54.624275   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:54.664820   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:54.726258   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/ImageReader_JNI: Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers
E/dlx: Skipped image because max images have been acquired from ImageReader.
E/dlx: Skipped image because max images have been acquired from ImageReader.
E/dlx: Skipped image because max images have been acquired from ImageReader.
E/dlx: Skipped image because max images have been acquired from ImageReader.
E/dlx: Skipped image because max images have been acquired from ImageReader.
W/native: W0316 17:46:54.870062   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
E/native: E0316 17:46:55.014388   15248 jni_helper.cc:109] Attempt to get JNIEnv* on thread not attached to JVM
W/native: W0316 17:46:55.014658   15248 scoped_global_ref.cc:29] Unable to delete global reference, JNIEnv* missing
E/native: E0316 17:46:55.014835   15248 jni_helper.cc:109] Attempt to get JNIEnv* on thread not attached to JVM
W/native: W0316 17:46:55.014985   15248 scoped_global_ref.cc:29] Unable to delete global reference, JNIEnv* missing
E/native: E0316 17:46:55.015123   15248 jni_helper.cc:109] Attempt to get JNIEnv* on thread not attached to JVM
W/native: W0316 17:46:55.015251   15248 scoped_global_ref.cc:29] Unable to delete global reference, JNIEnv* missing
V/third_party/redwood/infrastructure/jni_common/jni_helper.cc: JniHelper: attached thread (Called from line 149).
V/third_party/redwood/infrastructure/jni_common/jni_helper.cc: ~JniHelper: about to detach thread (Called from line 149).
V/third_party/redwood/infrastructure/jni_common/jni_helper.cc: ~JniHelper: detached thread (Called from line 149).
V/third_party/redwood/infrastructure/jni_common/jni_helper.cc: JniHelper: attached thread (Called from line 21).
V/third_party/redwood/infrastructure/jni_common/jni_helper.cc: ~JniHelper: about to detach thread (Called from line 21).
V/third_party/redwood/infrastructure/jni_common/jni_helper.cc: ~JniHelper: detached thread (Called from line 21).
W/native: W0316 17:46:55.018012   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:55.117394   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:55.202733   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:55.262652   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:55.386204   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:55.474781   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:55.575969   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:55.677998   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
I/native: I0316 17:46:55.702449   15266 motion_analysis_calculator.cc:645] Analyzed frame 838
W/native: W0316 17:46:55.772049   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
W/native: W0316 17:46:55.879000   10810 frame_buffer.cc:157] A pose node at t=139603646094415 could not be created; dropping the image: FAILED_PRECONDITION: No data available.
    === Source Location Trace: ===
    third_party/redwood/perception/pose_manager/latest_pose_tracker.cc:160
    third_party/redwood/perception/pose_manager/pose_manager.cc:116
W/native: W0316 17:46:55.887897   15248 motion_tracking_context.cc:1231] Triangulation measurement buffer is full. Removing the earliest measurement in the buffer.
E/RepeatCaptureReq(0): Capture failed.
E/StartedCameraController(0): Saw capture failed event, failure: android.hardware.camera2.CaptureFailure@2107e56.

[ThomasGorisse]

That's because ARCore is not returning any more Pose from your Anchor. Apparently because of this:
Skipped image because max images have been acquired from ImageReader.
I think it's related to this: #39
Can you try setting the plane render to PlaneRenderMode.RENDER_TOP_MOST?
We are currently working on other issues inside the PlaneRenderer that will be fixed soon.

[morhenny]

That's because ARCore is not returning any more Pose from your Anchor. Apparently because of this: Skipped image because max images have been acquired from ImageReader. I think it's related to this: #39 Can you try setting the plane render to PlaneRenderMode.RENDER_TOP_MOST? We are currently working on other issues inside the PlaneRenderer that will be fixed soon.

I've tried setting `PlaneRenderMode.RENDER_TOP_MOST, but that doesn't really work with my app use-case too well, because I'll always need the ground floor, and with changing that setting, it will quite often just place the 1 plane on top of some other object in my room, like my couch for example.

Since this bug only happened this one time to me and it seems to be very rare, I am not super worried about it.

Is there an easy way to programmatically reset all tracked planes and start over though, then I could let the user just reset if the wrong plane was found in render_top_most?

[grassydragon]

It is an accidentally created bug that was fixed yesterday. Now the first hit plane in the phone camera direction is rendered as it has always been in Sceneform.
You can try using only the plane placement and disabling all other methods like the depth placement to see if the native crashes disappear. I'm also experiencing them with the SceneView sample applications.