AR photo capture on Samsung Android: 3D objects zoomed in when compositing getUserMedia photo with offscreen renderer

[qween666-ux]

Environment

• Needle Engine: 4.15.0
• Device: Samsung Galaxy Android (Chrome)
• WebXR: immersive-ar (ARCore)

Problem

When capturing AR photo on Samsung Android, the 3D objects appear zoomed in compared to the actual AR view.

Camera background (getUserMedia + takePhoto) looks correct, but 3D objects are larger/closer than what the user sees.

What I'm doing

1. getUserMedia + ImageCapture.takePhoto() for camera background (no camera-access feature - causes GPU crash on Samsung)
2. Offscreen WebGLRenderer to render 3D scene separately
3. Composite camera photo + 3D scene on canvas

Logs

• renderer.domElement: 1080 x 2340 (devicePixelRatio: 2.625)
• xrCam.cameras[0] projectionMatrix fovY: 73.9, aspect: 0.462
• mainCamera.fov: 50, aspect: 0.511
• takePhoto() resolution: 3060 x 4080
• WebARSessionRoot scale: [1,1,1], position: [0,0,0]

What I've tried

• Copying xrCam.cameras[0].projectionMatrix directly to snapCam → still zoomed in
• Using mainCamera.fov (50) → still zoomed in
• Using fovY from projectionMatrix (73.9) → zoomed in
• renderer.xr.enabled = false + isPresenting = false before offscreen render → AR disappears, 3D objects gone
• onAfterRender: save projectionMatrix every frame, use in offscreen render → still zoomed in

Question

1. What is the correct way to do offscreen render that matches exactly what the user sees in AR?
2. Is there a way to get camera background without camera-access feature on Samsung (camera-access causes GPU crash)?
3. screenshot2() works but requires camera-access which crashes Samsung - is there an alternative?

Note: camera-access feature causes GPU crash on Samsung Galaxy devices even when just requested in optionalFeatures, without ever calling getCameraImage().

[marwie]

Hi, thanks for the report. Before investigating further could you please update to latest stable version of Needle Engine (5.0.10) and if the error still happens please also try again on the latest alpha version (5.1.x).

Thank you.

[qween666-ux]

Update: After testing more carefully, I notice that not only the 3D objects
but also the camera background appears slightly zoomed in compared to
the actual AR view.

This suggests the issue might be with how I'm cropping the getUserMedia
photo (3060×4080) to match the screen ratio (1080×2340).

Current crop method: cover crop (center crop to screen aspect ratio)
This crops significant portions of the photo, which may cause the zoom effect.

Question: What is the correct way to match the getUserMedia photo FOV
with the actual AR camera view FOV in Needle Engine?

Also: Updated to Needle Engine 5.0.10, same issue persists.

[qween666-ux]

Summary

I need to capture AR screenshots (camera background + 3D objects) on Samsung
Galaxy during an active immersive-ar session, but both available approaches
are blocked:

Approach 1: screenshot2() (official Needle AR screenshot)

• Requires the camera-access feature
• On Samsung Galaxy, requesting camera-access (even just adding it to
  optionalFeatures) causes a GPU crash / Chrome tab crash
• So screenshot2() is unusable on Samsung for our scene

Approach 2: Custom capture (getUserMedia + offscreen renderer)

• I avoid camera-access entirely
• For the camera background: getUserMedia + ImageCapture.takePhoto()
• For the 3D layer: a separate offscreen WebGLRenderer renders the scene
• Problem: the moment getUserMedia is called during the AR session,
  the AR view goes black / disappears (the XR render loop seems to stop)
• Result: only static/clone objects survive in the capture; the main
  animated objects are gone or misplaced

What I confirmed

• A pure WebXR + Three.js app (requiredFeatures: ['local'],
  optionalFeatures: ['dom-overlay']) on the SAME Samsung device CAN call
  getUserMedia + takePhoto during the AR session without the AR view
  disappearing. Capture works perfectly there.
• In Needle Engine, the same getUserMedia call makes the AR view go black.
• I saw the older thread where AR screenshot in Needle returned a black
  image, and that screenshot2 was added in 4.1.0 — but that path needs
  camera-access which crashes Samsung.

Environment

• Needle Engine: 5.0.10 (also tested 4.15.0)
• Device: Samsung Galaxy (Chrome, Android)
• Scene: animated GLB models (palm trees, monkeys, tiger) under an anchor

Questions

1. Why does getUserMedia during the AR session make Needle's AR view go
   black, when a minimal pure-WebXR session on the same device does not?
   Is Needle's session holding the camera/GPU more exclusively, or does
   the XR render loop stop?

2. Is there ANY way to capture the AR view (camera + 3D) on Samsung
   WITHOUT camera-access (since camera-access crashes Samsung GPU)?

3. If camera-access is the only official path, is there a known workaround
   for the Samsung GPU crash when camera-access is enabled?

[marwie]

Hi, thanks for the detailed report! Since you've tested on 4.15.0, could you also try the latest alpha (5.1.x) and let me know if it still happens there?

On the zoom issue itself: this isn't a projection bug you can fix by copying matrices — your getUserMedia photo and the WebXR view are two different cameras. Your own logs show it: the XR view is aspect 0.462, but takePhoto() is 3060x4080 = aspect 0.75, with a different FOV. The photo captures a wider scene than the XR projection renders, so the 3D will always look "zoomed in" relative to it. You don't get the photo's intrinsics back from ImageCapture, so there's no reliable way to register the two. This is exactly why camera-access exists — its raw camera image is guaranteed aligned to the XRView projection.

A few concrete things:

For the 3D layer, don't hand-roll the offscreen render. Copying projectionMatrix isn't enough — you also need the camera's world pose (view matrix) from the same XR frame, plus a matching viewport/aspect. screenshot2({ transparent: true, width, height }) does all of that internally (it calls renderer.xr.updateCamera() with the live frame), so the 3D comes out exactly matching the AR view. And it works without camera-access — it just skips the camera background. So you'd get a correct, transparent 3D layer to composite yourself.

Don't set xr.enabled = false / isPresenting = false mid-frame to render offscreen — that tears down the AR view, which is the "AR disappears, objects gone" you saw. screenshot2 saves and restores that state around the capture instead.

The remaining gap is the background. Even with a perfect 3D layer, your getUserMedia photo won't line up because of the FOV difference. Realistically you either crop the photo down to the XR frustum (approximate, since you're guessing its FOV), or treat the camera-access GPU crash as the real bug to chase — it's the only geometrically-correct path. If you can grab a chrome://gpu crash log from the camera-access attempt, that's the thing to report upstream to Chrome/ARCore.

To summarize: if the camera-access crash is indeed a platform/ARCore bug on these Samsung devices (which is what it looks like), there's nothing we can fix on the engine side — it would need to be reported to Chrome/ARCore. In the meantime the transparent screenshot2() 3D layer is the best workaround, accepting that the getUserMedia background won't perfectly align.

That said, to be sure it's not something on our end, it would help a lot if you could share:

• The exact Samsung model and Android/Chrome versions
• A chrome://gpu crash log (or the console output) from the moment camera-access crashes
• A minimal reproduction project if possible — even just the camera + a single animated object
• Whether the crash also happens with camera-access in a plain three.js/WebXR session (to isolate it from Needle)

With that we can dig in further and confirm whether it's purely platform-side or something we can address.

[qween666-ux]

Huge progress — screenshot2({ transparent: true }) works perfectly!
All animated 3D objects (monkeys, palms, tiger) are captured correctly
and aligned with the AR view. Thank you.

The only remaining issue is the camera background.

After screenshot2 completes, I call getUserMedia + ImageCapture.takePhoto()
to get the camera background. But it ALWAYS fails with:
NotReadableError: Could not start video source

I tried adding delays (300ms after screenshot2) and retrying 3 times —
still NotReadableError every time. It seems the active immersive-ar session
holds the camera exclusively, so getUserMedia can never open it
(consistent with WebXR AR spec issue #87).

Questions:

1. Is there any way to get the camera background frame WITHOUT camera-access
   while the AR session is active? (camera-access crashes our Samsung device)

2. Can I retrieve the XRView's FOV/projection that screenshot2 used internally?
   If I knew the exact FOV of the XR view, I could crop the getUserMedia photo
   to match — but the problem is getUserMedia itself fails during the session.

3. Is the camera-access GPU crash on Samsung the only geometrically-correct
   path? If so, I'll prepare the chrome://gpu crash log to report to Chrome/ARCore
   as you suggested.

Device: [Galaxy S25], Android [16], Chrome [148.0.7778.215]

[marwie]

Awesome, really glad screenshot2({ transparent: true }) sorted out the 3D layer!

On the camera background — short version: unfortunately no, there's no way around this one, and you've diagnosed it correctly.

Q1 — camera background without camera-access during the session?
No. While an immersive-ar session is active, ARCore holds the device camera exclusively, so getUserMedia will always fail with NotReadableError — that's the platform behavior (WebXR issue #87), not something Needle can work around. The only API that can read the AR camera feed during a live session is WebXR raw camera-access (XRWebGLBinding.getCameraImage / XRView.camera) — which is exactly what screenshot2 uses internally for the background. So without camera-access there's genuinely no way to get the live AR background frame. The only alternative would be to drop immersive-ar passthrough entirely and run your own getUserMedia

Q2 — can you read the XRView FOV/projection?
Yes, that part is easy. From the live XR frame:

const pose = frame.getViewerPose(referenceSpace);
const proj = pose.views[0].projectionMatrix; // column-major Float32Array
const fovY = 2 * Math.atan(1 / proj[5]);
const aspect = proj[5] / proj[0];

(or renderer.xr.getCamera().cameras[0].projectionMatrix for the same thing). But as you noted, this is moot as long as getUserMedia itself fails during the session — there's no photo to crop.

Q3 — is camera-access the only geometrically-correct path?
Yes. It's the only path that gives you a camera image aligned to the XRView projection and is available during the session. So if it crashes on your S25, that's the real bug to chase. Please do prepare the chrome://gpu crash log — and if you can, a minimal repro (just camera-access in optionalFeatures + a single object) makes it much easier to file a solid report with Chrome/ARCore. If you share both here I'm happy to help confirm it's purely platform-side before it goes upstream.