Video playback stops when entering immersive mode in Vision Pro

[dmarcos]

Tested on https://aframe.io/aframe/examples/test/video/

The modal in this example generates the user gesture event that triggers video play back as expected while in 2D mode.

When entering immersive mode the video stops playing. On Meta Browser (Quest) the video playback continues as expected.

[Clodo76]

I confirm this issue, i have a VisionPro. There are different kind of issues in many official examples,
if the owner of this project need a feedback, i can report a list.
And maybe the main website need a "howto in VisionPro", because WebXR is not active by default (Experimental, extra-flags etc in Settings > App > Safari).

[dmarcos]

@Clodo76 Feel free to open additional issues if you found other bugs. Thanks

[dmarcos]

cc @cabanier @AdaRoseCannon I think Meta browser behavior is the correct one and my hunch is that Safari's one not deliberate but not 100% sure.

[AdaRoseCannon]

This is a known bug sorry. You can start the audio again by resuming an audio context after the session starts.

Although I would prefer for this not to be fixed on the website side and to wait for a fix but I understand if you need to work around.

[dmarcos]

@AdaRoseCannon Thanks so much for the info and the workaround. For A-Frame core I think we will wait for a fix. Not sure if there's any public info but I imagine Safari updates tied to Vision OS releases? Having an approximate time horizon / update cadence will help determine when workarounds are worth vs waiting for a fix. Thanks so much for the help

[Clodo76]

@Clodo76 Feel free to open additional issues if you found other bugs. Thanks

I made a stylesheet here, i hope can be useful:
https://docs.google.com/spreadsheets/d/1Bf8A8x52NlAaI6esGeg2Nd9k_5wBLYuMyorKnPfjTPU/edit#gid=0
(but i don't yet perform Quest3 tests).
In VisionPro i tested aframe examples from github, aframe-stereo-component, and general webxr-samples.
I'm to newbie about aframe to understand what single issues are real or can be easy fixed, so for now i don't open specific issues and i reply here (i understand maybe offtopic based on this issue title, but are VisionPro issues).

About video (flat or 360 doesn't matter), when i start playback, are played in the browser.
When i click "Enter VR", VisionPro ask acknowledge for immersion,
when immersion start i see the video still playback, but stop after around a second.

I don't report the colors issue.

An example "pinch to click" for VisionPro can be integrated to fix a lots of unit-cases.

I never find yet a webxr AR example working.
Note that VisionPro/Safari show GLTF/GLB in AR space with the < model > html tag, not related to webxr from what i know.

[Clodo76]

Note: i made an example with video&audio 360 playback, toggle play/pause with click OR spacebar key.
Bluetooth keyboard connected to VisionPro.
Load webpage, hit space, video start. "Enter VR" stop it.
If i hit spacebar to toggle pause/play state, audio works as expected, video NEVER. If i close VR mode, video is ok in browser.
So, seem a texture-update issue, not a "VisionPro automatically stop video playback when enter in VR mode".

[psql]

Folks this sounds insane, but I found one site that makes this work.
You have to start the playback in immersive mode, but I can confirm it's functional.

Caveat: the journey to the solution is for the brave. It's an adult website... lmao. Tread carefully.
https://twitter.com/okpasquale/status/1761038091295371503

[dmarcos]

@psql Thanks. This is a known bug on Safari's side.

@AdaRoseCannon suggested a workaround You can start the audio again by resuming an audio context after the session starts but we will wait for a browser fix.

[dmarcos]

@Clodo76 Please file independent issues for any individual problems you encounter. This one is just for the issues concerning video playback stopping when entering immersive mode on Vision Pro. Thanks

FYI, Safari doesn't offer WebXR AR mode yet (immersive-ar). Not a bug but a deliberate decision by Apple

Also A-Frame issues are for bugs in A-Frame core. For 3rd party components please report in their respective repositories. Thanks

There are no issues with texture update to my knowledge. This is confirmed as a Safari bug. Quest browser works as expected

[hybridherbst]

In case it helps someone, this seems to be enough as a workaround in three.js-based apps/engines:
As an example, audio works here now on AVP: https://prefrontalcortex.de/digivid/12/

const ctx = AudioContext.getContext();
ctx.addEventListener ("statechange", async () => {
    if (ctx.state === "suspended" || ctx.state === "interrupted") {
        ctx.resume().then(() => {
            console. log("AudioContext resumed");
        }).catch((err) => {
            console.error ("AudioContext couldn't be resumed", err);
        });
    }
});

[danrossi]

In the webxr session start event, forcing video playback works. Now need to detect for visionOS to do that.

[dmarcos]

Don't have to detect VisionOS but a fix would require code only needed for Vision Pro. For now you can apply workaround described above. On A-Frame side we will wait for a browser fix.

[danrossi]

Pseudo code. platform detected goes between Ipad and MacIntel. Maybe just a straight up play is enough but I've tested this.

onSessionStart() {
          //force play for visionOS
          if (this.isIpad) {
           // this.video.play();

            const handlePaused = () => {
                this.video.removeEventListener("pause", handlePaused);
                this.video.play();
            };

            this.video.addEventListener("pause", handlePaused);


        }
}

[jparismorgan]

@danrossi Hmm, maybe something changed, but the onSessionStart() fix doesn't work for me in the Simulator running visionOS 1.1. I put together a repro using the three.js webxr_vr_video example: https://github.com/jparismorgan/threejs-webxr-vision-pro-video/tree/main

Would be curious if this repro works for anyone else, or if anyone knows what to update it with to get it working? Happy to upstream it as well to fix the three.js example.

webxr_repro.mp4

@hybridherbst, is this fix for the video playback issue? Or just audio? I tried adding this but didn't see any difference:

const ctx = new AudioContext(); // NOTE(paris): this instead of const ctx = AudioContext.getContext(); b/c otherwise we crash with ` 'AudioContext.getContext' is undefined)`.
ctx.addEventListener ("statechange", async () => {
	console.log("state changed: ", ctx.state);
		if (ctx.state === "suspended" || ctx.state === "interrupted") {
				ctx.resume().then(() => {
						console.log("AudioContext resumed");
				}).catch((err) => {
						console.error ("AudioContext couldn't be resumed", err);
				});
		}
});

Thanks for any help!

[danrossi]

I run all my tests under https to be able to launch webxr from my server on another machine. So try that. I'm trying to get my VR picture in picture hack working for Safari then will release my visionOS changes including transient-pointer controls support.

http-server -p 8084 -S -C /etc/letsencrypt/live/rtc.electroteque.org/cert.pem -K  /etc/letsencrypt/live/rtc.electroteque.org/privkey.pem  ./demo

[jparismorgan]

Thanks, put it up on Vercel and tested with https, but same error: https://threejs-webxr-vision-pro-video.vercel.app/

[cgauld]

The AudioContext fix has unfortunately not helped for us - we have no success with audio or video once immersive mode starts. We've filed a bug on the WebKit Bugzilla so that it's in the system :-)
https://bugs.webkit.org/show_bug.cgi?id=274385

[dmarcos]

@cgauld can you share code with the workaround?

[cgauld]

Hi @dmarcos unfortunately we don't have a working workaround :-( We had tried the code in the comment above, plus a few others, and nothing was successful.

[dmarcos]

@cgauld Can you share the code of your "not working" workaround with https://glitch.com/~aframe ?

[hoangman1108]

  const mesh = new THREE.Mesh(new THREE.BoxGeometry(1, 1, 1), new THREE.MeshBasicMaterial({ visible: false }));
  const meshAudioBg = new THREE.Mesh(
    new THREE.BoxGeometry(1, 1, 1),
    new THREE.MeshBasicMaterial({ visible: false }),
  );
  mesh.add(sound);
  meshAudioBg.add(soundBg);
  scene.add(mesh);
  scene.add(meshAudioBg);
  
  add sound to mesh and apply to scene it working 

[secondpathstudio]

Any updates if this has been fixed with VisionOS 2.0? Still struggling to get this to work.

[AdaRoseCannon]

Please can you make a demo that demonstrates the issue and file a report using feedback assistant including the URL to the demo.

[dmarcos]

@secondpathstudio Provide simple code with https://glitch.com/~aframe to reproduce the issue or it will be hard to help otherwise. Thanks