Fixing VR

[avaer]

Webaverse has a VR implementation with full avatars support, but it needs to be fixed.

This is a list of the main parts of doing that:

VR rendering

Add enter VR mode and make sure performance is smooth.

Session setup code - this should be refactored into a separate xr.js file.

We may need changes in the new post processing stack when dealing with frames:

app/post-processing.js

Line 198 in e2b7c05

class PostProcessing extends EventTarget {

Here is the VR enter button in React, which should be hooked up:

app/src/components/general/action-menu/ActionMenu.jsx

Line 76 in e2b7c05

<div className={ classnames( styles.btn, styles.vr, xrSupported ? null : styles.disabled ) } onClick={ handleVRBtnClick } >

Avatar rigging

We have IK for avatars that needs to be hooked up and fixed.

Old rig input injection code, should probably be moved to new xr.js.

The Avatar class takes input values in order to animate the avatar, and that input can be sourced from WebXR pose:

app/avatars/avatars.js

Line 659 in e2b7c05

this.sdkInputs = {

Top/bottom enabled for the avatar determines whether we apply IK:

app/avatars/avatars.js

Line 877 in e2b7c05

this.setTopEnabled(!!options.top);

app/avatars/vrarmik/ShoulderPoser.js

Line 90 in e2b7c05

this.updateHips();

app/avatars/vrarmik/ShoulderTransforms.js

Line 74 in e2b7c05

this.shoulderPoser.Update(this.enabled, this.handsEnabled[0], this.handsEnabled[1]);

This will need a lot of trial and error and debugging.

Voice input

Headset voice I/O should also be working, and animate the mouth. We already support this for normal microphone so it might just need to be tested:

app/avatars/avatars.js

Line 1919 in e2b7c05

this.volume = this.volume*0.8 + e.data*0.2;

app/avatars/avatars.js

Line 1650 in e2b7c05

morphTargetInfluences[aIndex] = volumeValue;

Movement mechanics

We need to set up abstract VRChat-style IO (game controllers) for VR, with walk and jump support.

VR UI

We need basic VR UI, for at least enabling/disabling the microphone, similar to VRChat.

Ideally in the future we can use the full React UI in-world with WebXR layers.

Items integration

Webaverse supports wearing and using many different objects in the world.

Wear code

The API is simple. One key to activate an object, and the object depends on its world transform to perform the action. So most of our objects are already VR-compatible.

We should fix the basic objects like pistols and sword to work with the VR rig in a unified way.

Debugging

We should make sure the kinematics and animations are smooth+accurate for various avatars in VR.

Optimization

This should not be dropping frames during gameplay. Other systems, like loading, may need to be optimized.

It is possible for us to rewrite parts of this in WASM, since we already have a build pipeline for that: https://github.com/webaverse/app-wasm.