Enva-XR - Environment Aware Augmented Reality

• Web-based framework for environment-aware rendering and interaction in augmented reality based on WebXR using three.js
• The framework handles geometry occlusions, matches the lighting of the environment, casts shadows, and provides physics interaction with real-world objects.
• Material from three.js can be reused the shader code required for AR occlusion is injected into the existing shaders using the onBeforeCompile callback.
• Capable of obtaining over 20 frames per second even on middle-range devices.
• WebXR AR features supported by the framework
  • Lighting Estimation
  • Depth Sensing Module
  • Hit Test
  • DOM Overlay
  • Anchors
  • Plane Detection

Getting Started

• Download the repository from git gh repo clone tentone/enva-xr
• Install Node and NPM.
• Ensure that your browser is compatible with WebXR and check features support (e.g. depth-estimation, hit-test, lighting).
• Install dependencies from NPM by running npm install and start the code running npm run start
• To allow easier debugging in mobile devices eruda can be used as a development tools alternative.

Usage Example

• Bellow is a simple usage example of the library the ARRenderer is responsible for most of the work required to setup the AR scene.
• The ARRenderer receives a configuration object that indicates wich WebXR features should be enabled.
• To enable AR rendering on existing three.js materials the AugmentedMaterial.transform() method should be used to transform regular materials into AR materials.
• The example bellow demonstrates how to create a simple AR scene with occlusion and lighting enabled.
• LightProbe object replicates the envornment lighting and position main light source position and direction. Internaly contains a three.js LightProbe and DirectionalLight with shadow casting enabled by default.

const renderer = new ARRenderer({
  depthSensing: true,
  depthTexture: true,
  lightProbe: true
});

let material: any = new MeshPhysicalMaterial({color: (Math.random() * 0xFFFFFF)});
material = AugmentedMaterial.transform(material);

let box = new Mesh(new BoxGeometry(), material);
box.receiveShadow = true;
box.castShadow = true;
renderer.scene.add(box);

const probe = new LightProbe();
renderer.scene.add(probe);

const floor = new FloorPlane();
renderer.scene.add(floor);

renderer.onFrame = function(time: number, renderer: ARRenderer) {
  box.rotation.x += 0.01;
};

renderer.start();

Rendering

• Depth data provided by WebXR can be used for occlusion in the 3D scene.
• Occlusion is calculated in the shader code injected using the AugmentedMaterial.transform() method.
• To enable realistic rendering of the scene the MeshPhysicalMaterial material should be used alonside PBR assets.

Physics

• cannon.js can be used for physics interaction between objects.
• The VoxelEnvironment provides a probabilistic voxel based model that maps the environment from depth data that is updated every frame.
• Alternativelly physics can rely on plane detection using the FloorPlane or Planes objects.
• Currently performance is limited might be improved using WebXR Real World Geometry API

License

• The code from the project is MIT licensed. The license is available on the project repository,