Project XR Glasses

Description

Select the best glasses is time consuming and need to physically search in different shops.

Here we implement a Real-Time Virtual Reality App for Android Devices that helps people to test a different variety of glasses without the need to physically test them.

Hardware & Software details

Developed in: Ubuntu 20.04 LTS
Unity version: 2020.3.20f1
Unity Packages:
ProBuilder
AR Foundation
AR Core XR Plugin
ARKit Face Tracking
ARKit XR Plugin
Intended Devices:
Android Smartphones
Versions >= 8.1.0 Oreo
Tested on:
Samsung Galaxy A32 (Android 11)
Samsung Galaxy S10 (Android 11)

Wireframes

Flowchart

Pseudo code:

start face detection
start landmark extraction
start glasses models

define exit
define next
define save

loop forever:
    if exit==0:
        PASS
    if exit==1:
        STOP
    if next==1:
        select the next glasses model
    if next==0:
        select the same glasses model
    if save=0:
        PASS
    if save=1:
        save model
        STOP
    read the facial landmarks
    update glasses position and direction
    show the glasses in front of face             

Updates

Update 23 Nov 2021:

Face detection and extraction of the most relevant points of the face were successfully implemented. The AR Foundation and ARKit Face Tracking packages were used for this purpose. The main challenges faced were: compatibility with android devices and successfully configuring the building options.

Update 30 Nov 2021:

Different lens models were collected in .obj format, but were converted to .prefab by creating a prefab object and then pulling the .obj lenses into that object. In addition, we tried to instantiate between a prefab object and a landmark point of the face, but we had problems with finding the indicated references.

Update 07 Dec 2021:

Successfully performed the attachment of a reference point of the face, and the placement of the lenses. This was done in several stages, in the first tests a prefab object was instantiated, then the position of the eyes was extracted and the position and rotation transformations were instantiated to that prefab object.

Demos

Demo on a Samsung Galaxy S10 (Android 11):

               

Using this repository

1. First, download or clone this repository.

2. Open Unity3D, and clic on "Add" in order to add the downloaded project

3. Go to "Settings Build and Run", then select "Android" and then "Switch Platform".

4. Install the packages: AR Fundation, AR Core XR Plugin, ARKit Face Tracking, ARKit XR Plugin.

5. Then you can moddify the project or Build and Run on your Android smartphone.

6. That's all, enjoy! :)

Group members

• Maikel BASTAWROUS (@MrMazagngy)
• Yao SINAN (@iyfyao)
• Dennis NUNEZ FERNANDEZ (@dennishnf)

References

• AR Foundation Framework - Main Website
• Setting Up AR Foundation
• ARKit Face Tracking
• Unity AR Foundation Tutorial - Getting Started with Face Tracking
• Unity3d ARKit Face Tracking and placing face game objects with anchor data
• Face Tracking using ARKit and AR Foundation with Unity
• ARKit 3 Face Tracking Tutorial
• ARKit And ARCore - How To Dynamically Select Face Masks