Barracuda (documentation) is a lightweight cross-platform library for neural network inference.
Unity_Detection2AR is a great (and one a very few) example for using Barracuda. You can use Detection2AR_apk.apk to run the simulation.
They make use of arfoundation-samples, which demonstrates the functionnalities of the ARFoundation package, allowing for AR functionnalities.
Here is how deep learning networks are added into Unity through Barracuda :
The model needs to be converted into ONNX or Barracuda format. These two formats support different operators.
Instructions here to convert from Pytorch, TensorFlow or Keras to ONNX.
Instructions here to convert TensorFlow models to Barracuda format.
If the network has the right format, it will appear as follow :
The detection happens in the script PhoneARCamera
With the following Plugins, I was able to run the examples but I was not able to integrate ARFoundation in order to put AR augmentations with ARFoundation to the scenes.
MediaPipeUnityPlugin implements the following Mediapipe functionnalities :
- Face Detection
- Face Mesh
- Iris
- Hands
- Pose
- Holistic (with iris)
- Hair Segmentation
- Object Detection
- Box Tracking
- Instant Motion Tracking
- Objectron
It does not recognize my camera on the phone and I wan not able to add any AR functionalities.
tf-lite-unity-sample the following functionalities into Unity :
- TensorFlow
- MNIST
- SSD Object Detection
- DeepLab
- PoseNet
- Style Transfer
- Text Classification
- Bert Question and Answer
- Super Resolution
- MediaPipe
- Hand Tracking
- Blaze Face
- Face Mesh
- Blaze Pose (Upper body)
- MLKit
- Blaze Pose (Full body)
- Meet Segmentation
Follow https://answers.unity.com/questions/1320966/android-debug-usb.html
adb devices -l
adb logcat -s Unity DEBUG
We evaluated two real-time monocular depth estimation networks: MobilePyDnet and FastDepth. Monocular because we cannot expect all smartphones to have more sensors than at least one camera. These networks work particularly well indoors.
Please use view3Ddepth.py to create the pointcloud that can later be vizualized in MeshLab.
MobilePyDnet is based on the PyDnet architecture, and trained with knowledge distillation of MiDaS.
To get a single inference, use their single_inference folder. It is in TensorFlow 1. You do not need to migrate it to TensorFlow 2, just create a virtual environment.
To use FastDepth, also create a virtual environment.
Data_augmentation/Distortions.ipynb gathers distortions from CollabAR (paper) and imgaug.augmenters.