Virtual Vision is a computer vision project whose goal is to improve the monitoring of complex and vast environments using multiple technologies such as deep learning, virtual reality and 3D scanning. This projects allows one to :
- Scan and obtain a 3D model of the environment to monitor
- Watch and move within the virtual world (the 3D model), using a virtual reality headset and a controller
- Detect and integrate dynamic objects such as humans, animals or vehicles within the virtual world, at their real location
This project is currently being developed using the following libraries and devices :
@article{Zhou2018,
author = {Qian-Yi Zhou and Jaesik Park and Vladlen Koltun},
title = {{Open3D}: {A} Modern Library for {3D} Data Processing},
journal = {arXiv:1801.09847},
year = {2018},
}
- Dataset acquisition using an Intel RealSense D435
- Dataset importation
- Dataset pre-Processing
- Shard generation
- Shard linkage
- Model cleaning/post-Processing
- Creating a Client / Server interface
- Enhancing the deployment of the server application
- Automatic alignment and calibration of the depth camera with the room's 3D model
- Detection of real dynamic objects (humans, animals, vehicles, ...)
- 3D localization of the detected objects
- Integration of the objects within the virtual world
- Integration of the VR headset
- Development of the hardware
- Interfacing the hardware with Unity
- Enhancing the ergonomy
usage: Launcher.py [-h] --address ADDRESS --port PORT [--workspace WORKSPACE]
Virtual Vision v0.1 - Cloud
optional arguments:
-h, --help show this help message and exit
--address ADDRESS Server's IP address
--port PORT Server's port
--workspace WORKSPACE
Path of the workspace, where the datasets will be
saved. Default : 'USER/Workspace'
usage: Virtual_Vision.py [-h]
{scan,reconstruct,cloud,calibration,detection} ...
Virtual Vision v0.1 - Client
optional arguments:
-h, --help show this help message and exit
Operating Mode:
{scan,reconstruct,cloud,calibration,detection}
Scan, Reconstruct or Cloud
scan Scanning mode
reconstruct Reconstruction mode
cloud Cloud based interaction and processing
calibration Depth camera calibration for real time 3D integration.
The arguments can be directly set in config.json
detection Real time detection and 3D integration. The arguments
can be directly set in config.json
usage: Virtual_Vision.py scan [-h] [--nsec NSEC] [--workspace WORKSPACE]
[--sharpening SHARPENING]
[--autoreconstruct AUTORECONSTRUCT]
[--depthanalysis DEPTHANALYSIS]
[--sconfig SCONFIG] [--fps FPS] [--width WIDTH]
[--height HEIGHT] [--vpreset VPRESET]
[--laserpower LASERPOWER] [--exposure EXPOSURE]
[--gain GAIN]
optional arguments:
-h, --help show this help message and exit
--nsec NSEC Scan duration in seconds (0 for unlimited, press Q to
quit). Default : 0
--workspace WORKSPACE
Path of the workspace, where the dataset will be
saved. Default : 'USER/Desktop/Workspace'
--sharpening SHARPENING
Allows to sharpen the images in order to reduce the
impact of the motion blur. Default : 0
--autoreconstruct AUTORECONSTRUCT
Automatically reconstruct the dataset after the scan.
Default : 1
--depthanalysis DEPTHANALYSIS
Process the dataset in order to estimate the coverage
of the depth frames. Default : 0
Automatic Configuration (JSON importation):
--sconfig SCONFIG Import a JSON RealSense configuration file (instead of
Manual Settings, can be generated using the RealSense
SDK) : path
Manual Settings:
--fps FPS Framerate of the capture (higher can reduce motion
blur). Default : 60
--width WIDTH Width of the captured frames. Default : 848
--height HEIGHT Height of the captured frames. Default : 480
--vpreset VPRESET Allows to choose some preset settings provided by the
RealSense SDK (0 -> Custom, 1 -> Default, 2 -> Hand, 3
-> High Accuracy, 4 -> High Density, 5 -> Medium
Density). Default 0 (custom)
--laserpower LASERPOWER
RealSense laser power. Default : 240
--exposure EXPOSURE Exposure time of the RealSense sensor. Default : 3200
--gain GAIN RealSense sensor gain. Default : 16
usage: Virtual_Vision.py reconstruct [-h] --rconfig RCONFIG
optional arguments:
-h, --help show this help message and exit
--rconfig RCONFIG Import a JSON dataset reconstruction settings file : path
usage: Virtual_Vision.py cloud [-h] [--address ADDRESS] [--port PORT]
[--reconstruct RECONSTRUCT] [--list LIST]
[--remove REMOVE] [--getresult GETRESULT]
[--mergeshards MERGESHARDS]
optional arguments:
-h, --help show this help message and exit
--address ADDRESS Cloud server's IP address.
--port PORT Cloud server's port.
--reconstruct RECONSTRUCT
Cloud based reconstruction, using the specified
reconstruction settings file : path
--list LIST Displays the name of the dataset present on the cloud.
--remove REMOVE Removes one dataset present on the cloud by specifying
its name
--getresult GETRESULT
Downloads the point cloud of the specified
reconstructed dataset (name)
--mergeshards MERGESHARDS
Merges the shards of the specified dataset (name).
Works only if the reconstruction ended before
successfully merging the shards
usage: Virtual_Vision.py calibration [-h]
optional arguments:
-h, --help show this help message and exit
The variables can be directly tweaked using the config.json file.
usage: Virtual_Vision.py detection [-h]
optional arguments:
-h, --help show this help message and exit
The variables can be directly tweaked using the config.json file.
Currently, Virtual-Vision allows to :
- Record a dataset using an Intel RealSense and the following command :
python Virtual_Vision.py scan
- Reconstruct a dataset by using a dataset JSON configuration file and the following command :
python Virtual_Vision.py reconstruct --rconfig PATH_OF_CONFIG_FILE
- Launch the cloud server by using the following command :
python Launcher.py --address ADDRESS --port PORT
- Interact with the cloud server by using the default config.json file (no argument speficied) or the server's address and port, in addition to a command (see below)
python Virtual_Vision.py cloud --address ADDRESS --port PORT --COMMAND
python Virtual_Vision.py cloud --COMMAND
- Reconstruct a dataset on the cloud using the dataset JSON configuration file and the following command :
python Virtual_Vision.py cloud --reconstruct PATH_OF_CONFIG_FILE
- List all the datasets on the cloud using the following command :
python Virtual_Vision.py cloud --list 1
- Download the result (assembled point cloud) of a dataset by using the following command :
python Virtual_Vision.py cloud --getresult DATASET_NAME
- Remove a dataset from the cloud's workspace by using the following command :
python Virtual_Vision.py cloud --remove DATASET_NAME
- Merge the shards of one dataset (useful if the reconstruction process stopped before the shard assembling as it allows to skip the shard geenration process) by using the following command :
python Virtual_Vision.py cloud --mergeshards DATASET_NAME
- Calibrate the camera by calculating the 3D transformation between the camera's point of and the "World" point cloud, by using the following command :
python Virtual_Vision.py calibrate
- Launch the real time detection and 3D integration by using the following command :
python Virtual_Vision.py detection