Utkarsh Tripathi edited this page Aug 14, 2018 · 7 revisions


In this project, ViSP and Unity are integrated to create Augmented Reality (AR) application. ViSP (Visual Servoing Platform) is a cross-platform library that provides real-time visual tracking and visual servoing implementation useful in robotics, computer vision, augmented reality and computer animation. Pose estimation or tracking algorithms in ViSP have the potential to benefit AR community and could be used for AR applications in general.

Unity is a cross-platform 3D game engine to build high quality games. Unity engine provides a framework to create/add/manipulate 3D scenes consisting of camera, lighting and 3D objects.

ViSP offers several methods to track and estimate the pose of the camera. Basic methods intend to estimate the camera pose from 2D/3D point correspondences. 2D point coordinates are obtained by detecting and tracking fiducial markers, for instance by tracking blobs or by detecting corners of an AprilTag. Advanced methods rely on the knowledge of the CAD model of the object to track. The model-based tracker allows to track and estimate the camera pose of a markerless object using multiple types of primitives (edges, texture or both).


This project aims to place arbitrary objects at specific places within the virtual-camera field of view which marked using AprilTags or markerless using model-based tracking approaches. While Unity supports creation of virtual scenes, it does not have libraries to perform computer vision tasks such as pose estimation and tracking. However, Unity allows integration of 3rd party libraries in the form of DLLs (dynamically linked libraries) which can be imported into Unity as Assets. So, it is possible to export ViSP library functions as DLL and import it in Unity. This combination of Unity and ViSP creates a highly capable AR system.

A scene created using Unity game engine consists of cameras, lighting and game objects. Unity allows creation and import of assets that influence the visual and behavioral aspects of game objects. A game object’s behavior can be programmed using a C# script. Unity also allows import of DLLs which can be called by C# behavior scripts to extend the functionality of Unity SDK. ViSP exports the pose estimation and tracking functionality to the Unity scripts associated with game objects.

Above figure shows the schematic of interoperability between ViSP and Unity. ViSP is implemented in C++ and is therefore unmanaged while the behaviour scripts are in managed C# environment. Therefore, implementation of exported API must take the following into consideration:

  • Data type interoperability: Data types of parameters and return values of API functions must be same in both environments (C# and C++).
  • Garbage collection: Any object created dynamically in C++ will not free the memory until explicitly freed. Therefore, API functions in C++ must also export suitable destructors which can be called by C# script once the use of required methods is complete.


Start by looking at this tutorial which explains how one can generate DLLs to be used in Unity Project using an example.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.