OpenARK is an open-source wearable augmented reality (AR) system founded at UC Berkeley in 2016. The C++ based software offers innovative core functionalities to power a wide range of off-the-shelf AR components, including see-through glasses, depth cameras, and IMUs.
C++ C# CMake C
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
cmake_modules added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
documentation added Linux build instructions, updated Windows build instructions; Mar 11, 2018
include update visualizeDepthMap function Aug 7, 2018
samplecode added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
svm added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
unity added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
.gitignore added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
CMakeLists.txt added Linux build instructions, updated Windows build instructions; Mar 11, 2018
Calibration.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
Converter.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
DepthCamera.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
Detector.cpp 0.9.3 Mar 1, 2018
FrameObject.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
FramePlane.cpp 0.9.3 Mar 1, 2018
Hand.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
HandClassifier.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
HandDetector.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
LICENSE.md Update License Owner Mar 20, 2017
PMDCamera.cpp 0.9.3 Mar 1, 2018
PlaneDetector.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
README.md fixed README links Mar 11, 2018
RGBCamera.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
RS2Camera.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
RT_Transform.txt beta 0.8 initial commit Feb 26, 2017
SR300Camera.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
StreamingAverager.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
Util.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
Version.h.in added Linux build instructions, updated Windows build instructions; Mar 11, 2018
Visualizer.cpp Update Visualizer.cpp Aug 7, 2018
Webcam.cpp added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018
doxygen.config Improved clustering and hand detection, 87%+ on tests (97% on first) Nov 21, 2017
main.cpp Update main.cpp Aug 7, 2018
stdafx.cpp Integrated SVM into project; Added precompiled header & modified Nov 18, 2017
stdafx.h added Linux (GCC) build support; added Unity plugin; added RealSense … Mar 11, 2018

README.md

OpenARK

OpenARK is an open-source wearable augmented reality (AR) system founded at UC Berkeley in 2016. The C++ based software offers innovative core functionalities to power a wide range of off-the-shelf AR components, including see-through glasses, depth cameras, and IMUs. The open-source platform includes fundamental tools such as AR-based camera calibration and SLAM, and it also includes higher-level modules to aid human-computer interaction, such as 3D gesture recognition, plane detection, and multi-user collaboration, all with real-time (30+ FPS) performance. Currently, it supports both PMD Pico Flexx and Intel RealSense SR300 cameras. The project builds natively on both Windows and Linux.

At a Glance

  • Technology stack: C++, OpenCV, PCL, Boost, Intel RealSense SDK (1 or 2) / PMD SDK
  • Status: Beta 0.9.3
  • Application Demo: Vimeo

Dependencies

Hardware

  • Depth Camera
  • RGB Camera
  • Transparent AR Glasses (optional)

Software

  • OpenCV 3.2.0+
  • PCL 1.8
  • Boost 1.6.4

Installation

Building From Scratch

Instructions are available in the following documents:

Prebuilt Binaries (For Windows)

Prebuilt binaries for 64-bit Windows are available here. Both the static library (with the headers) and the demo program are included. For the SVM to work properly, the svm/ folder must be present in the current directory OR under the directory pointed to by the OPENARK_DIR environment variable.

Usage

To use OpenARK in a Visual Studio C++ project:

After setting up all the dependencies (OpenCV, PCL, etc.) according the configuration instructions, add openark_x_x_x.lib to Project > MyProjectProperties > Linker > Input > Additional Dependencies. Then add OPENARK_DIR/include to C/C++ > General > Additional Include Directories. Finally, make sure that under C/C++ > Code Generation, Runtime Library is set to Multi-threaded DLL (/MD).

Now you can #include OpenARK's core header in any file and begin using OpenARK. You would probably also want to include one of the depth camera backend headers (e.g. SR300Camera.h).

Here is the outline of a program for performing hand detection:

...
#include "core.h"
#include "SR300Camera.h"

#include "opencv2/core.hpp"
#include <vector>
...
int main() {
    ark::DepthCamera & camera = ark::SR300Camera(); // OpenARK camera backend
    ark::HandDetector detector(); // OpenARK hand detector; also see PlaneDetector
    
    // start the camera; alternatively, call nextFrame() manually inside the loop (slower)
    camera.beginCapture(); 
    ...
    while (true) {
        cv::imshow("XYZ Map", camera.getXYZMap());

        detector.update(camera);
        std::vector<ark::Hand::Ptr> hands = detector.getHands();
        
        // do something with the hands detected
        ...
        
        // quit when q is pressed
        if (cv::waitKey(1) == 'q') break;
    }
    
    ...
    // automatically stops capture on exit
}

Additional sample code is available in samplecode/.

Customization

OpenARK is made for easy customization. Please feel free to build on top of this platform to fit your needs.

How to test the software

Code used to run the demo video is included in main.cpp. Additional sample code can be found in /samplecode/ and you would need to replace it with the main that comes with the project solution.

Known issues

OpenCV prior to 3.2.0 does not offer prebuilt VC14+ binaries. Running VC12 OpenCV binaries with VC14 will result in memories errors in findCountours(). If you are using VC12+ to compile OpenARK, you will need to use CMake to rebuilt OpenCV from source. We have used OpenCV 3.2.0 which comes with OpenCV binaries for VC14 to avoid this problem when using Visual Studio 2015.

Getting help

If you have questions, concerns, bug reports, etc, please file an issue in this repository's Issue Tracker.

Getting involved

The Center for Augmented Cognition welcomes interested industry partners to join our alliance to support the OpenARK platform. More information can be found on cac.berkeley.edu


Credits and references

Bill Zhou, Allen Y. Yang, S. Shankar Sastry, Will Huang, Larry Yang, Eric Nguyen, Michelli Ni, Peter Li, Jessica Jiang, Mona Jalal, Joseph Menke, Lawrence Chen, Kuan Lu, Rachel Lee, Justin Yang, Alex Yu.