Skip to content

OpenARK is an open-source wearable augmented reality (AR) system founded at UC Berkeley in 2016. The C++ based software offers innovative core functionalities to power a wide range of off-the-shelf AR components, including see-through glasses, depth cameras, and IMUs.

License

augcog/OpenARK

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

* delete deprecated file related to DepthCamera class

* prepare new structure with src & include/openark

* change file structure & move things to src

* put cached file back to src & include

* AzureKinect add old file & define new API & partial implementation

* reorganize_v1, still some files in src

* move src files to util, move webcam to camera

* reorganize some file & add comments

* finialize k4a api

* reorganize include files

* modify CMakeLists

* Clean up D435iCalibWriter.h

* Avatar.h cleaned

* OkvisSLAMSystem cleaned

* Concurrency.h cleaned

* CorrespondenceRansac.h cleaned

* glfwManager.h cleaned

* HumanDetector cleaned

* PointCostSolver cleaned

* PoseGraphSolver cleaned

* SegmentedMesh cleaned

* SingleConsumerPriorityQueue cleaned

* SLAMsystem cleaned

* SpareseMap cleaned

* Types.h

* modify headers of hand and avatar related files

* fix compiling, runnable

* update compile suppport

* update camera calss

* minor bug fix on include

* fix k4a camera bug & show hand demo work

* fix compilation bugs

* change include order

* change include order on cpp file

* add doc on setup k4a

* Fix compile error in Avatar.cpp & HumanDectector.cpp

* Fix avatar demo

* Revert "Fix avatar demo"

This reverts commit 94db710.

* update document

* add FAQ

* Avatar demo tested

Avatar demo tested & good to go

* Update CorrespondenceRansac.h

* Update CMakeLists.txt

* Update CorrespondenceRansac.h

* Update PointCostSolver.h

* Update PoseGraphSolver.h

* Update SingleConsumerPriorityQueue.h

* Update concurrency.h

* Update SparseMap.h

* Update D435iCamera.cpp

* Update PoseGraphSolver.cpp

* Update SingleConsumerPriorityQueue.cpp

* Update SparseMap.cpp

* fix 3D reconstruction cannot find stdafx.pch

* fix redefinition of default argument

* modify cmake to remove hand and avatar from general build settings and modify headers in file

* modify cmake to add openark/ to include path

* fix include path bugs

* turn off data_recording

* fix src_dir bug

* fix cmake bug

* fix webcam bug

* fix webcam bug v2

* put hand_and_avatar back into general build

* Added 3D recon offline

* remove DLoopDetector from cmake

* fix include errors

* update SlamReplaying for MockCamera

* remove libtorch imports for now

* some refactoring/renaming and documentation updates

* syuntax

* update docuemntation to point to my fork of okvis

* new datasets

* fixed 3drecondemo, definitely works online. Also made smal chnages to cmakelists for dloopdetector

* new openark-deps installer

* fix openark dependency installer okvis path

* orb voca support

* change orb_vocab.txt

* minor chnages to get working with ORB

* templated okvisslamsystem to enable brisk and orb

* changed SLAM Demo to BRISK and left rest as ORB

* changed default for 3d to brisk and slam demo to orb, added feature selection to okvisslamsystem

* openark dependency installer update

* changed source repo for ubuntu build

* removed cache (temp stuff)

Co-authored-by: Xiao Song <xiaosx@berkeley.edu>
Co-authored-by: jackchen03 <tim87618jack@gmail.com>
Co-authored-by: Tygen <tygensun@gmail.com>
Co-authored-by: Tygen <30679560+Tygen@users.noreply.github.com>
Co-authored-by: Yuandi-Sherry <1094050886@qq.com>
Co-authored-by: Yuandi-Sherry <31603947+Yuandi-Sherry@users.noreply.github.com>
Co-authored-by: afrozalm <alam.afroz011@gmail.com>
1204eed

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

OpenARK

OpenARK is an open-source wearable augmented reality (AR) system founded at UC Berkeley in 2016. The C++ based software offers innovative core functionalities to power a wide range of off-the-shelf AR components, including see-through glasses, depth cameras, and IMUs. The open-source platform includes higher-level modules to aid human-computer interaction, such as 3D gesture recognition, plane detection, avatar/pose tracking, and multi-user collaboration, and also contains fundamental tools such as AR-based camera calibration, depth-to-stereo, and SLAM, and 3D Reconstruction. Currently, it supports both PMD Pico Flexx and Intel RealSense SR300 cameras. The project builds natively on both Windows and Linux.

At a Glance,

  • Technology stack: C++, OpenCV, PCL, Boost, Intel RealSense SDK (1 or 2) / PMD SDK
  • Status: Beta 0.9.3
  • Application Demos: Vive Center

Dependencies

Hardware

  • Depth Camera
  • RGB Camera
  • Transparent AR Glasses (optional)

Software

  • Eigen 3
  • OpenCV 3.2.0+
  • PCL 1.8
  • Boost 1.6.4
  • Ceres-solver
  • DBoW2
  • etc. for the full list, see DEPENDENCIES.md

Installation

Windows

New: a dependency pack installer is now available, which accelerates the setup process. Please read documentation/Windows-build-instructions-quick.md for instructions. The installer may be found here.

New: Click below for the Youtube build tutorial that can help guide you through the accelerated install process, though the written instructions will be updated more frequently.

OpenArk Windows Installation Instructions

Building from source without dependency pack installer:

Check the following installation document for OpenARK for Windows : documentation/Windows-build-instructions.pdf

Prebuilt Binaries

Prebuilt binaries for 64-bit Windows are available here. This is only updated for major releases. Both the static library (with the headers) and the demo program are included. For the SVM to work properly, the svm/ folder must be present in the current directory OR under the directory pointed to by the OPENARK_DIR environment variable.

Linux

Check the following installation document for OpenARK for Linux : documentation/Linux-build-instructions.md

The Avatar Module/Demo

To use the avatar module/demo, please download the SMPL model files manually. The instructions are in data/avatar-model/README.md. You may also download the dataset sample by running data/avatar-dataset/download.sh.

Usage

To use OpenARK in a Visual Studio C++ project:

After setting up all the dependencies (OpenCV, PCL, etc.) according the configuration instructions, add openark_x_x_x.lib to Project > MyProjectProperties > Linker > Input > Additional Dependencies. Then add OPENARK_DIR/include to C/C++ > General > Additional Include Directories. Finally, make sure that under C/C++ > Code Generation, Runtime Library is set to Multi-threaded DLL (/MD).

Now you can #include OpenARK's core header in any file and begin using OpenARK. You would probably also want to include one of the depth camera backend headers (e.g. SR300Camera.h).

Here is the outline of a program for performing hand detection:

...
#include "openark/util/Core.h"
#include "openark/camera/SR300Camera.h"

#include "opencv2/core.hpp"
#include <vector>
...
int main() {
    ark::DepthCamera & camera = ark::SR300Camera(); // OpenARK camera backend
    ark::HandDetector detector(); // OpenARK hand detector; also see PlaneDetector

    // start the camera; alternatively, call nextFrame() manually inside the loop (slower)
    camera.beginCapture();
    ...
    while (true) {
        cv::imshow("XYZ Map", camera.getXYZMap());

        detector.update(camera);
        std::vector<ark::Hand::Ptr> hands = detector.getHands();

        // do something with the hands detected
        ...

        // quit when q is pressed
        if (cv::waitKey(1) == 'q') break;
    }

    ...
    // automatically stops capture on exit
}

Additional sample code is available in samplecode/.

Customization

OpenARK is made for easy customization. Please feel free to build on top of this platform to fit your needs.

How to test the software

Code used to run the demo video is included in HandDemo.cpp. Additional sample code can be found in /samplecode/ and you would need to replace it with the main that comes with the project solution.

Using OpenARK SLAM

Currently OpenARK only supports the Intel Realsense D435i Camera for use with OpenARK SLAM. OpenARK SLAM requires two configuration files to run, a camera intrinsics file, and a vocab file. Examples of both files can be found in the config folder. To run the OpenARK SLAM Demo run:

OpenARK_SLAM_demo.exe <intrinsics file> <vocab file>

While the vocab file provided should work for most purposes, for best performance a custom intrinsics file should be generated for each camera. You can generate a custom intrinsics file for your camera by running

d435i_intrinsics_writer <camera name>

This will generate a complete intrinsics file named <camera name>_intr.yaml

OpenARK SLAM heavily utilizes the open source packages DBoW2, Okvis, and Ceres. Please respect their Licences and credit/cite when appropriate.

Using OpenARK 3D Reconstruction

Currently OpenARK only supports the Intel Realsense D435i Camera for use with OpenARK 3D Reconstruction. OpenARK 3D Reconstruction requires two configuration files to run, a camera intrinsics file, and a vocab file. Examples of both files can be found in the config folder. It also accepts a frame output directory, which defaults to /frames/. To run the OpenARK 3D Reconstruction Demo run:

3dRecon_Data_Recording.exe <intrinsics file> <vocab file> <frame directory>

While the vocab file provided should work for most purposes, for best performance a custom intrinsics file should be generated for each camera. You can generate a custom intrinsics file for your camera by running

d435i_intrinsics_writer <camera name>

This will generate a complete intrinsics file named <camera name>_intr.yaml

Offline reconstruction can be performed on the recorded data output of the application using the Python script located in /scripts/OfflineReconstruction.py.

OpenARK 3D Reconstruction heavily utilizes the open source packages DBoW2, Okvis, Open3D, and Ceres. Please respect their Licences and credit/cite when appropriate.

Known issues

OpenCV prior to 3.2.0 does not offer prebuilt VC14+ binaries. Running VC12 OpenCV binaries with VC14 will result in memories errors in findCountours(). If you are using VC12+ to compile OpenARK, you will need to use CMake to rebuilt OpenCV from source. We have used OpenCV 3.2.0 which comes with OpenCV binaries for VC14 to avoid this problem when using Visual Studio 2015.

Getting help

If you have questions, concerns, bug reports, etc, please file an issue in this repository's Issue Tracker.

Getting involved

The Center for Augmented Cognition welcomes interested industry partners to join our alliance to support the OpenARK platform. More information can be found on cac.berkeley.edu


Credits and references

Bill Zhou, Allen Y. Yang, S. Shankar Sastry, Will Huang, Larry Yang, Eric Nguyen, Michelli Ni, Peter Li, Jessica Jiang, Mona Jalal, Joseph Menke, Lawrence Chen, Kuan Lu, Rachel Lee, Justin Yang, Alex Yu, Adam Chang, Woojin Ko

About

OpenARK is an open-source wearable augmented reality (AR) system founded at UC Berkeley in 2016. The C++ based software offers innovative core functionalities to power a wide range of off-the-shelf AR components, including see-through glasses, depth cameras, and IMUs.

Resources

License

Stars

Watchers

Forks

Packages

No packages published