We build a tool to enhance sightseeing using augmented reality. First we create a 3D model from images for a sight. Next AR content is created and fitted to the model. Then in real-time the AR content is superimposed precisely over the real world building.
The work is using hloc in the backend, a modular toolbox for state-of-the-art 6-DoF visual localization from Sarlin
et al. Hloc.
We use it for model creation and localization. The backend functions as a HTTP server, that a run-time carries out pose estimation in real-time, matching query images against the previously created models.
The Frontend (App) is developed with Unity and ARCore and displays the content over the building.
For the whole process we provide a Demo video as well as a step-by-step Tutorial video for setting up a new sight.
Our tool requires Python >=3.6, PyTorch >=1.1, and COLMAP.
Other dependencies are listed in the pyproject.toml and can be installed using Poetry with the command poetry install
The backend performs 2 major tasks.
Model Creation(in advance):
- Extract SuperPoint local features
- Match features with SuperGlue
- Build a 3D SfM model
Visual Localization (at runtime):
- Match the query images with SuperGlue
- Run the localization
- Visualize and debug
The frontend.
In advance:
- AR content is created with Gimp.
- Content is added to the SfM model in Unity.
- In unity the model is added to the
placesa list of GameObjects and the App can be redeployed.
At runtime:
- The app sends HTTP request with the image to the server for localization.
- The answer contains the users pose, relative to the building. With this the previously created AR content is now superimposed on the building.
- Switching between content and text based information is now available.
External contributions are very much welcome. Features that are currently not implemented: