Skip to content
This repository has been archived by the owner on Sep 8, 2023. It is now read-only.
/ 3dify-ios Public archive

An experimental iOS app to create awesome 3D effects from photos using adaptive monocular depth interpretation and parallax occlusion mapping.

License

Notifications You must be signed in to change notification settings

PhilippMatthes/3dify-ios

Repository files navigation

3Dify App

UNDER CONSTRUCTION

The 3Dify app is an experimental iOS app to create 3D effects on photos. The final result looks something like this:

demo

The effect is achieved by two steps:

  1. Create a disparity map for the photo. On devices with multi camera arrays in portrait mode, the disparity map is provided by the AVCaptureDelegate. On devices without, the app makes use of CoreML to interpret the depth map using a custom pipeline.
  2. Use a Metal driven view to apply parallax occlusion mapping, using the calculated disparity map as depth.

Contributing

UNDER CONSTRUCTION

Delivery

UNDER CONSTRUCTION

Pipeline

On devices without the capability to create disparity maps (e.g. iPhone SE First Gen), the app makes use of a custom CoreML driven pipeline to create a disparity image. The pipeline looks like this:

The pipeline takes the original image as input and creates three different depth estimations using the FCRN mlmodel, the FastDepth mlmodel and the Pydnet mlmodel. To evaluate which estimated depth map is of the best quality (for blending), the pipeline computes a specific rms error value for each depth map. This is done by applying a sobel image to both the depth map as well as the original image. Then, by measuring the distance between the sobel images, a RMS value can be computed. Based on the RMS value, the pipeline blends the depth maps together. To further smoothen the surfaces of the combined depth map while preserving edges, the pipeline applies a bilateral filter which takes the edge details from the original image and smoothes the combined depth map.

Related Work

This project makes use of models and work by other authors:

Licenses

Please see the provided LICENSES.MD for the licenses to thirs party software. These licenses only apply to third party software and not to this project. If you want to use my code, please contact me first. You can see which code is copyrighted to me by having a look to each file's header.

About

An experimental iOS app to create awesome 3D effects from photos using adaptive monocular depth interpretation and parallax occlusion mapping.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published