GSoC 2017: dynamicfusion

[mihaibujanca]

GSoC Project: Implementing and extending DynamicFusion (Newcombe et al 2015)
Mentors: Zhe Zhang, Reza Amayeh

For now just added the code from my repository, with CMakeLists.txt building the project as stand-alone. I will be working towards changing that to be an opencv_contrib module.

docker_image:Custom=ubuntu-cuda:16.04

[zzhang001]

This is this GSoC project's mentor. @mshabunin Please advise the next steps moving forward. There will be dependencies to be installed for this to run. Any instructions we should follow to pass the checks, additional cleanups/refactors needed for us to merge in? Thanks in advance!

[mshabunin]

@zzhang001 , as I can see, most of dependencies can be provided by OpenCV, for example OpenNI and CUDA. Module should be reconstructed to be supported by our build scripts, please take a look at some other modules in opencv_contrib.
General scheme is as follows:

├── CMakeLists.txt
├── doc
│   └── <module>.bib
├── include
│   └── opencv2
│       └── <module>.hpp
├── README.md
├── samples
│   ├── <sample_app1>.cpp
│   └── <sample_script1>.py
├── src
│   ├── <internal1>.hpp
│   └── <internal1>.cpp
├── test
│   ├── test_<block1>.cpp
│   ├── test_main.cpp
│   └── test_precomp.hpp
└── tutorials
    └── <tutorial1>.markdown

[mihaibujanca]

@mshabunin What about dependencies like Ceres and Boost?

[mshabunin]

Looks like it is possible to avoid using Boost in this case, as for Ceres - we have the sfm module which optionally uses this library, so you can take check corresponding cmake script. For this moment you should integrate it by yourself, but in future we can move Ceres detection to main repository.

[mshabunin]

You should not use REQUIRED option, instead just check whether the dependency has been found and disable your module if it can not be built without it:

find_package(Ceres QUIET)
if (NOT Ceres_FOUND)
    message(STATUS "${module} has been disabled because Ceres library has not been found")
    ocv_module_disable(${module})
endif()

[mshabunin]

Try to use existing HAVE_CUDA variable, check the documentation at the beginning of https://github.com/opencv/opencv/blob/master/cmake/FindCUDA.cmake

[mshabunin]

Looks like the Boost is not needed anymore.

[mshabunin]

I don't see any #include "opencv2/calib3d.hpp" in your code, it means that you don't need the dependency on opencv_calib3d module. Same for opencv_highgui.

Note that all samples will receive dependency on imgcodecs, videoio and highgui automatically, so you don't need to add them to your module.

[mshabunin]

It should be #include <opencv2/kfusion/utils/dual_quaternion.hpp>
You should create single include folder: include/opencv2/dynamicfusion and put all sub folders into it:

include/opencv2/
    dynamicfusion.hpp
    dynamicfusion/
        utils/
            ...
        kfusion/
            ...

Then: #include "opencv2/dynamicfusion/utils/dual_quaternion.hpp"

[mshabunin]

Is it possible to use existing flann module instead of nanoflann?

[mihaibujanca]

I'll look into porting from nanoflann to flann at some point later

[mshabunin]

Looks like this class is not used anymore, did you try to use VideoCapture with OPENNI backend?

VideoCapture cap(CAP_OPENNI2);

It has different properties which can be accessed via VideoCapture::get and ::set methods: link to documentation

[savuor]

@mihaibujanca After fixing different compilation errors I failed to run your code.
It ends at 5th frame with "out of memory" error trying to allocate more GPU memory. (On older GPUs even earlier when device's memory is exhausted).
And I get no 3d data in view window at all.
I use your test data downloaded with provided script.
Can you please provide a working version of your application? Maybe stand-alone one you made before creating a module?

[mihaibujanca]

@savuor it's changed quite a bit since the PR, but please check out https://github.com/mihaibujanca/dynamicfusion

Apologies for not having fixed it yet

[savuor]

@mihaibujanca The code from your repo doesn't work on your test data.
Some checks are missed, for example (on what I stopped now) WarpField is initialized on first frame size which is empty by some reason, and it leads to segfault.
But at least 3d viewer displays some data.
Can you fix this? At least in your standalone version but it'd be better to fix also this PR.

[mihaibujanca]

Will be working on it, though not completely sure why the first frame would be empty - can have you downloaded the data and passed the directory as an argument?

[savuor]

@mihaibujanca
After many unsuccessful attempts to run your code on my machine I decided to implement at least basic version of KinectFusion. Here it is: #1627
Since OpenCV is a cross-platform library each algorithm needs to have a CPU implementation at least (though it can be very slow).
Our version of KinectFusion is CPU-based but works fast enough on rough scale.
Please feel free to add GPU-version (no CUDA code, OpenCL only to let it work on a maximum amount of platforms).
Also you may implement DynamicFusion based on current KinectFusion implementation. This should be similar to your code since the base code is the same (kinfu-remake by A. Baksheev).
But there should be a CPU-only implementation for tests and accuracy checks (even if it's quite slow).

[mihaibujanca]

@savuor Thanks a lot for your work. I'm currently in the exam session so I don't expect to be able to do this for a while.
That being said, I apologise for my lack of activity on this and don't intend on leaving it the way it is now.

[savuor]

@mihaibujanca No problem, you can attend later when you have a time

[savuor]

@mihaibujanca FYI #1798