Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gestures recognition : CVPR-2015 Vision-Challenge submission #458

Closed
wants to merge 2 commits into from

Conversation

WGW101
Copy link

@WGW101 WGW101 commented Nov 30, 2015

Merge with extra: opencv/opencv_extra#280

The following pull-request corresponds to our submission to the CVPR-2015 Vision-Challenge in category « Gesture Recognition ».
The submitted code implements the gesture recognition method described in the paper

"'Natalia Neverova, Christian Wolf, Graham W. Taylor and Florian Nebout.
ModDrop: adaptive multi-modal gesture recognition.
To appear in IEEE Transactions on Pattern Analysis and Machine Intelligence".

An arxiv version is available at
http://arxiv.org/abs/1501.00102.

The original implementation in python has been coded by Natalia Neverova.
The python to C++ port has been implemented by Guillaume Bono, Awabot.

The code runs on Linux and has been tested on two different hardware platforms: x86 and ARM (Nvidia Jetson K1).

In its current state, the implementation doesn't use the auxiliary gestures detection network described in the article.
This first submission of the network wasn't trained using the "modality dropout" technique,
but it has already reached 82% accuracy on the validation database with the Caffe command-line interface.
Update of the network parameters have to be expected soon.

Please also note that there is a bug in the skeleton descriptor computation
that we are currently working on fixing.

All extra data can be found on the corresponding PR280 on the opencv_extra repository.
opencv/opencv_extra#280

We plan to continue to work on the code and submit future pull requests.

@alalek
Copy link
Member

alalek commented Nov 30, 2015

BTW, Patch with test data is very large, ~100 Mb. I'm not sure that we can accept it as is. Need to think how to resolve it.

@WGW101
Copy link
Author

WGW101 commented Dec 1, 2015

Working on it,
We'll use a python download script similar to the one in the dnn module,
just waiting for somewhere else to host the files.

Also working on the failing Caffe dependency.

ocv_warnings_disable(CMAKE_CXX_FLAGS -Wshadow -Wsign-compare)

set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_SOURCE_DIR})
find_package(Caffe REQUIRED)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't require Caffe dependency. Just disable module if Caffe not found.
Try something like this:

set(the_description "Gestures recognition framework")

set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_SOURCE_DIR})
find_package(Caffe QUIET)

if (NOT Caffe_FOUND)
  message(STATUS "Module ${the_description} is disabled (Caffe required).")
  return()
endif()

ocv_define_module(gestures opencv_core opencv_imgproc)

ocv_warnings_disable(CMAKE_CXX_FLAGS -Wshadow -Wsign-compare)
...

@vpisarev vpisarev assigned vpisarev and unassigned vpisarev Dec 1, 2015
@vpisarev
Copy link
Contributor

vpisarev commented Dec 1, 2015

@alalek, let me assign this to you for now; feel free to reassign it to me when the buildbot is green and you do not have complains on the code structure

@WGW101
Copy link
Author

WGW101 commented Dec 2, 2015

@alalek : Concerning Caffe, I'd like your advice.
The master branch of caffe doesn't support the 3d pooling operation (yet..)
which we need in our architecture, and was implemented in a parallel branch by the community.

Should I include this parallel version of caffe as a 3rd party library,
or just give instructions with the module on how to download and install the right version of caffe ?

@alalek
Copy link
Member

alalek commented Dec 2, 2015

I believe adding instructions is fine.

@alalek
Copy link
Member

alalek commented Dec 8, 2015

@vpisarev This patch looks good to me

@vpisarev
Copy link
Contributor

cool! may I ask you to do yet another little change before merging the code in? namely, could you please move the exposed functionality from cv namespace to cv::gestures, so that the probability of name conflicts is minimal?

@WGW101
Copy link
Author

WGW101 commented Dec 10, 2015

Should be done..
I also squashed all commits of the branch to have clean patch and history before merging.

Please note there are still a few bugs in the C++ port
that make the module fail to reach the same accuracy than the original python implementation.
We're hardly working on it and investigating where this comes from.

@vpisarev
Copy link
Contributor

thank you! the latest commit failed to pass the doc builder, could you please look at it? (click the "Details" link below)

@WGW101
Copy link
Author

WGW101 commented Dec 10, 2015

Conflict solved.

Do you mind waiting for us to solve the few bugs remaining to pass accuracy tests ?
It might be better to have a fully functional module before merging into master...

@vpisarev
Copy link
Contributor

@WGW101, yes, sure! take your time )

@WGW101 WGW101 force-pushed the gestures_recognition branch 3 times, most recently from b698481 to 5b1be4d Compare December 14, 2015 10:01
@WGW101
Copy link
Author

WGW101 commented Dec 17, 2015

Hi !

I need to transfer ownership of this forked repository to a new account created for long term support,
so I will certainly close this PR and make a new one from the new account @awabot-dev.

Once that transfer is done, we might be ready to merge,
as we now reach 84% precision accuracy on our test sample.
But I will confirm it from the other account.

Sorry for the inconvenience..

@awabot-dev awabot-dev force-pushed the gestures_recognition branch 2 times, most recently from e5e49ec to fb380c2 Compare December 17, 2015 09:37
@vpisarev
Copy link
Contributor

@WGW101, ok, thank you! waiting a confirmation from you then )

Guillaume Bono and others added 2 commits December 18, 2015 11:33
…n multi-modalities input : color, depth and mocap stream.

The current implementation is based on the deep neural network architecture described in:

    "'Natalia Neverova, Christian Wolf, Graham W. Taylor and Florian Nebout.
    ModDrop: adaptive multi-modal gesture recognition.
    To appear in IEEE Transactions on Pattern Analysis and Machine Intelligence".

An arxiv version is available at:
    http://arxiv.org/abs/1501.00102.

It corresponds to our submission to the CVPR-2015 Vision-Challenge in categorie « Gesture Recognition ».

+ Integration of Gestures Recognition through Deep Neural Network,
+ using Algorithm base class and exposing only abstract classes,
+ with a few running accuracy tests,
+ and basic documentation.

The module will be disabled if Caffe dependency is not found.
Please see installation instructions in README.md .
@awabot-dev
Copy link

Hi !
I think we are ready to merge once the build bot finishes :).

I have just rebased at HEAD of master branch,
tell me if there are any other things to do before we can proceed.

Thanks for your support.

@awabot-dev
Copy link

Hello,

Now everything is good, the merge can be done. Any idea about when this can be done ?

Thanks

@StevenPuttemans
Copy link

@vpisarev any reason why this is not in yet? I would eagearly start exploring without having to use a local clone :P

@mshabunin
Copy link
Contributor

Unfortunately this PR can not be refreshed easily. It uses custom fork of Caffe, test data and model can not be downloaded anymore. It must be rewritten to use dnn module to be revived and model should be uploaded to more stable storage.

@mshabunin mshabunin closed this Jul 25, 2019
@StevenPuttemans
Copy link

@mshabunin is there somewhere a list to keep track of such a closed down PR's. Else it's a pitty to see code like this vanish, even if we need alot of work redesigning the backend.

@mshabunin
Copy link
Contributor

@StevenPuttemans , there is no such list, but we can tag such PRs with a label. Feel free to propose label name and PRs to be marked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants