[Feature] Gesture recognition algorithm MTUT on NVGesture dataset #1380

Ben-Louis · 2022-05-18T03:30:28Z

Motivation

Add hand gesture recognition function to mmpose.

Modification

add NVGesture dataset
add I3D backbone to extract features from videos
add MTUT head
add GestureRecognizer to support general gesture recognition algorithms
add configs for training/test of I3D+MTUT model on NVGesture dataset

BC-breaking (Optional)

Use cases (Optional)

Checklist

Before PR:

I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
New functionalities are covered by complete unit tests. If not, please add more unit tests to ensure correctness.
The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

CLA has been signed and all committers have signed the CLA in this PR.

…ognition

codecov · 2022-05-18T03:38:12Z

Codecov Report

Merging #1380 (d698319) into master (a47e54e) will increase coverage by 1.14%.
The diff coverage is 84.15%.

@@            Coverage Diff             @@
##           master    #1380      +/-   ##
==========================================
+ Coverage   83.21%   84.35%   +1.14%     
==========================================
  Files         225      232       +7     
  Lines       19006    19308     +302     
  Branches     3395     3472      +77     
==========================================
+ Hits        15815    16287     +472     
+ Misses       2359     2154     -205     
- Partials      832      867      +35

Flag	Coverage Δ
unittests	`84.29% <84.15%> (+1.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmpose/apis/__init__.py	`100.00% <ø> (ø)`
mmpose/apis/inference_tracking.py	`61.41% <0.00%> (ø)`
mmpose/datasets/__init__.py	`100.00% <ø> (ø)`
mmpose/datasets/pipelines/loading.py	`74.11% <59.45%> (-11.30%)`	⬇️
mmpose/core/utils/model_util_hooks.py	`71.42% <71.42%> (ø)`
mmpose/models/detectors/gesture_recognizer.py	`72.83% <72.83%> (ø)`
mmpose/apis/inference.py	`59.88% <77.77%> (+2.12%)`	⬆️
.../datasets/datasets/gesture/gesture_base_dataset.py	`80.48% <80.48%> (ø)`
mmpose/datasets/pipelines/gesture_transform.py	`83.63% <83.63%> (ø)`
...ose/datasets/datasets/gesture/nvgesture_dataset.py	`87.35% <87.35%> (ø)`
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a47e54e...d698319. Read the comment docs.

CLAassistant · 2022-05-26T03:00:09Z

All committers have signed the CLA.

tools/webcam/configs/gesture/try_det.py

mmpose/models/heads/mtut_head.py

…o gesture_recognition

mmpose/datasets/datasets/gesture/nvgesture_dataset.py

jin-s13 · 2022-05-31T10:54:57Z

configs/hand/gesture_sview_rgbd_vid/mtut/nvgesture/i3d_nvgesture_bbox_112x112_fps15_rgb.py

+
+model = dict(
+    type='GestureRecognizer',
+    modality=['rgb'],


Is this config for rgb only?
Then the config file path gesture_sview_rgbd_vid is a little bit confusing.

Yes. As stated in /configs/hand/gesture_sview_rgbd_vid/mtut/nvgesture/i3d_nvgesture.md, MTUT supports multi-modal training and uni-modal inference. This config is mainly used for demo and webcam with RGB videos.

jin-s13 · 2022-05-31T10:55:44Z

mmpose/datasets/datasets/gesture/nvgesture_dataset.py

+            dict: Evaluation results for evaluation metric.
+        """
+        metrics = metric if isinstance(metric, list) else [metric]
+        allowed_metrics = ['AP']


AP. Is it less confusing if I change 'mAP' to 'AP_mean'?

jin-s13

Looks good to me in general. Unittests are required before merging.

mmpose/datasets/pipelines/gesture_transform.py

mmpose/models/backbones/utils/transformer.py

mmpose/datasets/pipelines/gesture_transform.py

ly015 · 2022-06-01T14:56:12Z

@Ben-Louis CI failed.

Ben-Louis · 2022-06-01T15:04:58Z

@Ben-Louis CI failed.

I see. Unittest fails because of incompatibility of isnan function in Pytorch 1.5. I use the method mentioned in yuantn/MI-AOD#2 to get rid of this function.

…en-mmlab#1380) * add nvgesture dataset * fix nvgesture pipelines * update gesture datasets * add ModelSetEpochHook * nvgesture dataset support multi-GPU evalutation * add i3d+mtut model * add nvgesture i3d configs * webcam add hand detector * gesture recognition with bbox * add hand detector config * fix gesture recognizer init bug * webcam/gesture - recognizer runs successfully * delete unnecessary comment * fix lint error in gesture configs * add nvgesture category info * webcam/gesture - display gesture recognition result * add gesture recognition related docs * update gesture related comments * update light hand det model in demo doc * update gesture recognition configs and results * auto modify model-index.yml * stabilize ssa loss in mtut * add multi-input node comment * synchronize tools/webcam with master * add gesture task-name mapping * move gesture configs to configs/hand/ * fix a bug in demo (open-mmlab#1373) * update gesture datasets * add ModelSetEpochHook * nvgesture dataset support multi-GPU evalutation * add i3d+mtut model * add nvgesture i3d configs * webcam add hand detector * gesture recognition with bbox * add hand detector config * fix gesture recognizer init bug * webcam/gesture - recognizer runs successfully * delete unnecessary comment * fix lint error in gesture configs * add nvgesture category info * webcam/gesture - display gesture recognition result * add gesture recognition related docs * update gesture related comments * update light hand det model in demo doc * update gesture recognition configs and results * auto modify model-index.yml * stabilize ssa loss in mtut * add multi-input node comment * synchronize tools/webcam with master * add gesture task-name mapping * move gesture configs to configs/hand/ * solve conflict in mmdet_modelzoo.md * add gesture recogition into webcam * hand gesture inference config explanation * add gesture recognizer node in __init__.py * add gesture webcam readme * Adjust inference tracking min keypoints (open-mmlab#1398) * Adjust inference tracking min keypoints * Special case for min_keypoints <= 0 doesn't seem to be required * remove unnecessary transformer utils (open-mmlab#1405) * add nvgesture dataset * fix nvgesture pipelines * update gesture datasets * add ModelSetEpochHook * nvgesture dataset support multi-GPU evalutation * add i3d+mtut model * add nvgesture i3d configs * webcam add hand detector * gesture recognition with bbox * add hand detector config * fix gesture recognizer init bug * webcam/gesture - recognizer runs successfully * delete unnecessary comment * fix lint error in gesture configs * add nvgesture category info * webcam/gesture - display gesture recognition result * add gesture recognition related docs * update gesture related comments * update light hand det model in demo doc * update gesture recognition configs and results * auto modify model-index.yml * stabilize ssa loss in mtut * add multi-input node comment * synchronize tools/webcam with master * add gesture task-name mapping * move gesture configs to configs/hand/ * fix grammer errors in docs * fix a lint error in doc * update nvgesture evaluation * add introduction and assertion to TemporalPooling * generalize NVGestureRandomFlip * add unittests for gesture pipelines * delete duplicated config * add gesture inference unittest * add gesture dataset unittest * fix gesture inference unittest error * add backbone I3D unittest * add mtut head unittest * fix mtut head unittest error * add gesture recognizer unittest Co-authored-by: Yining Li <liyining0712@gmail.com> Co-authored-by: Philipp Allgeuer <5592992+pallgeuer@users.noreply.github.com>

Ben-Louis added 8 commits May 16, 2022 13:56

add nvgesture dataset

1183943

fix nvgesture pipelines

06133d2

Merge branch 'master' of github.com:Ben-Louis/mmpose into gesture_rec…

6ffe1cf

…ognition

update gesture datasets

39950a5

add ModelSetEpochHook

42dccc9

nvgesture dataset support multi-GPU evalutation

33a5225

add i3d+mtut model

3aa43d1

add nvgesture i3d configs

c1f1273

Ben-Louis added 7 commits May 26, 2022 11:11

webcam add hand detector

9a6d538

gesture recognition with bbox

08fe70f

add hand detector config

b99182b

fix gesture recognizer init bug

000d685

webcam/gesture - recognizer runs successfully

6a8d1b6

delete unnecessary comment

e0614af

fix lint error in gesture configs

8eb6198

Ben-Louis force-pushed the gesture_recognition branch from 0736019 to 8eb6198 Compare May 26, 2022 03:37

Ben-Louis added 2 commits May 26, 2022 12:32

add nvgesture category info

4124afb

webcam/gesture - display gesture recognition result

87b4766

jin-s13 reviewed May 26, 2022

View reviewed changes

tools/webcam/configs/gesture/try_det.py Outdated Show resolved Hide resolved

jin-s13 reviewed May 26, 2022

View reviewed changes

mmpose/models/heads/mtut_head.py Show resolved Hide resolved

jin-s13 reviewed May 26, 2022

View reviewed changes

mmpose/models/heads/mtut_head.py Show resolved Hide resolved

jin-s13 requested a review from ly015 May 26, 2022 07:11

Ben-Louis added 6 commits May 30, 2022 00:18

add gesture recognition related docs

fc48962

update gesture related comments

e3d0236

update light hand det model in demo doc

08a642b

update gesture recognition configs and results

2bc31ed

auto modify model-index.yml

6cf6e98

stabilize ssa loss in mtut

7f2dacc

Ben-Louis added 3 commits May 31, 2022 14:47

Merge branch 'gesture_recognition' of github.com:Ben-Louis/mmpose int…

9aad796

…o gesture_recognition

fix grammer errors in docs

73930c1

fix a lint error in doc

3932e9e

jin-s13 reviewed May 31, 2022

View reviewed changes

mmpose/datasets/datasets/gesture/nvgesture_dataset.py Outdated Show resolved Hide resolved

update nvgesture evaluation

4ab441c

jin-s13 reviewed May 31, 2022

View reviewed changes

jin-s13 added the need tests label May 31, 2022

jin-s13 reviewed May 31, 2022

View reviewed changes

Ben-Louis added 8 commits May 31, 2022 21:09

add introduction and assertion to TemporalPooling

f9c9ea4

generalize NVGestureRandomFlip

c13d213

add unittests for gesture pipelines

dc6c240

delete duplicated config

9a85ce2

add gesture inference unittest

4c67b94

add gesture dataset unittest

2483247

fix gesture inference unittest error

23d0e75

add backbone I3D unittest

d090bd5

jin-s13 approved these changes Jun 1, 2022

View reviewed changes

add mtut head unittest

64b9699

fix mtut head unittest error

b75dea9

add gesture recognizer unittest

d698319

ly015 removed the need tests label Jun 2, 2022

ly015 approved these changes Jun 2, 2022

View reviewed changes

ly015 merged commit d3c17d5 into open-mmlab:master Jun 2, 2022

ly015 mentioned this pull request Jun 2, 2022

[Fix] fix doc page of model-zoo #1408

Merged

6 tasks

Ben-Louis deleted the gesture_recognition branch July 29, 2022 03:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Gesture recognition algorithm MTUT on NVGesture dataset #1380

[Feature] Gesture recognition algorithm MTUT on NVGesture dataset #1380

Ben-Louis commented May 18, 2022

codecov bot commented May 18, 2022 •

edited

Loading

CLAassistant commented May 26, 2022 •

edited

Loading

jin-s13 May 31, 2022

Ben-Louis May 31, 2022

jin-s13 May 31, 2022

Ben-Louis May 31, 2022

jin-s13 left a comment

ly015 commented Jun 1, 2022

Ben-Louis commented Jun 1, 2022

[Feature] Gesture recognition algorithm MTUT on NVGesture dataset #1380

[Feature] Gesture recognition algorithm MTUT on NVGesture dataset #1380

Conversation

Ben-Louis commented May 18, 2022

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

codecov bot commented May 18, 2022 • edited Loading

Codecov Report

CLAassistant commented May 26, 2022 • edited Loading

jin-s13 May 31, 2022

Choose a reason for hiding this comment

Ben-Louis May 31, 2022

Choose a reason for hiding this comment

jin-s13 May 31, 2022

Choose a reason for hiding this comment

Ben-Louis May 31, 2022

Choose a reason for hiding this comment

jin-s13 left a comment

Choose a reason for hiding this comment

ly015 commented Jun 1, 2022

Ben-Louis commented Jun 1, 2022

codecov bot commented May 18, 2022 •

edited

Loading

CLAassistant commented May 26, 2022 •

edited

Loading