Skeleton based action recognition

docs/reference/engine-data.md

@@ -140,3 +140,27 @@ The [PointCloudWithCalibration](#class_engine.data.PointCloudWithCalibration) cl
 #### numpy()
   Return a  [NumPy](https://numpy.org)-compatible representation of data.
   Given that *data* argument is already internally stored in [NumPy](https://numpy.org)-compatible format, this method is equivalent to `data()`.
+
+
+### class engine.data.SkeletonSequence
+Bases: `engine.data.Data`
+
+A class used for representing a sequence of body skeletons in a video.
+
+The [SkeletonSequence](#class_engine.data.SkeletonSequence) class has the following public methods:
+#### SkeletonSequence(data=None)
+  Construct a new [SkeletonSequence](#class_engine.data.SkeletonSequence) object based on *data*.
+  *data* is expected to be a 5-D array that can be casted into a 5-D [NumPy](https://numpy.org) array.
+
+#### data()
+  Return *data* argument.
+  Return type is uint8 [NumPy](https://numpy.org) array.
+
+#### data(data)
+  Set the internal *data* argument.
+  *data* is expected to be a 5-D array that can be casted into a 5-D [NumPy](https://numpy.org) array, where the
+  dimensions can be organized as e.g. (num_samples, channels, frames, joints, persons).
+
+#### numpy()
+  Return a  [NumPy](https://numpy.org)-compatible representation of data.
+  Given that *data* argument is already internally stored in [NumPy](https://numpy.org)-compatible format, this method is equivalent to `data()`.

docs/reference/engine-target.md

@@ -102,4 +102,30 @@ and optionally the prediction confidence.
 The [SpeechCommand](#class_engine.target.SpeechCommand) class has the following public methods:
 #### SpeechCommand(prediction, confidence=None)
 Construct a new [SpeechCommand](#class_engine.target.SpeechCommand) object based from *prediction*.
-*prediction* is expected to be an integer designating the class and optional *confidence* a float between 0 and 1.
+*prediction* is expected to be an integer designating the class and optional *confidence* a float between 0 and 1.
+
+
+### class engine.target.ActionCategory
+Bases: `engine.target.Target`
+
+This target is used for skeleton-based action recognition.
+
+
+The [ActionCategory](#class_engine.target.ActionCategory) class has the following public methods:
+#### ActionCategory(actioncategory, confidence=None)
+  Construct a new [ActionCategory](#class_engine.target.ActionCategory) object based from *actioncategory*.
+  *actioncategory* is expected to be an integer number which shows the action class number. 
+  Confidence indicates the confidence score of the classification. 
+
+
+### class engine.target.Pose
+Bases: `engine.target.Target`
+
+This target is used for pose estimation. It contains a list of Keypoints.
+Refer to kpt_names for keypoint naming.
+
+The [Pose](#class_engine.target.Pose) class has the following public methods:
+#### Pose(keypoints, confidence)
+  Construct a new [Pose](#class_engine.target.Pose) object based on *keypoints*.
+  *keypoints* is expected to be a list of [Keypoint](#class_engine.target.Keypoint) objects.
+

src/engine/data.py

@@ -401,3 +401,69 @@ def __str__(self):
         :rtype: str
         """
         return "Points: " + str(self.data) + "\nCalib:" + str(self.calib)
+
+
+class SkeletonSequence(Data):
+    """
+    A class used for representing a sequence of body skeletons in a video.
+
+    This class provides abstract methods for:
+    - returning a NumPy compatible representation of data (numpy())
+    """
+
+    def __init__(self, data=None):
+        super().__init__(data)
+
+        if data is not None:
+            self.data = data
+
+    @property
+    def data(self):
+        """
+        Getter of data. SkeletonSequence class returns a float32 NumPy array.
+
+        :return: the actual data held by the object
+        :rtype: A float32 NumPy array
+        """
+        if self._data is None:
+            raise ValueError("SkeletonSequence is empty")
+
+        return self._data
+
+    @data.setter
+    def data(self, data):
+        """
+        Setter for data.
+
+        :param: data to be used for creating a skeleton sequence
+        """
+        # Convert input data to a NumPy array
+        # Note that will also fail for non-numeric data (which is expected)
+        data = np.asarray(data, dtype=np.float32)
+
+        # Check if the supplied vector is 5D, e.g. (num_samples, channels, frames, joints, persons)
+        if len(data.shape) != 5:
+            raise ValueError(
+                "Only 5-D arrays are supported by SkeletonSequence. Please supply a data object that can be casted "
+                "into a 5-D NumPy array.")
+
+        self._data = data
+
+    def numpy(self):
+        """
+        Returns a NumPy-compatible representation of data.
+
+        :return: a NumPy-compatible representation of data
+        :rtype: numpy.ndarray
+        """
+        # Since this class stores the data as NumPy arrays, we can directly return the data
+        return self.data
+
+    def __str__(self):
+        """
+        Returns a human-friendly string-based representation of the data.
+
+        :return: a human-friendly string-based representation of the data
+        :rtype: str
+        """
+        return str(self.data)

src/engine/target.py

@@ -257,3 +257,16 @@ def __str__(self):
             return f"Class {self.data} speech command with confidence {self.confidence}"
         else:
             return f"Class {self.data} speech command"
+
+
+class ActionCategory(Target):
+    """
+    This target is used for skeleton-based action detection.
+    """
+    def __init__(self, actioncategory, confidence=None):
+        super().__init__()
+        self.data = actioncategory
+        self.confidence = confidence
+
+    def __str__(self):
+        return str(self.data)

src/perception/skeleton_based_action_recognition/README.md

@@ -0,0 +1,57 @@
+# Skeleton-based Human Action Recognition
+Python implementation of baseline method, ST-GCN [[1]](#1), and the proposed methods 
+TA-GCN [[2]](#2), ST-BLN [[3]](#3) and PST-GCN [[4]](#4) for Skeleton-based Human 
+Action Recognition. 
+The ST-GCN, TA-GCN and ST-BLN methods can be run and evaluated using stgcn_learner by specifying the model name. 
+The PST-GCN method can be run and evaluated using pstgcn_learner. 
+
+This implementation is adapted from the [OpenMMLAB toolbox](
+https://github.com/open-mmlab/mmskeleton/tree/b4c076baa9e02e69b5876c49fa7c509866d902c7).
+
+## Datasets
+### NTU-RGB+D-60 
+The NTU-RGB+D [[5]](#5) is the largest indoor-captured action recognition dataset which contains different data modalities 
+including the $3$D skeletons captured by Kinect-v2 camera. It contains 56,000 action clips from $60$ different action
+classes and each action clip is captured by 3 cameras with 3 different views, and provides two different benchmarks,
+cross-view (CV) and cross-subject (CS).
+In this dataset, the number of joints in each skeleton is 25 and each sample has a sequence of 300 skeletons with 3
+different channels each.
+### Kinetics-400 
+The Kinetics-Skeleton [[6]](#6) dataset is a widely used action recognition dataset which contains the skeleton data of 
+300,000 video clips of 400 different actions collected from YouTube. In this dataset each skeleton in a sequence has 18 
+joints which are estimated by the OpenPose toolbox [[7]](#7) and each joint is featured by its 2D coordinates and 
+confidence score. We used the preprocessed data provided by [[1]](#1) and it can be downloaded from [here](
+https://drive.google.com/file/d/103NOL9YYZSW1hLoWmYnv5Fs8mK-Ij7qb/view). 
+
+## References
+
+<a id="1">[1]</a> 
+[Yan, S., Xiong, Y., & Lin, D. (2018, April). Spatial temporal graph convolutional networks for skeleton-based action 
+recognition. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).](
+https://arxiv.org/abs/1609.02907)
+
+<a id="2">[2]</a> 
+[Heidari, N., & Iosifidis, A. (2020). Temporal Attention-Augmented Graph Convolutional Network for Efficient Skeleton-
+Based Human Action Recognition. arXiv preprint arXiv: 2010.12221.](https://arxiv.org/abs/2010.12221)
+
+<a id="3">[3]</a> 
+[Heidari, N., & Iosifidis, A. (2020). On the spatial attention in Spatio-Temporal Graph Convolutional Networks for 
+skeleton-based human action recognition. arXiv preprint arXiv: 2011.03833.](https://arxiv.org/abs/2011.03833)
+
+<a id="4">[4]</a> 
+[Heidari, N., & Iosifidis, A. (2020). Progressive Spatio-Temporal Graph Convolutional Network for Skeleton-Based Human 
+Action Recognition. arXiv preprint arXiv:2011.05668.](https://arxiv.org/pdf/2011.05668.pdf)
+
+<a id="5">[5]</a> 
+[Shahroudy, A., Liu, J., Ng, T. T., & Wang, G. (2016). Ntu rgb+ d: A large scale dataset for 3d human activity analysis.
+ In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1010-1019).](
+ https://openaccess.thecvf.com/content_cvpr_2016/html/Shahroudy_NTU_RGBD_A_CVPR_2016_paper.html)
+
+<a id="6">[6]</a>
+[Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., ... & Zisserman, A. (2017). 
+The kinetics human action video dataset. arXiv preprint arXiv:1705.06950.](https://arxiv.org/pdf/1705.06950.pdf) 
+
+<a id="7">[7]</a>
+[Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2017). Realtime multi-person 2d pose estimation using part affinity 
+fields. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7291-7299).](
+https://openaccess.thecvf.com/content_cvpr_2017/html/Cao_Realtime_Multi-Person_2D_CVPR_2017_paper.html)