Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
ute
 
 
 
 
 
 

Unsupervised learning of action classes with continuous temporal embedding

Official implementation in python. https://arxiv.org/abs/1904.04189

If you use the code, please cite

@inproceedings{kukleva2019unsupervised,
  title={Unsupervised learning of action classes with continuous temporal embedding},
  author={Kukleva, Anna and Kuehne, Hilde and Sener, Fadime and Gall, Jurgen},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR'19)},
  year={2019}
}

To train or reproduce numbers: HowTo

Pipeline for one activity class. Figure 1 in the paper.

alt text

Proposed pipeline for unsupervised learning with unknown activity classes. Figure 2 in the paper.

alt text

Visualization of embeddings via t-SNE on the 50Salads dataset

Each frame is color coded a) with the corresponding ground truth subaction label, b) with K assigned subaction labels after clustering as the second step in Fig.1 in our main paper, c)with the predicted labels after the decoding stage. The optimization of our network is performed with respect to relative timestep of each frame. In d) we show the respective relative time label in the continuous temporal embedding assigned to each frame feature. The color bar depicts that bright blue corresponds to 0 (startof the video) and pink to 1 (end of the video).

The number of subactions K

Breakfast dataset
Activity class name # subactions (K)
Coffe 7
Cereals 5
Tea 7
Milk 5
Juice 8
Sandwich 9
Scrambledegg 12
Friedegg 9
Salat 8
Pancake 14
YouTube Instractions dataset
Activity class name # subactions (K)
Changing tire 11
Making cofee 10
CPR 7
Jump car 12
Repot plant 8

Qualitative results

Breakfast dataset. The order of subactions: SIL, take bowl, pour cereals, pour milk, stir cereals, SIL

Breakfast dataset. The order of subactions: SIL, take cup, add teabag, pour water, SIL

Breakfast dataset. The order of subactions: SIL, spoon powder, pour milk, stir milk, SIL

Breakfast dataset. The order of subactions: SIL, take knife, cut orange, squeeze orange, pour juice, squeeze orange, pour juice, squeeze orange, pour juice, squeeze orange, pour juice, SIL

Breakfast dataset. The order of subactions: SIL, cut bun, smear butter, put toppingOnTop, SIL

50Salads dataset. The order of subactions: start, cut, place, cut, place, cut, place, cut, place, null, null, add oil, add pepper, mix dressing, end

50Salads dataset. The order of subactions: start, cut, place, cut, place, cut, place, peel cucumber, cut, place, mix ingredients, add oil, null, add pepper, null, mix dressing, serve salad onto plate, add dressing, end

About

Official implementation of the paper: Unsupervised learning of action classes with continuous temporal embedding (CVPR'19)

Topics

Resources

License

Releases

No releases published

Packages

No packages published

Languages