Keras - C3D - CK+

Keras implementation of C3D model on Extended Cohn-Kanade Dataset (CK+).

C3D is a neural network that introduces 3D convolutions and pooling layers. For more information please refer to the original paper: Tran, Du, et al. "Learning Spatiotemporal Features With 3D Convolutional Networks." Proceedings of the IEEE International Conference on Computer Vision. 2015.

The user may define the following parameters upon calling the python script:

Whether to use face extraction (using MTCNN) on the original images as a preprocessing step or not.
Temporal depth of the image sequence transformed into a single instance for the 3D convolutions.
Minimum sequence length of the video Sessions that are considered for training. By a combination of depth and this variable, the user defines the percentage of frames labeled per Session. For example, for a depth = 3 and minimum sequence length = 9, only Sessions with 9 frames or more are considered. In this example, the first 3 frames per sequence are labeled as neutral and the last 3 as the corresponding emotion.
Size into which the images will be resized to be feeded to the model.

Requirements

Please make sure to comply with everything mentioned in this section before execution.

Dataset

The code requires the user to specify the location of four folders:

main_dir: folder containing the three subfolders below.
labels_main_dir: original ck+ folder with emotion labels.
images_main_dir: original ck+ folder with images.
emotions_main_dir: new folder in which images will be separated by emotion.

A suggested folder structure (following the above order) is:

~/ckp/
~/ckp/emotion_labels/
~/ckp/cohn-kanade-images/
~/ckp/emotion_images/

To get the database, please refer to the download site: http://www.consortium.ri.cmu.edu/ckagree/.

Packages

Make sure to install the necessary packages:

matplotlib
numpy
mtcnn
opencv (>3.4)
scikit-learn
keras

Running the code

To train the network you must follow the next steps:

Creating the 3D dataset

Run create_dataset.py defining the following arguments:

  python create_dataset.py \
  main_dir \            
  labels_main_dir \     
  images_main_dir \     
  emotions_dir \      
  crop_faces \        # Boolean (1, 0, True, False) select whether to crop the faces or not
  neutral_label \     # String name of neutral label
  min_seq_len \       # Integer minimum sequence length
  depth \             # Integer number of frames per sequence
  t_height \          # Integer new image height
  t_width             # Integer new image width

For example:

python create_dataset.py ~/ckp/ ~/ckp/emotion_labels/ ~/ckp/cohn-kanade-images/ ~/ckp/emotion_images/ True 0 9 3 112 112

This script will add a folder per emotion to ~/ckp/emotion_images/ and fill them with Numpy binary files. Each file will have an image sequence with the shape (depth, t_height, t_width, channels) = (3, 112, 112, 3). For every file added to an emotion folder, another one will be added to the neutral folder.

Training the network

Run train.py defining the following arguments:

python train.py \
emotions_dir \        
neutral_instances \   # Number of neutral instances to use
valid_split \         # Percentage of dataset to use as validation set
test_split \          # Percentage of dataset to use as test set
batch_size \          # Batch size
epochs                # Epochs

For example:

 python train.py  ~/ckp/emotion_images/  43 .1 .3 32 10

Model and weights will be saved on the same folder as the train.py code.

There are a lot of other parameters that can be tuned.

Please feel free to modify whatever you want and have a happy training!

robsanpam/CK-Plus

ErrorLooks like something went wrong!

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Error
Looks like something went wrong!

Packages