Skip to content
Finalist entry for the M2CAI Workflow Challenge 2016
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

M2CAI Workflow Challenge 2016

This is the repo for the M2CAI Workflow Challenge 2016 of the following fellow PhD students :

  • The master of HMM, Thomas Robert, one to rule them all in python!
  • The super hero of fine tuning, Remi Cadene, the man who goes to Athens!

You can find our submitted paper on ArXiV or on M2CAI website (version may be older).

Do not hesitate to contact us. If you have any questions about our code and/or paper, please create an issue. For other queries, please send us an email.


To train CNN:

  • torch
  • torchnet
  • torchnet-vision
  • (gpu) cuda
  • (gpu) cudnn

To train HMM:

  • python
  • numpy
  • pandas
  • sklearn
  • matplotlib
  • seaborn

To evaluate with Jaccar score:

  • matlab

1. Extracting data

First we have to unzip and in data/raw.

Then, in data/interim, we extract images from videos and create several files that we will be using for training CNNs and HMMs.

$ ./src/main/data/ --pathraw data/raw --pathinterim data/interim
$ python src/main/data/
$ python src/main/data/

### 2. Training a Convolutional Neural Network (CNN)

First we train a CNN and we save the best model regarding the validation score for each epochs. The best set of hyperparameters is the default set of options for each models.

Fine tuning ResNet-200 (best result)

$ th src/main/classif/resnet.lua

Fine tuning Inception-V3

$ th src/main/classif/inceptionv3.lua

Fine tuning Inception-V3 with Weldon

$ th src/main/classif/inceptionv3weldon.lua

Inception-V3 as features extractor

The first way consists of fixing the parameters of the pretrain layers and to train only the last layer.

$ th src/main/classif/inceptionv3extraction.lua

The other way (often the usual way) consists of extracting features of a pretrain CNN on ImageNet and then to train a SVM. We only provide a code to extract features to a .csv.

$ th src/main/classif/extraction.lua

3. Extracting predictions from a CNN

Then we extract the log probabilities from a trained CNN on the training set, validation set and testing set.

$ th src/main/classif/extractpreds.lua \
   -pathnet experiments/resnet/datetime/net.t7 \
   -pathpreds experiments/resnet/datetime

4. Training a Hidden Markov Model (HMM)

Once we have extracted the log probabilities, we can train a Gaussian HMM on the training set and compute predictions on the validation set and testing set. We provide 4 temporal predictions :

  • cnn: no smoothing
  • cnn_smooth: smoothing on 5 frames
  • hmm: smoothing on 5 frames then smoothing with hmm in online mode
  • hmm_offline: same but in offline model


$ python src/main/temporal/
$ python src/main/temporal/

### 5. Evaluating predictions

Finally, to evaluate the temporal predictions, we must use the code given for the challenge. We can edit several variables: phaseGroundTruths, phaseGroundThruth and predFile.

$ matlab -nodisplay -nodesktop -r "run src/main/eval/Main.m"
You can’t perform that action at this time.