Demorun 2 Running small training and prediction with mito testsample dataset

Training with test dataset on Cloud

The following instructions use the mito_testsample dataset stored with this repository as an example on how to perform the entire process from training through prediction. Although this training dataset is too small to offer good segmentation, it is quick to process so one can try all the steps without long periods of waiting.

To just try prediction with a pre-trained mitochondria model click here

Step 1) Connect via ssh to instance created by these instructions

ssh ubuntu@PublicDNS_VALUEFROM_WEBPAGE_ABOVE

Quick and Dirty run

Here are the commands to run once logged into the instance. Below in Steps 2+ are more detailed instructions on what is happening.

cd ~
PreprocessTrainingData.m ~/cdeep3m/mito_testsample/training/images/ ~/cdeep3m/mito_testsample/training/labels/ ~/mito_testaugtrain
runtraining.sh --numiterations 100 ~/mito_testaugtrain ~/train_out
runprediction.sh ~/train_out/ ~/cdeep3m/mito_testsample/testset/ ~/predictout
# results are in ~/predictout/ensembled
ls ~/predictout/ensembled

Step 2) Run PreprocessTrainingData.m

More information about PreprocessTrainingData.m can be found here

cd ~
PreprocessTrainingData.m ~/cdeep3m/mito_testsample/training/images/ ~/cdeep3m/mito_testsample/training/labels/ ~/mito_testaugtrain

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Starting Training data Preprocessing
Training Image Path:
/home/ubuntu/cdeep3m/mito_testsample/training/images/
Training Label Path:
/home/ubuntu/cdeep3m/mito_testsample/training/labels/
Output Path:
/home/ubuntu/mito_testaugtrain
Loading:
/home/ubuntu/cdeep3m/mito_testsample/training/images/
Image importer loading ...
/home/ubuntu/cdeep3m/mito_testsample/training/images/
Reading file: /home/ubuntu/cdeep3m/mito_testsample/training/images/images.010.png
.
.
/home/ubuntu/cdeep3m/mito_testsample/training/labels/
Image importer loading ...
/home/ubuntu/cdeep3m/mito_testsample/training/labels/
Reading file: /home/ubuntu/cdeep3m/mito_testsample/training/labels/mitos_3D.010.png
.
.
Create variation 8 and 16
Saving: /home/ubuntu/mito_testaugtrain/training_full_stacks_v8.h5
Elapsed time is 15.1134 seconds.
-> Training data augmentation completed
Training data stored in /home/ubuntu/mito_testaugtrain
For training your model please run CreateTrainJob.m /home/ubuntu/mito_testaugtrain <desired output directory>

Step 3) Run runtraining.sh

Run runtraining.sh to train CDeep3M and created trained models.

runtraining.sh --numiterations 100 ~/mito_testaugtrain ~/train_out

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Verifying input training data is valid ... success
Copying over model files and creating run scripts ... success

A new directory has been created: /home/ubuntu/train_out
In this directory are 3 directories 1fm,3fm,5fm which
correspond to 3 caffe models that need to be trainedas well as two scripts:

caffe_train.sh -- Runs caffe for a single model
run_all_train.sh -- Runs caffe_train.sh serially for all 3 models

Single GPU detected.

Training has completed. Have a nice day!
Training has completed. Results are stored in /home/ubuntu/train_out
Have a nice day!

Step 4) Run runprediction.sh

runprediction.sh ~/train_out/ ~/cdeep3m/mito_testsample/testset/ ~/predictout

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Starting Image Augmentation
Check image size of: 
/home/ubuntu/cdeep3m/mito_testsample/testset/
Reading file: /home/ubuntu/cdeep3m/mito_testsample/testset/images.081.png
warning: your version of GraphicsMagick limits images to 16 bits per pixel
warning: called from
    imformats>default_formats at line 256 column 11
    imformats at line 79 column 3
    imageIO at line 106 column 11
    imfinfo at line 159 column 8
    check_image_size at line 43 column 18
    /home/ubuntu/cdeep3m/DefDataPackages.m at line 40 column 11
z_blocks =

   1   5


Running Prediction

Trained Model Dir: /home/ubuntu/train_out/
Image Dir: /home/ubuntu/cdeep3m/mito_testsample/testset/
Models: 1fm,3fm,5fm
Speed: 1

Running 1fm predict 1 package(s) to process
  Processing Pkg001_Z01 1 of 1 octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Starting Image Augmentation
z_stack =

   1   5

Image importer loading ... 
/home/ubuntu/cdeep3m/mito_testsample/testset/
Reading file: /home/ubuntu/cdeep3m/mito_testsample/testset/images.081.png
warning: your version of GraphicsMagick limits images to 16 bits per pixel
warning: called from
    imformats>default_formats at line 256 column 11
    imformats at line 79 column 3
    imageIO at line 92 column 11
    imread at line 106 column 30
    imageimporter_large at line 52 column 31
    /home/ubuntu/cdeep3m/PreprocessPackage.m at line 62 column 8
Reading file: /home/ubuntu/cdeep3m/mito_testsample/testset/images.082.png