Wrapper integrating Mri and caffe-cnn
Python Rebol
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
example
.gitignore
Dockerfile
README.md
config.template
config.txt.template
generate_tasks.py
main.py
parseMRI.py
subPlaceholder.R

README.md

Incooperate Mri and Caffe-cnn to perform hyper-parameter search.

Dependencies

Quick run

We provide some toy data on which you can quickly perform hyper-parameter searching, training and testing.You need to run it under the repository directory.

docker pull haoyangz/mri-wrapper
docker run -v $(pwd)/example:/data -v $(pwd)/example/model:/model -i --rm \
--device /dev/nvidiactl \
--device /dev/nvidia-uvm \
--device /dev/nvidia0 \
haoyangz/mri-wrapper python main.py /model/runparam.list 0

This is what happens if it runs:

  • Search for the best hyper-parameters for a toy Caffe model on some toy training and validation data.

  • Pick the set of parameters with best performance, and try a few times to train on the training data. The trial and iteration where the model reaches the best performance on the validation set will be picked for testing and prediction.

  • Make prediction with the best model on the toy testing data. The output will be under $REPO_HOME$/example/16_G/mri-best/best_trial/

  • Make prediction with a pre-trained model (specified by example/newdata/deploy.txt and example/newdata/bestiter.caffemodel) to predict on new data specified in example/newdata/test.txt. The output will be saved under example/newdata_output.

Data preparation

Follow the instruction in Caffe-cnn to prepare data under $DATADIR$/data.

Model preparation

All of the files in this part should be saved under $MODELDIR$.

Caffe model files

Follow the instruction in Caffe-cnn to prepare caffe model files:

  • trainval.prototxt : training architecture
  • solver.prototxt: solver parameter
  • deploy.prototxt: testing architecture

The only differences:

  • In trainval.prototxt, the "source" in the input layer should be /data/data/train.txt and /data/data/valid.txt
Modify model files for hyper-parameter search

For each hyperparameter to tune, replace the value with %{param}% instead. For example:

In solver.prototxt:

delta: %{delta}%

In trainval.prototxt / deploy.prototxt:

layer {
  name: "drop1"
  type: "Dropout"
  bottom: "fc1"
  top: "fc1"
  dropout_param{
    dropout_ratio: %{dropout_ratio}%
  }
}

Note that every parameter changed in trainval.prototxt should also be modified accordingly in deploy.prototxt

Specify value choices of hyper-parmeters

In "hyperparams.txt" file.

Specify model name

In "modelname" file.

Prepare runparam.list

General argument:

  • ORDER: The following orders are accepted. Multiple orders should be concatenated by comma, and the relative order should be maintained.
    • trainMRI: hyper-parameter tuning
    • update: pick the best hyper-parameter
    • trainCaffe: train on the training set with the best hyper-parameter
    • testCaffe: predict on the test set with the trained model
    • testEvalCaffe: evaluate the performance on test set
    • predictCaffe: predict with a pre-trained model to predict on new data

If you want to perform hyper-parameter tuning, training and testing:

  • MRI_MAXITER: The number of hyper-parameter setting to try.
  • search_max_iter: Maximum number of iteration in hyper-parameter search.
  • search_test_interval: The iteration interval to test on validation set in hyper-parameter search.
  • search_snapshot: The iteration iterval to save model in hyper-parameter search.
  • search_display: The iteration interval to display the training loss in hyper-parameter search.
  • train_trial: After the best hyper-params are chosen, we will train this number of times and pick the best trial.
  • debugmode: The verbosity of log ('NONE'<'INFO'<'DEBUG'). We recommend 'INFO' in most case.
  • optimwrt: Choose the best param and training trial with respect to either accuracy or loss
  • outputlayer: The name of the output layer that will be used as prediction in test phase.

If you want to predict on new data with a trained model:

  • outputlayer: Same as above.
  • deploy2predictW: path to the deploy.prototxt file of the trained model, which specifies the network architecture.
  • caffemodel2predictW: path to the snapshot of the trained model, which ends with '.caffemodel' and during training will be saved every snapshot number of iteration as specified in solver.prototxt.
  • data2predict: path to the manifest file specifying all HDF5 format data to predict on.
  • predict_outdir: the output directory, under which 'bestiter.pred' and 'bestiter.pred.params.pkl' will be saved. See description for outputlayer.

As Mri-wrapper lives in the docker container, it can only access the directory mounted to the container. For instance, in the toy example above, $(pwd)/example is mounted as /data and $(pwd)/example/model is mounted as /model. Therefore all the above path parameters should be relative to one of the mounted directory. But you can mount as many directory as you want with '-v' option as in the toy example.

Ready to go!

docker pull haoyangz/mri-wrapper
docker run -v DATADIR:/data -v MODELDIR:/model -i --rm \
	--device /dev/nvidiactl --device /dev/nvidia-uvm MOREDEVICE \
	haoyangz/mri-wrapper python main.py /model/runparam.list GPUNUM
  • DATADIR: The topfolder of the output. Your data should be under $DATADIR$/data.

  • MODELDIR: The folder containing the model files, hyperparam file and modelname file

  • MOREDEVICE: For each of the GPU device available on your machine,append one "--device /dev/nvidiaNUM" where NUM is the number of that device. For hsf1/hsf2 in Gifford Lab, since there are three GPU, it should be :

--device /dev/nvidia0 --device /dev/nvidia1 --device /dev/nvidia2

+ `GPUNUM`: The GPU device number to run


For hyper-parameter tuning, training and testing, all the output will be generated under *$DATADIR$/$thenameofthemodel$*, where `$thenameofthemodel$` is the name specified in the file [*$MODELDIR$/modelname*](https://github.com/gifford-lab/mri-wrapper/blob/master/example/model/modelname) .

After training, the information of the best model will be saved as *bestiter.caffemodel* and *deploy.prototxt* under  *$DATA_DIR$/$thenameofthemodel$/mri-best/best_trial/*.

For testing and prediction, the output of the first layer specified in `outputlayer` (see above section on params.list) will be saved to a text file named *bestiter.pred* and the output of all layers will be saved in a Pickle object named *bestiter.pred.params.pkl*. For testing, they will be both under *$DATA_DIR$/$thenameofthemodel$/mri-best/best_trial/*. For prediction, they will be under `predict_outdir` as specified in [runparams.list](https://github.com/gifford-lab/mri-wrapper/blob/master/example/model/runparam.list).