This repo will host all the materials related to deep Learning Seminar Projects
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
data
model
output
plots
src
.gitignore
README.md
cs395proj1_dev_work.ipynb
cs395t-F17-proj1-Amin_Diego_Farzan.pdf
report.pdf

README.md

Yearbook

cs395t project1: implementing yearbook project.

Dependencies

  • python (tested with python3.5)
  • numpy
  • sklearn
  • matplotlib + basemap
  • keras

Project Folder Structure

data
	yearbook
		train
			F
				000001.png
				...
			M
				000001.png
				...
		valid
			...
		test
			...
		yearbook_train.txt
		yearbook_valid.txt
		yearbook_test.txt
model
	model source code and final model
src
	grade.py
	run.py
	util.py
output
	yearbook_test_label.txt

Evaluation

Data setup

Download the data from the link and store it in data folder as described in the folder structure.

Models

Train the model and put the model in the Model folder

Running the evaluation

It will give the result based on the baseline 1 which is the median of the training image.

  1. For yearbook validation data
cd src &&  python grade.py --DATASET_TYPE=yearbook --type=valid

Generating Test Label for project submission

  1. For yearbook testing data
cd src &&  python grade.py --DATASET_TYPE=yearbook --type=test

Submission

  1. Put model and generated test_label files in their respective folder.
  2. Remove complete data from the data folder.
  3. Add readme.md file in your submission.
  4. Project should be run from the grade.py file as shown in the evaluation step and should be able to generate the test_label file.

cs395proj1_install_notes.txt

STEP A: installing anaconda on ubuntu

https://www.digitalocean.com/community/tutorials/how-to-install-the-anaconda-python-distribution-on-ubuntu-16-04

  1. curl -O https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
  2. bash Anaconda3-4.2.0-Linux-x86_64.sh
  3. source ~/.bashrc

STEP B: create conda environment and get all the dependencies

  1. conda create --name tf python=3
  2. source activate tf
  3. conda install keras
  4. conda install matplotlib
  5. conda install pandas
  6. conda install -c anaconda tensorflow-gpu

STEP C: Train our models using GPUs with "screen" so that you may leave the process running in the background ##https://askubuntu.com/questions/8653/how-to-keep-processes-running-after-ending-ssh-session

  1. screen -S diego395

If USING microdeep server run either 11a for one experiment or 11b for multiple ( both from within model/ folder ) 11a) CUDA_VISIBLE_DEVICES=3 python fine-tune.py --data_dir="../data/yearbook" --input_dir="train" --valid_dir="valid" --train_file="yearbook_train.txt" --valid_file="yearbook_valid.txt" --model_name="inceptionv3"

11a) CUDA_VISIBLE_DEVICES=3 python run_experiments.py

ctrl-A ctrl-D #to detach process and leave it running in background screen -r diego395 #to get back to screen session to check on process

nvidia-smi # to see gpu usage on microdeep top # to see cpu usage on microdeep

How ro predict based on a model and a dataset (train or valid)

  1. run the following from the model directory:

python fine-tune.py --make_prediction="yes" --pred_dataset="train" --pred_model="m_2017-10-06_02:10_inceptionv3_categorical_crossentropy_adam_lr0.001_epochs50_regnone_decay0.0_ft.model"

NOTE:

  • what this does is that it stores y_pred_train.csv and y_true_train.csv in the plot directory. if you run it for test it will produce the corresponding predictions
  • the model should be in fitted_models
  • depending on the dataset you are interested in, you should choose train or valid
  • the make_conf_mat should be yes if you want to make predictions
  • note that depending on which model you are using the prediction for, you must provide the size of the input image as pred_target_size exactly in the format of the example above
  • if you want to store the predictions for multiple models put the current predictions in another folder and rename them cause they will be overwriteen
  1. The next step is to use the model/plot/conf_matrix.ipynb for post processing

once you are in the ipython, you need to specify the dataset of interest, if the predictions for that dataset are stored, it will read it and find confusion matrix and plot it. It will also produce min, mean, max L1