cs395t project1: implementing yearbook project.
- More information at: http://www.philkr.net/cs395t/project1/
- Report in report.pdf
- Short presentation in cs395t-F17-proj1-Amin_Diego_Farzan.pdf
- python (tested with python3.5)
- matplotlib + basemap
Project Folder Structure
data yearbook train F 000001.png ... M 000001.png ... valid ... test ... yearbook_train.txt yearbook_valid.txt yearbook_test.txt model model source code and final model src grade.py run.py util.py output yearbook_test_label.txt
Download the data from the link and store it in data folder as described in the folder structure.
Train the model and put the model in the
Running the evaluation
It will give the result based on the baseline 1 which is the median of the training image.
- For yearbook validation data
cd src && python grade.py --DATASET_TYPE=yearbook --type=valid
Generating Test Label for project submission
- For yearbook testing data
cd src && python grade.py --DATASET_TYPE=yearbook --type=test
- Put model and generated test_label files in their respective folder.
- Remove complete data from the data folder.
- Add readme.md file in your submission.
- Project should be run from the grade.py file as shown in the evaluation step and should be able to generate the test_label file.
STEP A: installing anaconda on ubuntu
- curl -O https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
- bash Anaconda3-4.2.0-Linux-x86_64.sh
- source ~/.bashrc
STEP B: create conda environment and get all the dependencies
- conda create --name tf python=3
- source activate tf
- conda install keras
- conda install matplotlib
- conda install pandas
- conda install -c anaconda tensorflow-gpu
STEP C: Train our models using GPUs with "screen" so that you may leave the process running in the background ##https://askubuntu.com/questions/8653/how-to-keep-processes-running-after-ending-ssh-session
- screen -S diego395
If USING microdeep server run either 11a for one experiment or 11b for multiple ( both from within model/ folder ) 11a) CUDA_VISIBLE_DEVICES=3 python fine-tune.py --data_dir="../data/yearbook" --input_dir="train" --valid_dir="valid" --train_file="yearbook_train.txt" --valid_file="yearbook_valid.txt" --model_name="inceptionv3"
11a) CUDA_VISIBLE_DEVICES=3 python run_experiments.py
ctrl-A ctrl-D #to detach process and leave it running in background screen -r diego395 #to get back to screen session to check on process
nvidia-smi # to see gpu usage on microdeep top # to see cpu usage on microdeep
How ro predict based on a model and a dataset (train or valid)
- run the following from the model directory:
python fine-tune.py --make_prediction="yes" --pred_dataset="train" --pred_model="m_2017-10-06_02:10_inceptionv3_categorical_crossentropy_adam_lr0.001_epochs50_regnone_decay0.0_ft.model"
- what this does is that it stores y_pred_train.csv and y_true_train.csv in the plot directory. if you run it for test it will produce the corresponding predictions
- the model should be in fitted_models
- depending on the dataset you are interested in, you should choose train or valid
- the make_conf_mat should be yes if you want to make predictions
- note that depending on which model you are using the prediction for, you must provide the size of the input image as pred_target_size exactly in the format of the example above
- if you want to store the predictions for multiple models put the current predictions in another folder and rename them cause they will be overwriteen
- The next step is to use the model/plot/conf_matrix.ipynb for post processing
once you are in the ipython, you need to specify the dataset of interest, if the predictions for that dataset are stored, it will read it and find confusion matrix and plot it. It will also produce min, mean, max L1