tybalt-with-candle

This repository shows how to run a grid search hyperparameter optimization of Tybalt using CANDLE.

The repository starts tracking from the exact files located in /usr/local/apps/Tybalt/20210709/src as of about 11:50 AM on 8/23/21 and shows step-by-step (see the commit log) how to run them using CANDLE.

These steps are inferred directly from the guide to CANDLE on Biowulf; see here in particular for a summary of the steps.

In summary, here are the steps required to run HPO on Tybalt using CANDLE:

1. Remove requirement of having the type of model as a command line argument (it will simply be a hyperparameter) since we currently need to be able to run the model script like python MY_MODEL_SCRIPT.py with no arguments. Files affected: options.py and train.py
2. Define the hyperparameters in the model script train.py. File affected: train.py
3. Define the performance metric in this case by assigning the output of the Keras fit() method to a variable named history. File affected: train.py
4. Perform the copy step from the Tybalt Biowulf help page in the model script itself since each hyperparameter set will generate its own different checkpoints. File affected: train.py
5. Add a CANDLE input file; this file can be put in any directory, optionally an empty one, and be run using candle submit-job candle_grid_search_for_tybalt.in. File affected: candle_grid_search_for_tybalt.in
6. Add a CSV file of the full grid HPO results, which was generated using candle aggregate-results $(pwd)/last-candle-job. This job was run using 12 GPUs (nworkers=12) and completed in 13 minutes. See last-candle-job/run/*/subprocess_out_and_err.txt for the output from each hyperparameter set. File affected: candle_results.csv

Notes for Gennady:

• As seen in the steps above, it is very easy to use CANDLE. It is not just for advanced users and can certainly be (and has been) used by beginners!
• You will not run into any issues if you follow the steps above, which come straight from the CANDLE documentation.
• In particular, there are no issues or conflicts at all with the Python libraries or versions. E.g., your code does not need to depend on python/3.7 in any way, nor do you need to know the Python packages on which CANDLE depends.
• I think you could write a simple wrapper around this in order to present it clearly to your class as you previously tried to do. I would be happy to help with this or with anything else!

Gennady, please let me know ((240) 276-5891 or andrew.weisman@nih.gov) if you have any particular questions or if something is still unclear!