TRAIN output_options
As a result of training, several review files are created in addition to the trained model file. The following options allow a user change the default output settings pf model training.
Many of the options below allow for formatting tags, whereby certain keywords in curly braces get replaced by their corresponding variable value. A common one for example is {TRAINING_ID}
, where the training id specified by the 3rd neuston_net TRAIN positional argument (neuston_net.py TRAIN SRC MODEL TRAINING_ID
) replaces the variable name and curly braces at runtime.
Output Options:
--outdir OUTDIR Default is "training-output/{TRAINING_ID}"
--model-id ID Set a specific model id. Patterns {TRAIN_DATE} and {TRAIN_ID} are recognized. Default is "{TRAIN_ID}"
--epochs-log ELOG Specify a csv filename. Includes epoch, loss, validation loss, and f1 scores.
Default is "epochs.csv".
--args-log ALOG Specify a human-readable yml output filename containing all user-specified
and default training parameters. Default is "args.yml".
--onnx Additionally output an onnx version of the model.
--results FNAME [SERIES ...]
FNAME: Specify a validation-results filename or pattern.
Valid patterns are: "{epoch}". Accepts .json .h5 and .mat file formats.
SERIES: Data to include in FNAME file.
The following are always included and need not be specified:
model_id, timestamp, class_labels, input_classes, output_classes.
Options are: image_basenames image_fullpaths
output_scores output_winscores
confusion_matrix
classes_by_{count|f1|recall|precision}
{f1|recall|precision}_{macro|weighted|perclass}
{counts|val_counts|train_counts}_perclass
--results may be specified multiple times in order to create different files.
If not invoked, the default options are:
FNAME = results.mat
SERIES = image_basenames output_scores counts_perclass
confusion_matrix f1_perclass f1_weighted f1_macro
OUTDIR
specifies the output directory for all media produced by a neuston_net TRAIN. It defaults to "training-output/{TRAINING_ID}". If the directory path is not an absolute path, the directory path is considered to be relative to the current working directory neuston_net.py was invoked from. If the directory specified doesn't exist, it will be created.
Available formatting tags are:
-
{TRAIN_DATE}
- the date training was initiated in "YYYY-MM-DD" format. -
{TRAINING_ID}
- the training id specified by the 3rd neuston_net TRAIN positional argument (neuston_net.py TRAIN SRC MODEL TRAINING_ID
)
The filename as well as the internally stored name of the output model. Model ID can further be used as a formatting tag during neuston_net RUN.
By default this is "{TRAIN_DATE}__{TRAINING_ID}". This will result in a model file with a filename such as 2021-12-21__SomeParticularTrainingName.ptl
Available formatting tags are:
-
{TRAIN_DATE}
- the date training was initiated in "YYYY-MM-DD" format. -
{TRAINING_ID}
- the training id specified by the 3rd neuston_net TRAIN positional argument (neuston_net.py TRAIN SRC MODEL TRAINING_ID
)
By including this flag, an .onnx version of the model will be created alongside the .ptl model. ONNX is a widely-supported open-source format for AI/ML model inferencing.
-
args.yml
- The Args log is a yaml file that captures the input parameters of a training run. The filename can be changed using the--args-log
flag. -
epochs.csv
- The Epochs log is a csv file that captures the performance (loss for training and validation datasets) of a model across its training epochs. The filename can be changed using the--epochs-log
flag.
Besides the model and the log files, one (or more) results file is produced by neuston_net TRAIN. A results file features the classification output of the model's best-epoch against the Validation dataset, as well as many other possible summary statistics. The output flag follows the following pattern:
--results FNAME [SERIES ...]
Where FNAME is result file's filename, and SERIES is a list of possible values and statistics to include.
FNAME recognizes .mat
, .h5
and .json
as viable output formats (matlab, hdf5, and json respectively). Additionally, {epoch}
is recognized as a valid formatting tag; this allows for the results of multiple epochs to later be compared.
These Series are always included in a results file
-
model_id
- the MODEL-ID of the trained model -
timestamp
- the timestamp the model training was initiated -
class_labels
- the list ofm
class labels. The order of this list is consistent with all further per-class list orderings -
input_classes
- the list ofn
true input classes, as integer indices corresponding to its respectiveclass_labels
label - output_classes - the list of
n
predicted output classes, also as integer indices
These Series must be specified in order to be included in a results file when --results
is manually invoked.
-
image_basenames
- list ofn
image filenames as determined bySRC
and runtime dataset configurations -
image_fullpaths
- as above, except the images' fullpaths are used -
output_scores
- Ann
-by-m
matrix of scores, wheren
is number of inputs/images, andm
is the number of classes. Values for a given image are softmaxed such that their sum equals 1; higher values indicate a higher classification confidence score from the model. -
output_winscores
- A list of the highest classification confidence score for each image. -
confusion_matrix
- Anm
-by-m
matrix comparing image true labels vs. predicted labels. -
classes_by_{count|f1|recall|precision}
... - list of classes as ordered by count, f1 score, recall score, or precision score.classes_by_count
classes_by_f1
classes_by_recall
classes_by_precision
-
{f1|recall|precision}_{macro|weighted|perclass}
- f1, recall, or precision score by macro averaging, weighted averaging, or perclass ("samples" averaging mode in sklearn). See sklearn.metrics.f1_score for more details.f1_macro
f1_weighted
f1_perclass
recall_macro
recall_weighted
recall_perclass
precision_macro
precision_weighted
precision_perclass
-
{counts|val_counts|train_counts}_perclass
- list of counts (total, or for the training or validation dataset) on a per-class basis.counts_perclass
val_counts_perclass
train_counts_perclass
It is possible to invoke the --results
flag multiple times, each time allowing the user to specify a new results FNAME and set of SERIES. This may be used to output multiple different file formats at the same time, and-or to specify a different set of series for different files.
The flags --results short_results.json --results full_results.h5 image_fullpaths output_scores
would result in two files being generated.
-
short_results.json
- only contains model_id, timestamp, class_labels, input_classes, output_classes (the always-included series) -
full_results.h5
- would additionally contain the largen
-by-m
matrix of scores and then
-length list of image fullpaths.
If --results
is not specified at runtime, by default neuston_net TRAIN acts as though it were specified as --results results.mat image_basenames output_scores counts_perclass confusion_matrix f1_perclass f1_weighted f1_macro
.
These files do no have manual/flaggable options.
-
training_images.list
- list of training images -
validation_images.list
- list of validation images -
chkpts/
- a directory of best-epoch checkpoints. The checkpoint files are interoperable with the final .ptl file. -
logs/default/version_0
- contains runtime versions ofargs.yml
andepochs.csv
(namedhparams.yaml
andmetrics.csv
respectively here). Upon completion of a Training, these two files are copied and renamed into the mainOUTDIR
directory. If a training with the sameOUTDIR
directory is started, the "version" number is incremented and a new version directory is created. Other files in theOUTDIR
directory will be overwritten.
- Overview
- Installation
- local
- whoi hpc
- Training a Model
- Running a Model
- Utilities
- SLURM SBATCH Tool ⊛
- Dupes Training ⊛
- Tips
- HPC Patch Notes