Tune DNN hyperparameters

[bfhealy]

The config file contains hyperparameters that may be adjusted for each class to attain optimal DNN performance. We should experiment with these values to determine an appropriate combination for each class. The hyperparameters include:

-threshold specifying the minimum probability to determine positive examples of a class
-balance specifying the ratio of over-to-underrepresented examples in a class
-weight_per_class boolean; if True the loss function responds differently to pos. vs. neg. class examples based on number

The architecture of the DNN is also controlled by hyperparameters (number of neurons, dropout fractions, convolution windows, etc). For now I think we should maintain the DNN structure as-is (consistent with the published architecture) and experiment with the above values in the config file. An alternative architecture is shown in panel (b) of Fig. 8, but with the warning that this architecture tends to have greater variance than the current model.

[bfhealy]

Done for this round of active learning, see #181 and config.defaults.yaml for the current hyperparameter configurations.

[bfhealy]

Reopening to obtain better model performance on inference fields.

[bfhealy]

Closing again; currently using the full training set for each classifier rather than limiting sources with the balance hyperparameter. This reduces recall but gives good precision for several classes. weight_per_class is currently False - while this hyperparameter can improve recall, on the full dataset it comes with a drastic reduction in precision for highly imbalanced classes.

[mcoughlin]

@bfhealy In principle we can turn this on or off at the class level?

[bfhealy]

@mcoughlin Yes, the config file allows us to customize these parameters for each class going forward.

[mcoughlin]

@bfhealy cool!