FAQ
To help Keras users, without changing their current workflow, develop well generalizing hyperparameter configurations that yield state-of-the-art prediction results for any prediction challenge.
Talos is built from the first line of code with one strategic objective in mind; to provide an abstraction layer for the next progression in artificial intelligence workflow. Whereas optimization solutions generally focus on a given optimization problem as an end in itself, Talos is built as a building block that allows optimizing itself. In other words, allowing hyperparameter optimization optimization.
Replace the hyperparameter inputs (e.g. 'adam') with a reference to a dictionary (e.g. params['optimizer']) is the only notable change. See the Iris Example Model for a better understanding how Talos can be incorporated to your Keras workflow.
Talos is built 100% focused on Keras models. This allows us to provide a system that appreciates Keras workflow in the way of changing the way you work with your models the absolute minimum.
Because of the way Keras wraps several backends, including Theano and Keras, it's possible to use any backend supported by Keras. Note that there is one Tensorflow specific parameter (for clearing the tf graph) which would obviously not work.
While the user is free to use any Keras metric, or custom built model, part of the effort with Talos is to create a single metric that can be used for as an objective measure for all kinds of prediction tasks. Appreciating the fact that it might not be possible to objectively compare a prediction capability for continuous and categorical tasks, a less optimal solution is where there are two objective metrics. One for continuous prediction tasks, and one for all kinds of categorical prediction tasks (single label, multi-label, and multi-class). Why is this important? More about this below.
Yes, you can use custom metrics just as you would use them in Keras. The important thing to notice here is that you have to include the string 'acc' in the name of the metric in order for peak epoch being recorded in the right way.
Depending on the type of experiments, one might accumulate thousands or even millions of observations in the log. Each observation consist of the hyperparameter configuration, type of prediction challenge, type of features (not implemented yet) and the objective measure. These are stored for the purpose of allowing learning later, both through human analysis, and as a data to guide hyperparameter boundary selection and other parts of the hyperparamater optimization process.