Preliminary support for multi-objective optimization

[mstimberg]

The details of the syntax are still unclear, but here's a first attempt to get multi-objective optimization working, i.e. to make it possible to fit several variables at the same time (e.g. the membrane potential and a behavioural variable). This comes with a lot of restrictions at the moment, e.g. both variables have to be recorded with the same time step, but it should basically work. The main approach is the following (see examples/multiobjective.py for a full example):

1. During the creation of TraceFitter, specify a list of fitted variables (output_var) and a list of target traces (output), instead of a single one.
2. When calling TraceFitter.fit, provide either a single Metric (which will be used for both traces) or a list of Metric objects (this would allow you for example to ignore different parts of the respective traces by specifying t_start or t_weights), and a list/array of metric_weights that combines the two errors into a single one.

The combination of the errors currently ignores units, I don't quite know what an elegant way of handling this would be. Also note that:

• The callback only displays the total error
• TraceFitter.refine does not work with more than one fitted variable
• this obviously needs tests and documentation...

There seems to be some preliminary/work-in-progress support for multi-objective optimization in Nevergrad itself (see its documentation), but it also simply seems to combine the error into a single error, so not sure that this is very helpful.

[thesamovar]

Do you want feedback from me? If so, November OK?

[mstimberg]

Thanks, but I think @romainbrette will have a look, it's a feature that he needs :)

[romainbrette]

It looks nice, but we need to be able to use refine (gradient descent, right?). It would seem ok to combine into a single error actually, don't you think? (eg just have an L2 metric that is a weighted sum of errors on both variables).

Then another issue is that the behavioral variable is measured at 30 Hz while the electrophysiology at 40 kHz. A simple trick of course is to upsample the behavioral variable, that would work to some extent but it's not ideal. I suppose it's a matter of defining a metric that is applied on a series of time points (the trigger times of the camera). Does that make it complicated for the gradient descent maybe?

[mstimberg]

I feel like they are a few different issues here: yes, I want to make refine work with it, I just did not yet get around doing it. We are combining things into a single error, otherwise we could not use it with Nevergrad. I don't want to restrict it to L2 in general, because this seems unnecessarily restrictive and many users will use something else given that it does not work well for spiking neurons. For refine, this is not relevant since it always assumes an L2 metric due to the way lmfit works. Getting multiple variables to work with the symbolic calculation of the gradient (i.e. calc_gradient=True, not sure whether you are using that?) might be a bit more complicated, I did not look into that yet.

Regarding the sampling: this is a bit trickier. Currently, there is only a global dt, both for the simulation and the input/output variables. This is already too restrictive even for single variables, e.g. you might want to simulate with a smaller time step than your recordings for numerical reasons. The limitation that all dts are the same is currently hardcoded, but you are right that we could leave it up to the metric to deal with this. For refine that again needs a bit more extra work, but lmfit has a very straightforward approach to the residual and does not care about time points, etc. So e.g. to deal with multiple variables you can simply concatenate the residuals, and if those variables do not use the same dt, it does not care. We'd only have to scale the residuals correctly so that the variable with more points does not dominate the total error calculation.

[romainbrette]

Yes it was indeed for the gradient descent with symbolic calculation, that's the one that really works well.
The sampling thing would be useful but I think we can already work a bit without it (a matter of careful interpolation).

[mstimberg]

It's still a bit rough and needs more error checking, testing, etc., but refine should work now with multiple variables, including the calc_gradient option.

[romainbrette]

Great! I'll try it soon.

[mstimberg]

As discussed with @romainbrette, having both metric_weights and the normalization attribute of the Metric got a bit confusing. In the latest version I have therefore removed metric_weights, hence the normalization attribute is now the only way to scale variables against each other as well as getting the units consistent. The usual approach will be to have a normalization attribute of the same units as the variable itself. So e.g. in the multiobjective example I am now using two metrics for the variables v and m like this:

metric_v = MSEMetric(t_start=5*ms, normalization=10*mV)
metric_m = MSEMetric(t_start=5*ms, normalization=0.1)

I think this is fairly intuitive: it means that a 10mV difference in the membrane potential should be comparable to a 0.1 difference in the gating variable m.

Another recent change: the individual errors of each objective are now accessible as part of the data structures returned by Fitter.results. The final "best" errors before and after normalization are also accessible as Fitter.best_objective_errors and Fitter.best_objective_errors_normalized in the form of dictionaries mapping variable name to error.

From my side, this finishes the basic work on the features for this PR. If no one runs into further issues I'll clean up the code and add tests and documentation.

[romainbrette]

It's nice. I wonder whether it would then makes sense to use dictionaries instead of lists. eg in TraceFitter we have output_var and output, it could simply be output = {'v' : ..., 'v2' : ...}. In fit() metrics is a list.

[mstimberg]

I had the same thought, but wondered whether there are maybe some use cases like using both MSEMetric and FeatureMetric for the same variable which you currently could do by specifying output_var = ['v', 'v']. But then of course this wouldn't work for things like best_objective_errors... Maybe it's not a real use case (one could specify something like MSEMetric as part of FeatureMetric).

[romainbrette]

Ah I didn't think about that. Wouldn't be better in this case to have instead a combined metric class? (CombinedMetric(MSEMetric,FeatureMetric) or even MSEMetric+FeatureMetric with operator overloading).

[mstimberg]

Yes, I think something like this would make more sense. In a way the FeatureMetric is already a combined metric, but it only supports features handled by eFEL.

[mstimberg]

I think this is good enough to merge as it is now. @romainbrette I hope I did not break anything for you with my latest changes? Anything else (related to multiobjective fitting) that I forgot to implement?

[romainbrette]

It runs!

[mstimberg]

It runs!

Great, thanks for checking. Your current code probably gets a warning about if you use a list of metrics, this should now be a dictionary (like output). I'll merge the branch now, don't get confused if you try to update it the next time ;)