Information criteria for a stochastic model come from a single noisy simulation

[wshlavacek]

What happens now

Results/information_criteria.txt reports AIC, BIC and AICc for the best fit. The log likelihood behind those numbers comes from re-simulating the best fit parameter set exactly once, in Algorithm._compute_information_criteria in pybnf/algorithms/base.py. Results/information_criteria_backup.txt is written the same way, by _checkpoint_information_criteria.

When the model is stochastic, that one simulation is a draw from a distribution rather than a fixed answer. So the reported AIC is a noisy number. Run the same fit twice and the same parameter set gets two different AIC values, by an amount that has nothing to do with the model and everything to do with which trajectory the simulation happened to take.

This only affects a fit whose objective is a proper likelihood, since no information criterion is computed for anything else.

Why this is a problem

The whole point of an information criterion is to compare models. A difference of 2 in AIC is treated as meaningful in the literature. If the noise in a single stochastic simulation moves the log likelihood by more than that, the comparison is measuring the simulation rather than the model, and nothing in the output says which.

Minimal repro

Take any model with a stochastic simulation method and a likelihood objective, for example:

edition = 2
job_type = de
noise_model = normal, sigma = 1

with an experiment line whose method is ssa or nf. Run the fit twice with the same random seed. Results/information_criteria.txt reports a different log_likelihood, and so a different AIC, both times, for a parameter set that did not change.

What would need deciding

The obvious fix is to simulate the best fit several times and average, which is what the new best fit confirmation stage does for the objective value (#659, PR #675). It is not obvious what to average here. The mean of the log likelihoods over replicates and the log of the mean likelihood are different numbers, and they answer different questions. The second is the one that matches what a likelihood means for a model with a stochastic simulator, and it is the harder one to estimate well.

So this needs its own decision about what quantity to report, before any code. It also needs a measurement of how large the effect actually is on a real model, which the table the new stage writes now makes easy to get.

Note

After #659 lands, a stochastic fit reports an averaged objective value in Results/best_fit_confirmation.txt while Results/information_criteria.txt still reports a single draw for the same parameter set, so the two files will visibly disagree.