Does the joint model suffer from issues related to overfitting?

[ec-ho-ra-mos]

Hello!

Before asking my question, let me first run what I understand about the static model and the joint model from the mokapot JPoS paper.

• A static model uses an external training dataset to learn a model. So essentially, for training, it uses a large dataset X, to evaluate a small-scale experiment-of-interest Y.
• On the other hand, a joint model uses the aggregate of all the PSM in the dataset to assign a confidence score within each experiment. So, from what I understand, for training, it uses the combination of experiments A, B, C, D, E, F, G, and H as a single training dataset. Then it uses that same learned model to individually evaluate the same dataset. (e.g. A -> B -> C -> D -> E -> F - > G).

My question now is, wouldn't this cause some kind of overfitting issue especially if the number of experiments used for training the joint model is not big enough?

Thank you very much for your attention to this question.

[wfondrie]

Hi @ec-ho-ra-mos,

Thanks for giving mokapot a try and the question!

The short answer: No, using a joint model should result in no more overfitting than a standard Percolator/mokapot analysis.

The long answer:
In the joint modeling procedure, PSMs from multiple experiments are aggregated together into one large dataset. After this aggregation step, each PSM is assigned to a cross-validation split where multiple PSMs for the same mass spectrum are guaranteed to be in the same split. Multiple models are then trained using a cross-validation procedure, whereby one model is training on all but one split. The trained models are used to re-score only PSMs that were in the held-out for each model. Note that this cross-validation procedure is identical to the one used by Percolator, described by Granholm et al. Finally, the PSMs from each experiment are separated again and confidence estimates are assigned within each experiment.

The purpose of this approach is to be able to leverage data from multiple experiments, which may be small individually, to reduce the variance of the learned model. If each experiment is itself fairly large, the results from a joint model and modeling each experiment individually should converge.

Note that the static modeling approach tackles a similar issue, but in a different context. It assumes we have a large amount of prior knowledge that we can use to build a model, which is often the case. We can then use the model learned from prior data to analyze new experiments without cross-validation, because there is no additional model fitting. Learning a model from large amounts of prior data also serves to reduce the variance of the learned model.

[ec-ho-ra-mos]

Hello @wfondrie,

Thank you very much for your reply. I have read the cross-validation scheme by V. Granholm et al. and found that this paper, with your reply, answered my question. Thank you very much again.