Design choice behind the default model

[jspaezp]

Hello there! It is me again!

I was wondering why the default model for the python API is the percolator (svm) model?
I noticed in the paper that xgboost out-performs the svm model in all depicted instances. Therefore, I wondered why is the default model in the python API is not xgboost?

Kindest wishes,
Sebastian

partially related: would you be interested in adding a section to the cookbook on how to use non-standard models with the package? I could make a PR for it in a couple of weeks.

[wfondrie]

Hi @jspaezp 👋

The reason I've decided to stick with the Percolator default are really two-fold:

1. The primary reason is that we know the default parameter search grid for the Percolator SVM works well across many datasets analyzed by many different search engines (and thus different features). I haven't done that kind of robustness testing with any other model and I want the default to be something that "just works".

2. At least in the case of XGBoost, making it the default would require an extra dependency. We've tried the Scikit-learn equivalent, the GradientBoostingClassifier, but I couldn't get it to work as well as XGBoost does. Additionally, we now have data that indicates that LightGBM may work even better 😄.

partially related: would you be interested in adding a section to the cookbook on how to use non-standard models with the package? I could make a PR for it in a couple of weeks.

Absolutely! 💯

[bittremieux]

Interestingly, the Percolator-like SVM also outperforms a (vanilla) random forest in recent tests, despite that the latter is generally considered to be a high-performing and robust classifier.

The tried-and-tested SVM seems to work pretty well out-of-the-box and is well understood, so I think it makes sense as the default model, rather than a more complex alternative.

[bittremieux]

This was indeed a counterintuitive result for us as well, but I don't have an immediate explanation.

[wfondrie]

I suspect this has to do with the hyperparameters that we optimize with the Percolator SVM, Cpos and Cneg. These are the relative cost of making an error on positive and negative examples,respectively. My hunch is that we may see improvement from trying to find the optimal class_weight for other sklearn-type models.

In the case of the mokapot paper, I attribute the improvement in the OMS setting to the additional delta mass feature, which a tree-based model is more well suited to utilize.

[jspaezp]

Looking a lil into it, it seems to be dataset dependent. phosphoproteomics seems to benefit more from tree based models, whilst plasma seems to work well but have high variance with a standard svg/glm (i removed the svg and rf for the grid search because it was too slow in the implementation I was using, will add them later). Maybe these kinds of data benefit more from feature stratification/cross-feature interaction. (x axis is different combinations of features, I will gladly send over the details but dont feel 100% confident uploading them to "the internets").

[jspaezp]

@wfondrie I have been thinking about that a lot, making the optimization problem translate directly could be an interesting approach (since the loss function is hard to define as "recall at 1% FDR" as a cost metric)

[bittremieux]

In ANN-SoLo we do a grid search over the same class weights for the random forest as for the SVM, but maybe that hyperparameter can indeed be more specific than just [0.1, 1, 10].

My intuition was also that for OMS a non-linear model would have been more useful, because it might learn different patterns corresponding to different modifications. But at least on our evaluation dataset, this turned out not to be the case. We should indeed test on different types of data, and I expect there to be differences in performance for sure.