ProteoBench Goes Predictive: Extending community-driven benchmarking to prediction models

[Cajac102]

Title

ProteoBench Goes Predictive: Extending community-driven benchmarking to prediction models

Abstract

Benchmarking can make or break the adoption of computational proteomics methods, but today's benchmark studies are often difficult to reproduce and quickly become outdated. ProteoBench is an open EuBIC-MS platform for community-driven and continuous benchmarking of proteomics data analysis workflows, currently covering tasks such as label-free quantification, de novo identification, and FDR assessment. A central goal of ProteoBench is to keep benchmarking accessible to everyone: users should be able to contribute results with limited technical overhead, while maintaining rigorous and fair benchmarking.

This hackathon aims to extend that philosophy to benchmarking predictive models used in proteomics workflows, for example models for retention time, ion mobility, MS/MS spectra, peptide detectability, MHC binding, PTM localisation, and potentially protein language model-derived predictions.

This extension is challenging because machine learning model benchmarking often requires stricter controls than conventional workflow benchmarking. Test data may overlap with public training data, causing train-test leakage or memorization. Predictions may also need post-processing (e.g. calibration) to be made comparable. Additionally, model architectures and training procedures are flexible, which can lead to results being difficult to compare unless the whole prediction pipeline and its parameters are explicitly defined. In addition, differences in training data, input encoding, supported modifications, stochastic prediction behaviour, etc. can strongly affect the performance and may also hinder straightforward parsing and benchmarking.

The hackathon will therefore focus on designing a ProteoBench-compatible framework that preserves low-barrier participation while adding the safeguards required for transparent, leakage-aware, and fair prediction model evaluation.
As a practical outcome, we will prototype at least one new prediction-focused ProteoBench module.

Project Plan

The hackathon will focus on six tasks:

1. Collect and prioritize candidate module ideas. Participants will be encouraged to propose prediction-focused ProteoBench modules based on their expertise, available datasets, and community needs.

2. Define a general framework for prediction model benchmarking. We will determine how predictive models can be evaluated within ProteoBench while preserving low-barrier participation. This includes defining how model outputs, parameters, metadata, model versions, and prediction settings should be submitted; how calibration, fine-tuning, or retraining should be handled; and how safeguards against train-test leakage and benchmark overfitting can be implemented.

3. Select one module for prototyping. Based on the collected ideas, we will choose one module and define its benchmark scope, required data, evaluation metrics, input and output formats, evaluation modes, candidate models, implementation strategy, and visualizations.

4. Implement the selected module. We will start developing the module within the existing ProteoBench framework, following current platform conventions.

5. Apply prediction models to the selected benchmark data. Where feasible, we will run a selection models on the selected test data to generate example benchmark submissions.

6. Benchmark and interpret the results. We will use the prototype module to compare model outputs, inspect metric behaviour, identify limitations, and define the next steps required for review, refinement, and publication.

After the hackathon, we can implement additional modules within the framework developed here, and we can write a follow up paper about the extension of ProteoBench towards predictive models in proteomics.

Technical Details

Implementation will use Python and build on the existing ProteoBench framework. Datasets, dependencies, and metrics will be selected during the hackathon based on the chosen module, data availability, metadata quality, and leakage-aware benchmark design.

Contact information

Caroline Jachmann (caroline.jachmann@ugent.be), Robbe Devreese (robbe.devreese@ugent.be), VIB - Ghent University, Belgium

[ayla-s]

Hey Caroline and Robbe,

Thanks for the clear and interesting proposal and project plan!

At Wilhelmlab, we have been working on DLOmix, a framework for applied deep learning for proteomics. Our goals are to lower the entry barrier for researchers, standardize processing and feature extraction (the ML-specific part), and ensure reproducibility by capturing artifacts throughout the workflow (datasets, preprocessing logic, model artifacts, etc.). We rely heavily on the hugging face datasets package for the data pipeline and the hugging face hub for saving and loading artifacts (along with the local option). DLOmix currently supports TF/Keras and Torch (since the last EuBIC-MS Hackathon!) for models and training. We have a couple of ML tasks and models already implemented, and users are free to build their own models using Keras and torch primitives (tf.keras.model, torch.nn.module).

In line with this, we believe there is clear potential for cross-lab collaboration to bring the community together around standardized approaches, notation, and reproducibility. From your project plan, we think step 2 aligns closely with the efforts we have been making in dlomix to capture the life cycle of data and models, save/reload them, and generate predictions. This can be a meaningful start for this step during the hackathon: look at what dlomix captures and stores, identify the existing gaps, address them, generalize, and develop a manifest/schema that future researchers can provide other implementations for, while abiding by the contract defined during the hackathon. A follow-up step (resonates with steps 4 and 5 in your plan), during or after the hackathon, could be also to ensure dlomix is compitable with the final updated schema/manifest, more like having one manifestation of the schema inside dlomix as an example for others, then people can implement further manifestations (runners) that can make predictions, run benchmarks and provide resutls back to ProteoBench.

Let us know if this sounds relevant and interesting. We can also share more details. We also completely understand if you prefer to stick to your original plan and scope. Some of Wilhelmlab will be joining the EuBIC Hackathon (Ayla so far), and collaboration can still occur in multiple forms as time and context allow.

Thank you.

Omar (o.shouman@tum.de, https://github.com/omsh) & Ayla (ayla.schroeder@tum.de)

[Cajac102]

Hi Omar, hi Ayla,

Thanks for reaching out! Indeed our projects seem to have good connection points. The possibility to easily retrain the implemented models on the same data would be great to have for ProteoBench modules! It would be great if you could join our hackathon (in case it is chosen by EuBIC of couse ;) ) even just for a day so we can pick your brains a bit :)

Cheers,
Caro