CoreMS version 4.0.0

Version 4.0 is a major release that focuses on multi-sample LC-MS analysis. It introduces a new LCMSCollection framework for aligning, gap-filling, and reporting on cohorts of related samples; overhauls molecular-annotation infrastructure by retiring the remote MetabRef API in favor of bundled offline libraries; and modernizes the build system and dependencies (Python ≥3.10, numpy ≥2.1, pandas ≥2.2, h5py ≥3.13). This release contains limited breaking API changes — please review the Removed and Changed sections before upgrading production pipelines.

Changelog

[4.0.0] — 2026-07-09

Added

Multi-sample LC-MS analysis (LCMSCollection framework)

• Added LCMSCollection class in corems/mass_spectra/factory/lc_class.py for organizing 2+ LCMSBase objects into a single processable unit with unified retention-time alignment, consensus mass-feature clustering, and cohort-level statistics.
• Added retention-time alignment with anchoring support and plot_alignments() visualization for diagnosing drift across a batch.
• Added consensus mass-feature generation, gap-filling for missing features across samples, and cluster assignment persistence.
• Added LCMSCollectionExport in corems/mass_spectra/output/export.py for saving collection manifests, alignment tables, cluster assignments, and induced (gap-filled) features to HDF5.
• Added ReadCoreMSHDFMassSpectraCollection parser for round-trip import of collection HDF5 files.
• Added MultiSamplePartitions utility in corems/chroma_peak/calc/subset.py for lazy m/z- or RT-windowed iteration over multi-sample DataFrames.
• See LCMS Collection tutorial notebook

Pluggable sample-operations pipeline for LCMSCollection framework

• Added new module corems/mass_spectra/calc/lc_calc_operations.py with SampleOperation base class and concrete operations (GapFillOperation, ReloadFeaturesOperation).
• Added process_consensus_features() and process_samples_pipeline() for chaining operations (gap-fill → representative-feature reload → MS1/MS2 association → molecular formula search → MS2 library search) into a single parallelized pass.
• Parallelized induced-mass-feature molecular-formula search and cosine-similarity molecular-network calculations.
• See LCMS Collection tutorial notebook

Peak-quality metrics for LCMSBase class

• Added gaussian_similarity, noise_score, and tailing_factor attributes to LCMSMassFeature in corems/chroma_peak/factory/chroma_peak_classes.py for quality-based feature ranking and filtering.
• Added corresponding calculators in corems/chroma_peak/calc/ChromaPeakCalc.py.

Metabolomics & lipidomics workflows

• Added LCMSMetabolomicsExport and refined LipidomicsExport classes with consensus lipid annotation to LIPID MAPS species/molecular-species levels.
• Extended LipidMetadata dataclass in corems/molecular_id/factory/lipid_molecular_metadata.py with lipid_category, lipid_class, lipid_subclass, lipid_summed_name, and structure_level fields.
• Added targeted mass-feature search: search_for_targeted_mass_features_batch() in corems/mass_spectra/calc/lc_calc.py queries known m/z values with configurable ppm and RT windows, in single-sample or batch mode.
• See LCMS Targeted Search Tutorial notebook

Offline spectral-library interfaces (MetabRef replacement)

• Added GCMSLibraryInterface for local MSP-based GC-MS spectral searches.
• Added LCLipidLibraryInterface for local SQLite-based lipid annotation (202412_lipid_ref.sqlite, downloaded on-demand).
• Added generic MSPInterface class for user-supplied MSP files.
• Bundled reference libraries: corems/molecular_id/data/PNNLMetV20191015.msp and corems/molecular_id/data/FAMES_REF.msp.
• Added environment variables GCMS_LIBRARY_PATH, FAMES_LIBRARY_PATH, COREMS_LIPIDOMICS_SQLITE_PATH for user overrides.

Instrument & format support

• Added basic Bruker imaging-mode ingestion via new helpers in corems/mass_spectra/input/brukerSolarix_utils.py. Note this just allows access to the ImagingInfo.xml scan indexes necessary for parsing the ser binary transient file. Imaging data analysis is not formally supported. The functions here will be updated in a future release per the TODO notes in the docstrings.
• Added instrument-info and acquisition-timestamp extraction to LC-MS parsers.
• Extended Thermo RawFileReader support and added safeguards for legacy Thermo TIC handling.
• Added "centroided_persistent_homology" peak-picking method for centroided LCMS data.
• Added auto_process parameter to ReadCoremsMasslist.get_mass_spectrum in corems/mass_spectrum/input/massList.py enabling delayed processing.

CI, packaging, and quality

• Added GitHub Actions workflow (.github/workflows/tests.yml) running on ubuntu-latest / Python 3.13 with ci-test-source and ci-test-notebooks targets, harmonized with the GitLab CI source and shared Makefile targets.
• Introduced pyproject.toml as the primary build definition; setup.py reduced to a shim.
• Added first-pass molecular-formula database validation with auto-heal (corems/molecular_id/factory/molecularSQL.py, tests in tests/test_molecular_formula_db_factory.py).
• Added lipidomics_db and molecular_db pytest markers plus --skip-lipidomics-db / --skip-molecular-db options in conftest.py so CI can skip tests that require the SQLite lipidomics library or the Postgres molecular-formula database.
• Added cross-platform Makefile targets (Windows-friendly lipidomics download, ci-test-source / ci-test-notebooks, per-notebook selection) and pytest-xdist as a dev dependency for parallel test execution.
• Added CITATION.cff with author ORCIDs and future-proofed Zenodo references.
• Reorganized and deduplicated .gitignore.
• Added ~2,700 lines of new tests: tests/test_lcms_collection.py, tests/test_lcms_metabolomics.py, tests/test_noise_calc.py, tests/test_time_range_filtering.py, tests/test_input.py, plus expansions to existing suites.

Changed

• MassSpectraBase.spectra_parser converted from attribute to on-demand property, backed by spectra_parser_class; added raw_file_location property to support relocated raw data files after HDF5 round-trip.
• LCMSBase.mass_features_to_df() now accepts induced_features, drop_na_cols, and include_cols (all defaulted; backwards compatible).
• LCMSBase.mass_features_ms1_annot_to_df() and mass_features_ms2_annot_to_df() now accept suppress_warnings (defaulted; backwards compatible).
• LCMSExport now delegates collection-level state (manifests, alignments, cluster assignments) to LCMSCollectionExport; sample-level exports keep the sample-only surface.
• Refactored persistent-homology 2D peak picking for large memory savings, with a speed/memory tradeoff parameter on the LC rollup routine (ph_centroid_memory_improvements).
• Refactored SQL database connection handling in corems/molecular_id/search/molecularFormulaSearch.py to support safe parallel searches.
• Improved noise-based intensity filtering with SNR fallback for centroid-only data.
• Reorganized example notebooks and archived legacy examples.
• Modernized Dockerfile to python:3.13-slim base with inline .NET 8 runtime installation.

Fixed

• Fixed TIC/MS-level retrieval in ImportMassSpectraThermoMSFileReader.
• Fixed noise ROI inclusive-selection logic with regression tests.
• Fixed mzML parser handling of non-spectral data.
• Added checker to prevent duplicate reprocessing when adding mass spectra to a collection.
• Fixed division-by-zero in percentage_assigned reporting; corrected SNR recalibration edge case.
• Fixed Windows-specific HDF5 and mzML parser issues.

Removed

• Removed MetabRefGCInterface and MetabRefLCInterface and the majority of MetabRefInterface from corems/molecular_id/search/database_interfaces.py. The remote MetabRef API has been retired; code importing these classes must migrate to GCMSLibraryInterface, LCLipidLibraryInterface, or MSPInterface. A deprecation stub of MetabRefInterface remains for one release with a DeprecationWarning pointing users to bundled libraries.
• Removed old MetabRef bundled data files: FAMES_REF.MSL, PNNLMetV20191015.MSL, gcms_library_raw.json, fames_library_raw.json. Replaced by the .msp bundles listed under Added.
• Removed transient corems/mass_spectra/factory/lc_collection.py; its class definitions now live inline in corems/mass_spectra/factory/lc_class.py.
• Removed unused NMDC support scripts under support_code/nmdc/ (metadata parser/generator, workflow shims).
• Removed obsolete lipidomics example mzML fixture; replaced with test_centroid_neg_RP_metab.mzML and companion test_db.msp under tests/tests_data/lcms/.

Breaking changes — please read before upgrading

Users upgrading from v3.3.0 to v4.0.0 should audit their code for the following user-visible breaks:

1. Python ≥3.10 required. pyproject.toml sets requires-python = ">=3.10". Users on 3.8 or 3.9 must upgrade their runtime. The CI target is Python 3.13.
2. Major dependency bumps (>= lower bounds):
   • numpy >=2.1 (was ~1.24) — NumPy 2.x changes dtype/casting semantics - we have updated the codebase to be numpy 2.x compatible while maintaining backwards compatibility with numpy 1.24.
   • pandas >=2.2 (was ~1.5) — removed .ix, stricter type handling.
   • scipy >=1.14 (was ~1.10), h5py >=3.13 (was ~3.8), lmfit >=1.3 (was ~1.1), matplotlib >=3.9 (was ~3.7), pythonnet >=3.0.3 (was ~3.0.1)
3. MetabRef API removed. Any code calling MetabRefGCInterface() or MetabRefLCInterface() will fail. Migrate to GCMSLibraryInterface / LCLipidLibraryInterface / MSPInterface. Token-based authentication is no longer used or supported; the interfaces read local library files.
4. Bundled library data files renamed. Code that referenced PNNLMetV20191015.MSL, FAMES_REF.MSL, or the gcms_library_raw.json / fames_library_raw.json files by path must switch to the .msp files under corems/molecular_id/data/ or override the paths via GCMS_LIBRARY_PATH / FAMES_LIBRARY_PATH.
5. Default calibration ref-match method changed from "legacy" to "merged" in LiquidChromatographSetting (corems/encapsulation/factory/processingSetting.py). Mass-calibration results will differ silently; set parameters.mass_spectrum.calibration_ref_match_method = "legacy" to reproduce v3.3.0 behavior.
6. MolecularFormulaSearchSettings.db_jobs default lowered from 3 to 1 for stability. Database molecular-formula searches now run single-threaded by default; opt back in explicitly by setting db_jobs = 3 where your Python environment supports it.
7. Removed NMDC support scripts under support_code/nmdc/. If you depended on these, pin an older release or vendor them locally.

Migration notes

• Replace remote MetabRef calls with the bundled offline interfaces; expect deprecation warnings on any residual MetabRefInterface usage during v4.0.x.
• Re-run a representative dataset with the new default calibration_ref_match_method = "merged" and compare against a v3.x run before promoting v4.0.0 to production.
• If you rely on parallel DB-backed formula search, set MolecularFormulaSearchSettings.db_jobs = 3 (or higher, with care) in your parameter file — the default now favors stability over parallelism.