SegFlowProcessor (Segmented Flow MS Processor)

This repository contains a Python-based data processing pipeline designed for automated slug-flow mass spectrometry (MS) analytics. The tool automates vendor data conversion, corrects for matrix-induced ionization efficiency changes using a Post-Column Internal Standard (PCIS), detects individual liquid segments, and performs precise chromatographic peak integration across all monitored transitions.

Key Features

• Vendor-Agnostic Conversion: Orchestrates the conversion of raw mass spectrometry vendor files (e.g., Agilent .d directories) into standardized formats using a raw_to_parquet wrapper architecture.
• PCIS Ionization Correction: Dynamically normalizes analyte signal transitions against the PCIS trace to mathematically compensate for electrospray ion suppression/enhancement.
• Automated Segment Segmentation: Identifies individual slug boundaries within high-throughput segmented flow data utilizing peak prominence, width metrics, and Savitzky-Golay filtering.
• Total Slug Integration: Quantifies entire segment payloads using dynamically calculated boundary thresholds and outputs both numerical reports and diagnostic visual plots.

Repository Structure

├── .gitignore          # Version control ignore rules
├── convert.sh          # Shell script wrapper for automation/Docker tasks
├── pymsconvert.py      # Core vendor data conversion utilities
├── helper.py           # File handling and parquet conversion wrappers
├── process.py          # Main execution script for peak detection & integration
├── requirements.txt    # Frozen environment dependencies
└── README.md           # Repository documentation

Installation & setup

To replicate the exact Python environment used to develop and validate this pipeline, clone the repository and install the frozen dependencies via pip:

# Clone the repository
git clone https://github.com/hankemeierlab/SegFlowProcessor.git
cd segmented-flow-processor

# Install environment dependencies
pip install -r requirements.txt

Core environment requirements

The pipeline is verified on Python 3 and utilizes the following key scientific computing packages defined in requirements.txt:

• numpy (v2.1.0) & pandas (v2.2.2) — Data structures and vectorization

• scipy (v1.14.1) — Digital filtering, peak width, and prominence calculations

• matplotlib (v3.9.2) — High-fidelity PDF rendering for quality control

• detecta (v0.0.5) — Core peak detection architecture

• pyarrow (v17.0.0) — High-performance storage and parsing of parquet formats

Data Processing Pipeline Workflow

The architecture executes across four distinct operational stages:

1. Conversion (pymsconvert.py / helper.py): Translates raw vendor data structures (such as Agilent .d directories) into optimized parquet arrays via a centralized wrapper pipeline.

2. Baseline Normalization (process.py): Monitors the PCIS transition baseline across the run sequence. Analyte transition streams are continuously normalized against the local PCIS intensity to yield an ionization-corrected profile.

3. Segment Demarcation (process.py): The corrected ion profile is smoothed using a Savitzky-Golay filter. Advanced peak prominence and width analytics map out the exact start and end coordinates of every individual fluid slug.

4. Integration & Export (process.py): The tool integrates the area under the curve (AUC) for each analyte transition strictly within the verified segment borders.

How to Run

To execute the data extraction pipeline, map your input directories within the configuration layer of the main script and execute process.py from your terminal interface:

python process.py

Input Requirements

Raw vendor MS data formats, for instance Agilent .d files, can be used as input for this Pipeline, as well as MZML files.

Generated Outputs

Upon completion, the pipeline automatically writes two deliverables to your specified output directory:

1. CSV Report: A unified master data table tracking calculated peak area integrations for every transition across all detected segments and samples.

2. Diagnostic Graphics PDFs: A visual validation document rendering individual trace plots of each slug's integration boundaries per transition, enabling rapid, manual quality control checks of the automated segmentation.

Citing Segmented Flow MS Processor

To cite Segmented Flow MS Processor in publications, please use:

Bremer I, Wouters B, Harms AC, Clark CP, Hankemeier T (2026). “Segmented flow metabolite extraction using inline monolithic µSPE.” in submission