Biosaur2 0.6.0

Cython hybrid speed increase

Version 0.6.0 keeps the hybrid scientific decision rules unchanged while
substantially accelerating generic-local recovery for runs with Percolator PSM
results. The runtime now has one fixed Cython/NumPy implementation; it does not
ship, discover, or select an optional Rust backend.

The main performance work is in biosaur2/cutils.pyx:

extract_trace_values() is the scalar selected-ion trace kernel.
extract_traces_values() extracts all isotope targets for a selected RT
window in one batched Cython call.

RawMS1Store.extract_traces() uses numpy.searchsorted to select the RT window
once before it calls the batch kernel. Generic-local processing also materializes
the residual snapshot as NumPy arrays instead of repeatedly traversing Python
objects. Candidate-cache options are JSON-stable, and read-only memory-mapped
raw caches are accepted without copying.

Measured cold-cache performance

Four independent PXD010154 mzML files with matching Percolator TSVs were run
with four workers each, explicit empty cache directories, and no cache reuse.
All four completed successfully.

Run	MS2 events	Generic-local events	Generic stage	Total wall time
1554452	39,309	29,116	18:24.823	31:24.04
1554453	50,626	31,398	20:06.115	30:40.86
1554454	48,555	36,774	24:08.836	34:03.54
1554455	38,561	33,215	24:56.761	36:05.57

These files sustained 22.2-26.4 generic-local events per second. Against the
old Python 1554451 measurement (about 1.49 events per second before its former
final-stage scope failure), this is a conservative cross-file indication of
about 14.9-17.7x higher throughput. A same-input stage comparison reached the
equivalent point in about 9:14 instead of 3:39:26, but that approximately 24x
figure is input-specific and should not be treated as a universal guarantee.

Upgrade notes

--hybrid-backend has been removed from the single-run and project CLIs.
Remove this option from existing scripts; hybrid mode always uses Cython.
Local candidate-cache fingerprints now record trace_extractor: cython.
Existing candidate caches will be safely rebuilt on their next use.
Hybrid output provenance records the fixed Cython trace extractor. Scientific
feature acceptance and quantification semantics are otherwise unchanged.

Validation and installation

The release was built and installed in the base conda environment with:

python setup.py build_ext --inplace
python -m pip install -e .
python -m pytest -q

The final suite result was 256 passed, 6 warnings; the warnings are the known
Python 3.12 multiprocessing fork() deprecation warnings.

2026-07-30 update: identification-aware residual feature detection

Summary

This update adds an opt-in hybrid DDA workflow whose primary product is an
accurate, de-duplicated MS1 feature population with one reliable quantitative
value per accepted feature. It uses q-value-filtered direct PSMs and generic
MS2 precursor evidence to associate or recover features where scientifically
defensible, while leaving non-MS2 feature thresholds unchanged.

The current milestone target was approximately 60% aggregate quantitative
MS2-feature coverage on a frozen representative panel, subject to strict
feature preservation, positive unique quantification, intensity conservation,
target/decoy behavior and honest unresolved outcomes. The broad 12-run standard
result reached 70.5727%. A narrower guarded-relaxed four-run experiment reached
69.7888% and added 1,119 reliable links over its same-input standard baseline
without losing any standard link or baseline strict scientific row.

The hybrid workflow remains optional and default-off. The conservative relaxed
retry is a separate opt-in switch and also defaults off.

Motivation

The earlier workflow primarily detected MS1 features first and treated MS2 as
weak downstream evidence. That left many MS2 events unmatched even when their
RT, precursor context or high-confidence PSM could help distinguish the best
MS1 representation.

The new design keeps the complete strict MS1 population as the foundation, but
adds targeted evidence and an intensity-conserving residual model so that:

multiple MS2 events can associate with one feature without duplicating its
abundance;
exact identified isotope distributions can repair or recover difficult local
features;
unidentified MS2 events can use bounded charge/C13/averagine hypotheses;
strict, MS2-guided and final-residual candidates can be compared locally;
unresolved events remain auditable rather than being forced into weak
features.

User-visible changes

Explicit feature modes

The single-run CLI now supports:

--feature-mode legacy
--feature-mode hybrid

Project execution has corresponding --mode values. Both CLIs default to
legacy; hybrid processing must be explicitly selected. The former weak-MS2
compatibility mode and its --ms2-seed alias have been removed.

New or revised scientific parameters

Parameter	Default	Purpose
`--psm-q-value-max`	0.01	Filter Percolator PSM input before direct-assay construction.
`--psm-pep-max`	none	Optional additional PEP filter.
`--fixed-mod`	none	Explicit repeatable fixed modification such as `C=UNIMOD:4`.
`--generic-ms2-ppm`	10 ppm	Selected-ion tolerance for generic precursor hypotheses.
`--generic-ms2-isotope-errors`	`0,1,2,3`	Interpret the selected peak as M through M+3; advanced negative values require explicit validation.
`--generic-local-isotope-count`	5	Isotope channels examined by generic local recovery.
`--generic-local-min-mono-points`	3	Standard monoisotopic point minimum.
`--generic-local-min-channel-points`	3	Standard supported-channel point minimum.
`--generic-local-min-supported-channels`	2	Standard supported-channel count.
`--generic-local-min-isotope-cosine`	0.90	Standard observed/averagine envelope similarity.
`--generic-local-max-width-sec`	`auto`	Strict-feature width q99 clamped to 15-60 s, falling back to 30 s.
relaxed generic point/channel/cosine controls	2 / 2 / 2 / 0.95	Guarded retry thresholds exposed by `--help-all`.
`--ms2-rt-tolerance-sec`	120 s	Initial bounded local search around an MS2 event.
`--max-charge`	7	Increased supported charge ceiling.
`--generic-ms2-refine`	true in hybrid	Enable generic unidentified-MS2 association/recovery.
`--generic-q-value-max`	0.01	Separate paired target/decoy extraction q-value.
`--relaxed-ms2-feature`	false	One conservative retry for unresolved MS2-supported evidence only.
`--quant-method`	`all`	Write all three measurements; envelope area remains the primary `quant_value`.
`--feature-baseline`	project: `edge_linear`	Optional baseline preprocessing.
`--cache-dir`	`.biosaur2_cache`	One root for raw MS1, strict-stage, candidate, and project caches.
`--keep-cache`	false	Retain fingerprinted caches for a later compatible command.
`--workers`	4	Total CPU worker-process budget shared across active runs.
`--format`	legacy: TSV; hybrid: Parquet	One public format control; `duckdb` writes one database per input.

Direct PSM input continues to use q-value <= the configured maximum. The extra
direct relaxed exception is deliberately narrower and requires same-run PSM
q-value < 0.01. No q-value threshold was raised to improve coverage.

Quantification methods

Hybrid mode reports three feature-level measurements:

envelope_area: trapezoidal area of the final assigned isotope envelope over
real RT seconds;
mono_area: area of the final monoisotopic contribution;
envelope_apex: maximum summed assigned isotope intensity on one common MS1
scan.

The default all method writes quant_envelope_area, quant_mono_area, and
quant_envelope_apex; envelope area is also stored as quant_value. The
selected value is recorded together with raw/corrected areas, quantification
status, feature origin, confidence tier, support counts and quality flags.

GitHub documentation and CLI help

The release README is now a practical user guide rather than only an output
schema reference. It links directly to design.md, this
update's validation results, and the
expected-versus-actual conclusions. It also
documents:

accepted mzML, reusable-hill and Percolator input shapes;
a minimal PSM table and safe spectrum-identity mapping expectations;
a workflow decision table for legacy, generic-only hybrid, direct PSM
hybrid, guarded relaxation, caches and multi-run projects;
recommended single-run commands and when each should be used;
minimal and full project manifest examples;
grouped detection, hybrid, quantification, output, cache, concurrency and
experimental-DIA option references;
important scientific cautions about distinct q-value families, repeated MS2,
single-scan signal, external intensity and RT/area units.

biosaur2 --help now focuses on everyday parameters and prints their parser
or resolved automatic defaults. biosaur2 --help-all adds scientific tuning,
diagnostic exports and output-engine controls with complete defaults and
explanations. biosaur2 project run uses the same help split for project
parallelism, restart policy and direct/generic/external controls.
biosaur2 project make-manifest --help now dispatches to its subcommand parser
instead of being intercepted by the parent help screen.

The later CLI simplification keeps the scientific thresholds, residual
ownership, and cache payload formats unchanged, but it intentionally replaces
the public cache/parallelism flags and advances the hybrid sidecar schema to v4
for the three explicit final quantification columns. design.md
describes the resulting contract.

Algorithm improvements

One-pass MS1/MS2 ingestion

The reader now retains all metadata needed by residual processing:

original MS1 centroid points and scan offsets;
normalized RT seconds and native scan identity;
every DDA MS2 event and resolved precursor MS1 relation;
selected ion and isolation boundaries;
charge, FAIMS and ion-mobility metadata where present.

The compact raw store can be memory-mapped from a cache so parameter tuning does
not repeatedly decompress and parse mzML.

Robust Percolator and chemistry support

The update adds:

q-value-first semantic parsing of Percolator target PSM tables;
compression, encoding/BOM, delimiter and header handling;
safe PSM-to-MS2 mapping by native identity or validated run/scan parsing;
canonical modification normalization;
a pinned local Unimod subset with no runtime network requirement;
explicit fixed modifications;
exact-formula, mass-only and unavailable modification states;
sequence/formula-specific theoretical isotope libraries;
selected C13 isotope-index evaluation.

PSMs provide strong prior evidence but cannot create a quantitative feature in
the absence of coherent MS1 chromatography.

Residual MS1 ownership

ResidualMS1Ledger tracks original, assigned and remaining intensity at raw
point level. It enforces non-negative allocation, prevents over-subtraction and
provides reversible provenance. The central invariant is:

claimed feature intensity + residual intensity = original MS1 intensity

Initial strict features claim their valid contribution first. Direct/generic
recovered features and final-residual strict features must then allocate from
the remaining signal. This prevents an MS2-supported candidate from silently
consuming an established high-quality feature.

Direct identified feature association and recovery

Every exact direct assay first searches the strict population. Unmatched assays
can extract bounded raw MS1 isotope traces around the MS2 event. Acceptance
requires multi-scan chromatographic support, isotope evidence, mass/RT/charge
compatibility, event and isolation localization, non-conflict with strict
ownership, successful residual allocation and reliable quantification.

High-confidence direct-to-strict matches calibrate run-specific mass error,
feature width and MS2-to-apex RT behavior. One monotonic retry may use this
calibration. Repeated PSMs or MS2 events reuse the same feature.

Losing processed-hill competitors and cache v2

The strict detector may destructively assign a hill before a direct assay is
processed. To avoid losing all information about plausible alternatives, the
new direct_competitors.py path captures direct-relevant candidates after
ordinary strict mass/cosine filtering and before greedy ownership.

Capture is deterministic and bounded to the best three representations per
PSM. Strict-stage cache version 2 persists the candidates. They can guide the
existing one direct retry, but cannot bypass raw extraction, monotonic evidence,
conflict, quantification, residual or conservation gates.

Generic MS2 association and local recovery

Unidentified MS2 events now receive bounded charge and C13 isotope-offset
hypotheses with averagine-like expected distributions. Target candidates and
paired deterministic decoys are scored using m/z, RT, precursor scan, isolation,
charge, selected-isotope, event-apex, isotope count/cosine, coelution and point
support.

Generic processing first associates defensible strict features, then examines
unresolved events against raw/residual local traces. Candidate extraction and
confidence control are separate from Percolator q-values.

Local hill repair and decomposition

The update adds bounded, reversible local actions:

common-boundary RT splitting across isotope traces;
merge of compatible fragments;
short-gap/boundary extension;
local relinking of incorrect trace relationships;
non-negative decomposition for identifiable shared signal.

Repeated compatible MS2 events are clustered so that one feature can satisfy
several events. Different envelopes that share signal without an identifiable
joint solution remain explicit conflicts.

Guarded relaxed MS2-only recovery

The optional relaxed round permits shorter but still multi-scan evidence and
some partial envelopes. It never applies to a feature without MS2. Generic
relaxation retains paired target/decoy control and strong isotope-fit criteria.

The first unguarded experiment found three cases where a relaxed candidate
changed a better final-residual strict representation. A same-residual strict
competitor guard was therefore added:

an equivalent strict candidate wins when its isotope/chromatographic evidence
is clearly superior;
an MS2-supported candidate may win only inside the defined uncertainty margin;
different charge/mass envelopes sharing raw points are rejected when the
allocation is not identifiable.

The guard rejected exactly the three problematic assignments while preserving
the remaining relaxed gains.

Final strict residual detection and MS2 recheck

After targeted recovery, the unchanged strict detector runs on remaining
residual MS1. It discovers untargeted features, including features without MS2,
using ordinary strict thresholds. Equivalent already accepted features are
de-duplicated before allocation.

Unresolved direct and generic MS2 events are then rechecked against newly found
strict features. This ordering allows targeted evidence and strict population
completeness to complement one another without lowering non-MS2 standards.

Project manifests and external assays

Project mode adds deterministic manifest parsing, bounded per-run execution,
resume/overwrite semantics, atomic publication and a compact DuckDB status
database.

Compatible runs may be aligned using robust monotonic RT mappings built from
shared high-confidence peptide/charge anchors. External exact assays are weaker
than direct assays and use their own target/decoy control. Recipient-run MS1
signal is always extracted and quantified; donor intensity is never copied.

Outputs and auditability

Hybrid output is consolidated around two primary per-input tables:

<stem>.features.parquet
<stem>.identifications.parquet
<stem>.external_id_evidence.parquet   project external stage only
<stem>.biosaur2.duckdb                alternative one-database format
project.duckdb

Each feature row now includes its quantification fields and a typed list of
zero or more linked MS2 events. Each identification row includes nullable
direct-assay fields. An MS2 event with a PSM but no feature remains in
identifications; an event with neither a feature nor PSM is omitted row-by-row
and retained only in run summary metadata/logs. Internal audit outcomes still
distinguish insufficient chromatographic evidence, ambiguity, precursor-only
signal, metadata/assay problems and statistical rejection.

The obsolete public format controls (--feature-format, --hills-format,
--ms1-format, --duckdb-output, and --parquet-temp-dir) were removed in
favor of --format and --cache-dir. --write-ms2 is now an advanced,
legacy-only normalized precursor diagnostic.

Performance and reproducibility improvements

Raw, strict-stage and local-candidate cache layers avoid repeated mzML/hill
work during parameter tuning.
Cache keys include source, scientific parameters, implementation signature
and residual ownership state where relevant.
Atomic staging prevents partial output/cache publication.
Deterministic ordering is used for candidates, conflicts, IDs and project
scheduling.
One --workers budget is distributed over active files and their internal
work without exceeding the detected CPU limit.
Development validation was staged on one to four files before expansion and
never exceeded six real mzML files per tuning experiment.

Validation results

Standard 12-run frozen panel

Metric	Result
total MS2 events	514,529
quantitative feature links	363,117
aggregate coverage	70.5727%
accepted features	2,066,266
positive quant rows	2,066,266
runs passing intensity conservation	12/12
features linked to multiple MS2 events	44,765
repeated MS2 events on those features	104,222

The repeated events reuse one feature abundance rather than duplicating it.
This standard configuration is the broadest validated recommendation.

Guarded relaxed four-run A/B

run	all MS2	standard links	guarded relaxed links	standard coverage	guarded coverage	gain
1554883	38,704	27,254	27,646	70.4165%	71.4293%	+392
1555013	50,361	39,332	39,523	78.1001%	78.4794%	+191
1555082	42,684	29,655	29,966	69.4757%	70.2043%	+311
1556157	34,027	18,333	18,558	53.8778%	54.5390%	+225
aggregate	165,776	114,574	115,693	69.1137%	69.7888%	+1,119

Additional reliability checks:

783,049/783,049 leading input-strict scientific rows were exact;
4,543/4,543 baseline final-strict scientific rows were exact;
lost final-strict rows: 0;
794,341 features and exactly 794,341 positive quant rows;
feature IDs and quant IDs were positive, unique and equal by run;
1,115 partial features, including 236 two-point and zero single-point;
all four residual ledgers conserved intensity;
115,693 linked events referenced 101,359 unique linked features.

The guarded relaxed configuration is scientifically reliable on this four-run
panel but remains opt-in/default-off because its validation is narrower than
the standard 12-run result.

Real strict-stage cache-v2 experiment

Run 1555082 rebuilt the strict stage from real hills while reusing only the raw
and downstream-safe caches:

Metric	Result
direct assays	4,655
captured losing-competitor matches	465
represented assays	443
unique representations	443
processed-hill retries attempted	17
retries selected by monotonic evidence	14
final accepted/reused contribution	0/0
gained/lost quantitative links	0/0

Feature, quantification, identification, assay and MS2 tables were exactly equal
to the same-input standard baseline after excluding provenance metadata. Only
four still-null MS2 audit rows changed their failure-status category; no feature
ID, abundance or link changed. The branch is therefore safe and genuinely
executed, but it has no measured coverage gain on this run and was not expanded
merely to seek a nonzero result.

Why some MS2 events remain unresolved

For standard run 1555082, 29,655/42,684 events had quantitative features and
13,029 remained null. The dominant mutually exclusive reasons were:

Reason	Events	Fraction of null events
insufficient monoisotopic points	5,133	39.3967%
no chromatographic component at event scan	2,741	21.0377%
selected isotope absent at event scan	1,598	12.2649%
insufficient isotope-channel support	1,286	9.8703%
low averagine isotope cosine	1,165	8.9416%
isolation/metadata unavailable	256	1.9648%
isotope apex spread	247	1.8958%
raw-point conflict/ambiguity	197	1.5120%

Any local MS1-signal association was present for 99.4002% of events, but signal
presence alone does not define a coherent, multi-scan, non-duplicated feature.
Only 26 events were rejected by q-value/decoy criteria, so weakening q-value
control would not address the dominant limitations.

Test and build validation

The final implementation was tested in zsh with conda base activated and
/data/p/bin on PATH.

Automated suite

python -m pytest -q
215 passed, 6 warnings, 43.36 s

The six warnings are the known Python 3.12 multiprocessing fork() deprecation
warnings; there were no test failures. The test suite covers:

Percolator parsing, PSM mapping and modification chemistry;
exact and averagine isotope hypotheses;
direct association and calibrated local extraction;
generic target/decoy competition and q-values;
residual allocation and intensity conservation;
local split/merge/extend/relink and non-negative decomposition;
strict preservation and guarded relaxed competition;
repeated-MS2 de-duplication and quant support counts;
feature quantification and baseline status;
output schemas, atomic publication and DuckDB validation;
raw, strict-stage v2 and local-candidate caches;
project manifests, alignment, external evidence and bounded parallelism;
legacy/weak/hybrid CLI option behavior.

Packaging and smoke tests

python -m compileall -q biosaur2 tests    passed
python -m build                          sdist and wheel passed
biosaur2 -h                              passed
biosaur2 project --help                  passed
git diff --check                         passed

Post-completion cleanup validation

Three zero-call private helpers and unused imports/locals were removed without
changing algorithm behavior. No whole test file was deleted because every file
covered a live module or compatibility/reliability contract. Per the request to
avoid long retesting, cleanup used a selected 40-test set:

40 passed in 13.15 s
ruff F401/F841                            passed
compileall and both CLI help smokes       passed
git diff --check                          passed

The pre-cleanup full-suite baseline remains the authoritative 215-pass result.

Expected versus actual outcome

Goal	Expected	Actual conclusion
aggregate quantitative MS2 coverage	approximately >=60%	Achieved: 70.5727% on 12 runs.
improve coverage without sacrificing strict features	positive reliable gain	Achieved: guarded relaxed +1,119 links, lost links=0, lost strict rows=0.
complete and accurate MS1 population	retain non-MS2 strict features	Achieved within measured scope: unchanged strict thresholds plus final residual strict detection.
one quantitative value per feature	no repeated-MS2 abundance duplication	Achieved: feature/quant IDs one-to-one and positive.
intensity conservation	claimed + residual = original	Achieved: 12/12 standard and 4/4 guarded runs.
use direct PSM chemistry safely	q-filter first, require MS1 evidence	Achieved.
use unidentified MS2 evidence	charge/C13/averagine plus target/decoy	Achieved.
improve split/merge/gap/overlap handling	bounded reversible repair	Achieved for identifiable cases; unidentifiable cross-envelope sharing remains ambiguity.
processed losing-hill recovery	prove path and seek safe gain	Path validated; no final gain on 1555082.
aligned external transfer	implement controlled recipient extraction	Implemented; no accepted q<=0.01 gain on the present panel.
preserve legacy and weak behavior	hybrid remains optional	Achieved: new modes are explicit and default-off.

Final decision

The approximately-60% milestone and mandatory reliability invariants are
complete. The recommended broad configuration is standard hybrid residual mode
with relaxation disabled. Guarded relaxation is available as a validated but
narrower opt-in configuration.

The remaining limitations are research directions rather than reasons to lower
standards:

universal cross-envelope decomposition is not always identifiable;
external assays may produce no valid recipient feature;
processed-hill competitors may improve local evidence without surviving all
quantitative/conflict gates;
sparse or event-misaligned isotope traces must remain unresolved.

Future work should prioritize multi-scan trace recovery and identifiable
regional overlap models. It must continue to forbid single-scan quantitative
features, duplicated shared intensity, donor-abundance copying, relaxed non-MS2
thresholds and weakened q-value control.

For the stable algorithm description, see design.md. Detailed
development evidence remains in .local/2026-07-29-ms2-seed.md and
.local/2026-07-29-target-completion-audit.md.

biosaur2 v0.6.0