SIG Processing Pipeline

A pure-Python pipeline for processing SVC HR-1024i field hyperspectral .sig files. Reads raw multi-detector spectra, performs sensor stitching and radiometric correction, applies resolution-matched Gaussian smoothing, and resamples onto a uniform 1 nm grid from 400–2500 nm. Numerically verified against the legacy R/spectrolab reference to better than 1 × 10⁻⁶ absolute reflectance.

This README is the canonical entry point for both humans and LLMs. Every directory in the repo has its own README.md describing its contents in detail. Follow the links below — do not assume anything that is not stated here or in the linked docs.

---

Repository map

Path	Purpose	Read me first
TUTORIAL.md	Beginner walkthrough — starts in the Jupyter notebook (gentle path), with an advanced CLI track.	Start here if new
pyproject.toml	Package metadata and svc-pipeline console script.	This file ↓
pipeline/	Core Python package: CLI, SigFileProcessor, resample_spectra, SVCDataProcessor.	pipeline/README.md
tests/	Pytest suite, including the R-vs-Python parity test.	tests/README.md
config/	Run configs + instrument calibration JSONs.	config/README.md
docs/	Manuscript-grade methods, parity reports, LLM re-test prompt.	docs/README.md
archived_r_scripts/	Frozen R/spectrolab reference (Pipeline A) — kept only for parity verification.	archived_r_scripts/README.md
notebooks/	Demo and visualization notebooks (not on the production path).	notebooks/README.md
naming_ids/	Private CSV lookup tables for grouping scans into samples; only the schema README is tracked.	naming_ids/README.md
FOLDER_STRUCTURE.md	Authoritative tree + reading order.	—
ACKNOWLEDGMENTS.md	Attribution, third-party licenses, and citation.	—
LICENSE	GNU General Public License v3.0.	—

Generated outputs (gitignored): pipeline_outputs/sig_processed/<run>/, pipeline_outputs/sig_resampled/<run>/.

---

Quick start

New to the pipeline? TUTORIAL.md starts with the interactive Jupyter notebook (the gentle path) and has an advanced track for the CLI below.

Installed from PyPI, in a new project folder (no clone of this repo needed):

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install svc-processing   # package name

svc-pipeline --init-config   # command name — different from the package name above
# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
svc-pipeline

Working from a clone of this repo (config/config.json already exists) — run the setup script once, then activate the environment it created:

./setup.sh            # creates .venv and installs the project + notebook tools
# ./setup.sh --dev    # add the test & lint tools for development

source .venv/bin/activate          # macOS / Linux
# .venv\Scripts\activate           # Windows PowerShell

# Edit config/config.json — replace "<PATH_TO_SIG_INPUT_ROOT>" with your data path.
svc-pipeline config.json

The script always installs into this repo's .venv no matter which folder you run it from, so it avoids the "Directory '.' is not installable" and "No module named …" errors that come from being in the wrong directory. Prefer to do it by hand?

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev,demo]"
svc-pipeline config.json

Outputs land under pipeline_outputs/ by default. See config/README.md for every supported key.

---

Pipeline architecture

   raw *.sig files (sig_input_dir)
            │
            ▼
   ┌────────────────────────────────────┐
   │ Stage 1: SigFileProcessor          │  pipeline/sig_processor.py
   │   - instrument-consistency check   │
   │   - truncate at calibration        │
   │     end-line wavelength            │
   │   - write summary CSV              │
   └────────────────────────────────────┘
            │
            ▼  processed *.sig + *_processed_sig_summary.csv
   ┌────────────────────────────────────┐
   │ Stage 2: resample_spectra()        │  pipeline/resampler.py
   │   - detect sensor segments         │
   │   - guess_splice_at + trim         │
   │   - match_sensors (iter = 1)       │
   │   - smooth_fwhm (k=3 kmeans)       │
   │   - Gaussian resample fwhm=10      │
   │     onto 400–2500 nm @ 1 nm        │
   └────────────────────────────────────┘
            │
            ▼  <run>_merged_spectra.csv
   ┌────────────────────────────────────┐
   │ Stage 3 (post-hoc, optional):      │  pipeline/processor.py
   │   SVCDataProcessor /               │  + notebooks/
   │   SigSpectraAverager — group &     │
   │   average scans into samples.      │
   └────────────────────────────────────┘

The svc-pipeline console script runs Stages 1 and 2 through pipeline.cli, and Stage 3 too once a groups_csv is configured (or passed via --groups-csv) — see config/README.md. Stage 3 is also available directly from notebooks against the Stage 2 output. The pipeline is single-pass and idempotent per input directory — previous processed .sig files in the target directory are deleted at the start of each run.

For the formal algorithmic spec (every constant, every formula), read docs/supplementary_methods.md. That document — not this README — is the source of truth for what the code does.

---

Running the pipeline

svc-pipeline [config] [options]

Argument	Meaning
config	Run-config JSON (positional, optional; default config/config.json). Bare names resolve under config/, so config.json, config, and config/config.json are equivalent. See config/README.md for schema.
--input-dir <path>	Override sig_input_dir and process only this directory.
--step {1,2,3,all}	1 = process + summary CSV only; 2 = resample only (requires Stage 1); 3 = group + average only (requires Stage 2 and groups_csv); all = every stage that's configured (default).
--groups-csv <path>	Override the config's groups_csv to group repeat scans and average them.
--group-method {mean,median,sum,min,max}	Override the config's group_agg_method for Stage 3 (default mean).
--verbose	Print INFO/DEBUG logs before and after each stage.

Expected output layout

pipeline_outputs/
├── sig_processed/<input_dir_name>/
│   ├── <truncated *.sig files>
│   └── <input_dir_name>_processed_sig_summary.csv
└── sig_resampled/<input_dir_name>/
    ├── <input_dir_name>_merged_spectra.csv     # 2101 columns (400–2500 nm)
    └── <input_dir_name>_grouped_spectra.csv    # only if groups_csv is configured

summary_csv_name, merged_csv_name, and grouped_csv_name in the run config are suffixes; the real filenames are prefixed with the input directory name. Verify your config in config/README.md.

---

Verification (R-vs-Python parity)

The Python resampler is verified against the legacy R/spectrolab script (archived_r_scripts/merge_resample_sig.R). Acceptance threshold: 1 × 10⁻³ absolute reflectance (0.1 %, well below the HR-1024i radiometric noise floor). Historical results:

Dataset	Samples	Max abs diff	Mean abs diff
Silver instrument (Serial 1202103)	66	1.10 × 10⁻⁶	4.0 × 10⁻⁸
Bronze instrument (Serial 2212118), a4any_sb_2025-cn_ch-svc-aviris_bottom	15	9.4 × 10⁻⁷	3.4 × 10⁻⁸

Run the parity test (skipped automatically without reference data):

pytest tests/test_resampler_parity.py \
    --r-reference-csv=/path/to/r_output/merged_spectra.csv \
    --r-input-dir=/path/to/processed_sig_files/

To re-run parity on a new dataset, hand the prompt at docs/parity_retest_prompt.md to any capable coding LLM. Full details in tests/README.md and docs/README.md.

---

Public Python API

The most-used entry points (all importable from their concrete modules — pipeline/__init__.py is intentionally empty):

from pipeline.sig_processor import SigFileProcessor   # truncation + instrument inspection
from pipeline.resampler      import process_sig_file, resample_spectra
from pipeline.notebook       import Spectrum, SpectraCollection, build_config
from pipeline.processor      import (
    SVCDataProcessor,      # chainable load/group/average
    SigSpectraAverager,    # facade — pass a DataFrame, get aggregated DataFrame back
    GroupSpec,             # GroupSpec.from_csv("naming_ids/<file>.csv")
    find_spectra_by_name,  # cross-DataFrame name search
)

Detailed signatures and behavioural notes in pipeline/README.md.

---

For LLMs working in this repo

1. Read FOLDER_STRUCTURE.md first — it lists every directory and the reading order.
2. Before modifying pipeline/resampler.py, re-read docs/supplementary_methods.md. The constants _FWHM_NM, _SIGMA_NM, _INTERP_WVL, _FIXED_SENSOR, _BAND_MIN, _BAND_MAX, and the algorithm steps are load-bearing for the parity claim. Changing any of them requires re-running the parity test and writing a new docs/parity_<dataset>_<date>.md.
3. The R script at archived_r_scripts/merge_resample_sig.R is frozen. Treat it as a behavioural reference, not as live code to edit.
4. Never commit machine paths or private data. Use the placeholders already present in config/config.json and the notebooks.
5. Run pytest after any change to pipeline/. The parity test will skip if reference data is unavailable; the rest of the suite still runs.

---

Demo Notebook

notebooks/pipeline_demo.ipynb is designed to be shared as a standalone file: its first code cell checks for the notebook helpers, installs svc-processing[demo]>=0.1.5 into the current kernel when needed, and falls back to the public GitHub source archive while that release is not yet on PyPI. The tutorial assumes you have an authorized folder of raw .sig scans; set DATA_FOLDER, OUTPUT_FOLDER, INSTRUMENT, and END_LINE in the opening cells. Raw field scans are not shipped in the public repository because their headers can contain time and GPS metadata. See notebooks/pipeline_demo/README.md.

Authorized parity work can still stage the separate external field dataset with scripts/prepare_demo_data.py; those raw files never enter the package or default notebook path.

---

Requirements

Python 3.11 is the supported runtime. Runtime, demo, and development dependencies are declared in pyproject.toml.

• numpy, scipy, pandas — numerical core.
• matplotlib, ipykernel — demo notebook plotting and execution.
• pytest>=8.3.0 — test runner for local verification.

Install with python -m pip install -e ".[dev,demo]" for local development and use svc-pipeline = "pipeline.cli:main" as the command-line entry point.

R is not required for the production pipeline. It is needed only to regenerate Pipeline A parity references; see archived_r_scripts/README.md.

---

License

Released under the GNU General Public License v3.0 © 2026 Cole Regnier (regs08, nr466@cornell.edu), Gold Lab (GrapeSPEC project), Cornell University. The package uses the SPDX expression GPL-3.0-only, chosen for compatibility with the GPL-3 spectrolab reference implementation.

---

Acknowledgments & Citation

This project is an independent reimplementation of the spectrolab algorithm (no source copied) and builds on prior field-spectroscopy work. See ACKNOWLEDGMENTS.md for full attribution, third-party licenses, and how to cite this software and spectrolab.