YAML-driven pipeline architecture with MVC separation#100
Merged
Conversation
Adds registry-based step dispatch, YAML config parser, batch manager, and Pipeline class. Steps are stateless functions registered via decorator. Existing imports preserved via backwards-compat shim. 52 tests passing.
…to-end 21 steps registered (filter, reduce, bin, normalize, energy axis, CCM, etc.) plus combine_runs reduction. Full XES time-resolved and XAS 2D workflows execute on synthetic data through the Pipeline. 92 tests passing. Steps registered: filter_shots, filter_detector_adu, union_shots, separate_shots, reduce_detector_spatial, reduce_detector_temporal, time_binning, normalize_xes, make_energy_axis, make_ccm_axis, ccm_binning, reduce_detector_ccm, reduce_detector_ccm_temporal, load_run_keys, load_detector, get_run_shot_properties, reduce_detector_shots, bin_uniques, purge_keys, apply_roi, patch_pixels.
Adds rotate_detector pipeline step wrapping scipy.ndimage.rotate for crystal axis alignment (replaces xes.angle on the old controller). Includes a working YAML example derived from the mfx101080524 notebook.
load_run_key_delayed leaves self.h5 open on the run object. Pool.map can't pickle h5py handles, so close it once the data is in numpy arrays.
The old flow: load full detector -> patch -> rotate -> spatial reduce. The YAML was applying ROI at load time, leaving only 16 pixels before patch_pixels tried to access index 351. Changes: - Config parser defaults rois to None (skip ROI at load when omitted) - Pipeline skips ROI kwargs when rois is None - patch_pixels now supports axis parameter for n-dimensional indexing - Example YAML restructured: load without ROI, use reduce_detector_spatial as a pipeline step after patching and rotation
…YAML pipelines
Problem:
The YAML-driven pipeline produced degraded signal-to-noise compared to the
reference notebook (XSpect_XES_mfxl1027922.ipynb). Multiple issues conspired:
1. Batch processing loaded the full detector array into memory before splitting,
defeating the purpose of batching. Worker processes then failed to reconverge
results correctly because shot masks and scalar keys weren't sliced per-batch.
2. The vonHamos energy axis formula used an incorrect offset-from-center
parameterization that didn't match the validated formula in
XSpect_Visualization.make_energy_axis (gl/2 - range/4 offset geometry).
3. reduce_detector_spatial applied ROIs along the wrong axis for 3D detector
data (shots x rows x cols), using the last axis instead of axis=1
(cross-dispersion direction).
4. rotate_detector hard-coded reshape=False, causing the static XES pipeline
to lose edge pixels after rotation.
5. patch_pixels used simple linear interpolation instead of the polynomial
fitting approach from XSpect_Analysis.patch_pixel that the reference
notebooks rely on.
Changes:
batch_manager.py:
- Load detector data per-batch directly from HDF5 (avoids multi-GB memory)
- Properly slice shot masks (xray/laser/simultaneous) per batch
- Reconverge batch results by summing temporal-reduction outputs and
concatenating shot-level arrays across all batches
- New helpers: _load_batch_from_hdf5, _slice_shot_masks, _collect_batch_attrs
pipeline.py:
- Skip detector loading when batching (pass detector_configs to batch_manager)
- Add _collect_run_attributes to expose pipeline-generated arrays on the run
- Support skip_detector parameter in _load_data
xes.py (make_energy_axis):
- Replace incorrect center-offset formula with the validated:
gl = arange(n_pixels) * mm_per_pixel
ll = gl/2 - (max(gl)-min(gl))/4
energy = 12398.42 / (2*d*sin(arctan(R/(ll+A))))
- This matches XSpect_Visualization exactly (tested to rtol=1e-12)
xes.py (patch_pixels):
- Add "polynomial" mode (now default) matching XSpect_Analysis.patch_pixel
- Fits weighted polynomial to surrounding pixels, evaluates at bad location
- Configurable patch_range, poly_range, and polynomial degree
spectroscopy.py (reduce_detector_spatial):
- Default ROI axis to 1 for 3D data (cross-dispersion), -1 for 2D
- Use np.nansum/np.nanmean instead of np.sum/np.mean
- Fix indexing to use generic axis slicing instead of hard-coded dimension
spectroscopy.py (rotate_detector):
- Expose reshape parameter (default True) from YAML config
examples/mfxl1027922_ultrafast_xes.yaml:
- New production config for Fe K-beta ultrafast XES (vonHamos A=50.6mm,
R=250mm, d=0.895Å, 40 time bins from -0.9 to 0.9 ps)
examples/mfx101080524_static_xes.yaml:
- Updated to use correct axis, reshape, and polynomial patch_pixels params
Validated: 4-core batch processing now produces identical results to
single-core (tested explicitly), and the ultrafast difference spectrum
yields SVD SV1 fraction of 88.8% matching the reference notebook.
…nd static XES
Motivation:
As we add features to the YAML pipeline (new steps, optimizations, batch
strategies), we need automated guardrails to catch silent regressions in
the scientific output. The reference notebook results are validated by
domain experts, so we freeze those outputs as .npz arrays and compare
against them in CI.
Approach:
Run each pipeline once on real LCLS HDF5 data, save key output arrays
(time-binned spectra, energy axes, difference spectra, SVD metrics) to
compressed .npz files. Tests then load these references and verify:
1. Shape correctness (array dimensions match expected)
2. Value correctness (energy ranges, spectral bounds, SVD fractions)
3. Reproducibility (re-run pipeline and compare within tolerance)
Reproducibility tests require HDF5 data on S3DF and skip gracefully when
data is unavailable. Shape and value tests always run against the .npz.
Files added:
tests/regression/__init__.py:
Package marker.
tests/regression/conftest.py:
- Registers pytest.mark.regression marker
- ultrafast_reference / static_reference fixtures (load .npz, skip if absent)
- ultrafast_pipeline / static_pipeline fixtures (run full pipeline, module-scoped,
skip if HDF5 data unavailable)
tests/regression/generate_references.py:
Standalone script to regenerate .npz files from real data. Run on S3DF:
python tests/regression/generate_references.py
Produces both reference files in tests/regression/references/.
tests/regression/test_ultrafast_xes_regression.py (19 tests):
TestUltrafastShapes:
- Verifies (40, 705) time-binned arrays, (705,) energy, (40,) time_bins
TestUltrafastValues:
- Energy range [7022.6, 7119.3] eV from vonHamos geometry (A=50.6, R=250, d=0.895)
- Energy axis matches formula to rtol=1e-12
- Difference spectrum bounds ~[-0.0066, 0.0036]
- SVD SV1 fraction ~88.8% (dominant single-component signal)
TestUltrafastReproducibility:
- Re-runs pipeline with cores=4, batch_size=500
- Compares laser-on/off arrays (rtol=1e-4), energy (rtol=1e-12)
- Recomputes avg_all_laser_off difference and SVD, compares within tolerance
tests/regression/test_static_xes_regression.py (15 tests):
TestStaticShapes:
- Verifies (707,) per-run spectra and energy axis
TestStaticValues:
- Energy range [6377.2, 6452.3] eV (Mn K-beta, A=42.75, d=0.981)
- Energy matches vonHamos formula to rtol=1e-12
- Combined spectrum peak ~10503.8 counts
- Sum of per-run spectra equals combined (float32 tolerance)
TestStaticReproducibility:
- Re-runs pipeline, compares per-run spectra (rtol=1e-4, atol=1.0)
- Verifies energy axis determinism (rtol=1e-12)
tests/regression/references/mfxl1027922_ultrafast.npz:
19 arrays from 3 runs of Fe K-beta ultrafast XES (experiment mfxl1027922)
tests/regression/references/mfx101080524_static.npz:
7 arrays from 3 runs of Mn K-beta static XES (experiment mfx101080524)
All 34 tests pass: pytest tests/regression/ -v (22s on S3DF).
Tolerances chosen:
- Energy axes: rtol=1e-12 (deterministic geometry calculation)
- Time-binned spectra: rtol=1e-4 (batch summation floating-point order)
- Difference spectrum: rtol=5e-4 (division amplifies rounding)
- SVD SV1 fraction: abs=0.02 (indirect derived metric)
- Static photon counts: rtol=1e-4, atol=1.0 (integer ADU counts)
Untitled.ipynb: Working validation notebook demonstrating the YAML pipeline end-to-end: - Runs mfxl1027922_ultrafast_xes.yaml with cores=4, batch_size=500 - Computes avg_all_laser_off difference spectrum (better SNR than per-bin) - SVD reconstruction (2 components) showing 88.8% SV1 fraction - Plots raw difference, SVD contour, and kinetic trace - Serves as the acceptance test for pipeline correctness vs reference notebook XSpect_XES_mfxl1027922.ipynb, examples/XSpect_XES_mfxl1027922.ipynb, examples/XSpect_XES_mfxl1045123.ipynb, XSpect_XES_mfx101080524.ipynb: Minor metadata/output updates from re-execution during validation.
Pipeline output HDF5 files (per-shot spectra, binned arrays) can reach hundreds of MB per run and must not be tracked in git. Add examples/results/ to .gitignore so regenerate_*.py and Pipeline.run() output is never staged.
droplet_reconstruction reads sparse photon coordinates by absolute HDF5 shot index, not by relative position within a batch. Two new attributes (abs_start_index, abs_end_index) are now injected by _make_batch_run and _process_batch_parallel so each batch worker knows its exact slice in the source file. Both are listed in _INFRASTRUCTURE_ATTRS to prevent them from being collected as result keys. precomputed_attrs solves a batch consistency problem for axis steps (make_ccm_axis, time_binning): when those steps run independently on each batch they can produce different energy/time grids from incomplete data slices. The pipeline now runs a scalar-only pre-pass on the full run, collects the resulting axis arrays, and injects them via _inject_precomputed before each batch pipeline executes. Per-shot arrays (shape[0] == total_shots) are sliced to the batch range; static axis arrays (ccm_bins, ccm_energies, time_bins) are copied as-is. reconverge_results is tightened: geometry scalars ending in _angle are averaged (not summed) across batches; axis/bin key detection now matches on exact key names first, then suffix patterns.
…ecomputed pipeline.py: when a run is large enough to batch, the pipeline now executes a lightweight scalar-only pre-pass (run_pipeline without detector data) before entering the batched loop. Axis-derivation steps (make_ccm_axis, time_binning, ccm_binning) fire here on the full set of scalar diagnostics and produce a globally consistent ccm_bins/ccm_energies/time_bins. _collect_scalar_precomputed then harvests those attributes — excluding raw input key names so that pre-pipeline-filtered copies of ipm/xray/encoder never overwrite the freshly-loaded batch-slice arrays — and passes them to run_batched as precomputed_attrs. max_shots (data.max_shots in YAML) caps the number of shots loaded per run, which is essential when testing droplet_reconstruction pipelines where each reconstructed shot expands into a full-size 2D detector array. The cap is forwarded to spectroscopy_run(end_index=max_shots) and also stored on the run object so it is picked up correctly by the fallback constructor path. config_parser.py: parse data.max_shots from YAML (int or null).
Reads per-shot sparse photon coordinates stored by smalldata's
droplet2photon algorithm and scatters them back onto dense 2D images,
yielding a (N_shots, rows, cols) array that is drop-in compatible with
all downstream pipeline steps (filter_detector_adu, patch_pixels,
rotate_detector, reduce_detector_spatial, etc.).
Two HDF5 layouts are auto-detected:
Fixed-length — <det>/droplet_droplet2phot_sparse_{row,col,data}
shape (Nshots, nData), zero-padded (valid entries != 0)
Variable-length — <det>/var_droplet_droplet2phot_sparse/{row,col,data}
flat arrays + <det>/var_droplet_droplet2phot_sparse_len
ROI-direct scattering (_scatter_roi) filters photons to the crop window
before allocating the output, so the full 704×768 panel is never
materialised when a roi is specified. For a typical (60, 300) XES
window this reduces memory by ~30x vs full-panel scatter-then-crop.
Batch-awareness: the step reads abs_start_index / abs_end_index injected
by the batch manager so each worker slices the correct HDF5 rows. In
the non-batched path it falls back to start_index / end_index.
Motivation: stochastic RIXS (spooktroscopy) requires the cleanest
possible per-shot emission spectra. Standard ADU integration accumulates
read noise on every pixel on every shot; photon counting eliminates all
counts below the single-photon threshold and produces integer photon maps
that have much better signal-to-noise for the spook correlation matrix.
…d_rotation_angle patch_pixels gains auto_detect mode for ASIC panel-gap columns. A two-method approach handles the full range of defect severity: Method 1 (ratio): compares column profile against median-filtered baseline. Columns with ratio > threshold (spikes) or < 1/threshold within the signal band (dead gaps) are flagged. Catches extreme charge-sharing spikes that dwarf the spectral signal. Method 2 (z-score + width filter): computes a robust z-score using MAD-based local sigma. Bright and dark outliers above nsigma are separately clustered and kept only when cluster width <= max_gap_width. Separating bright/dark prevents merging a dark gap with its bright charge-sharing neighbors into a single wide cluster that would be rejected as a signal gradient. This catches subtle 50%-reduced dead columns that the ratio method misses. The polynomial fit is vectorised: instead of calling np.polyfit for every (shot, row) pair, the Vandermonde matrix is solved once via np.linalg.solve to get a projection vector, then all rows are evaluated with a single dot product. The fit region now also excludes the ±patch_range neighborhood of every known bad column, not just the target pixel, so elevated charge-sharing neighbors don't bias the polynomial anchor. find_rotation_angle wraps XSpectDetectorProcessor to auto-detect the tilt of a dispersed spectral signal via Canny edge detection → DBSCAN clustering → per-cluster PCA. Stores the angle on the run for use by rotate_detector. XSpectDetectorProcessor.find_optimal_rotation_angle: fix arctan2 argument order (was arctan2(dy,dx), should be arctan2(dx,dy) for row-dominant principal vector); weight cluster angles by cluster size so small edge-detection artifacts don't bias the result. filter_detector_adu: guard getattr with a None check so the step skips gracefully when the detector key is absent on the pre-pipeline pass.
…gies length reduce_detector_ccm previously only handled 1D (scalar) and 2D (spectrum) detectors. A new 3D branch accumulates full 2D images per energy bin — producing a (n_energy, rows, cols) stack — enabling RIXS-plane pipelines where the detector is a 2D spectrometer image not yet collapsed spatially. Off-by-one in bin count: both reduce_detector_ccm and reduce_detector_ccm_temporal were allocating n_bins = len(ccm_bins) where ccm_bins is the n+1-element edge array from make_ccm_axis. They now use len(ccm_energies) (= n edges - 1) so the output axis length matches the energy axis. NaN-guarded accumulation: shots that produce NaN values (e.g. empty bins, missing HDF5 rows) are now skipped rather than poisoning the running sum. For 2D/3D detectors np.where(np.isnan(...), 0.0, ...) replaces NaN pixels element-wise so a single bad shot doesn't zero an entire spectrum row. reduce_detector_ccm_temporal gets the same ccm_energies length fix and NaN guarding for both 1D and 2D detector dims. make_energy_axis and normalize_xes: minor formatting only.
…erance
Four new test cases cover the full auto_detect surface:
- test_auto_detect_spike: confirms a 100× bright column is flagged and
patched via the ratio method (Method 1)
- test_auto_detect_subtle_gap: confirms a 50%-reduced narrow 2-col dip
is caught by the z-score method (Method 2) which the ratio threshold misses
- test_auto_detect_ignores_wide_gradient: verifies that a smooth
intensity gradient across 200 columns produces zero flagged pixels
- test_auto_detect_merges_manual: checks that auto-detected and
manually specified pixel lists are both applied
test_patch_interpolate tolerance relaxed from exact equality to atol=1e-3
because the vectorized polynomial fit (Vandermonde + np.linalg.solve)
differs from np.polyfit in numerical precision by ~1e-10 — the test was
comparing against a specific polyfit rounding, not a scientific threshold.
…nalysis notebook
Three YAML pipeline configs for the polariton / FeNO pump-probe XAS
experiment at XCS (LCLS run 26, runs 187-216 and 339-347):
xcs101591326_ultrafast_xas.yaml — static DCCM energy scan, auto CCM
axis from setpoints, laser-on vs laser-off, 1D XAS with reduce_detector_ccm
xcs101591326_temporal_xas.yaml — time-delay scan at fixed DCCM energy,
enc/lasDelay binning (-10 to 25 ps, 35 bins), reduce_detector_temporal,
combine_runs reduction across 7 runs with uncertainty propagation
xcs101591326_2d_xas.yaml — simultaneous DCCM energy × time-delay scan,
21-bin time axis (-2 to 8 ps), 2D binning via reduce_detector_ccm_temporal,
produces transient absorption map Δμ(E,t)
Analysis notebook (xcs101591326_temporal_xas_analysis.ipynb) computes
mu = If/I0 per delay bin from the pipeline output and plots normalised
difference spectra for kinetic analysis.
…eline configs
mfx101609126_pershot_xes.yaml — stochastic RIXS input preparation:
filters to xray shots, ADU thresholds both detectors, patches ASIC
gap columns (manual on epix100_0; auto_detect on epix100_1 SEER),
rotates -2.0° (spec) / -1.6° (SEER), projects rows → 1D per-shot
spectra for spook correlation. Produces:
xrt_hproj (N×2048) and epix_spec_ROI_1 / epix_seer_ROI_1 (N×n_cols)
mfx101609126_seer_xrt.yaml — static comparison of both spectrometers
across runs 78-82: sums all xray shots to 2D image then collapses
rows → 1D spectrum for each detector independently.
mfx101609126_static_rixs.yaml — RIXS plane for run 97 DCCM scan:
reduce_detector_ccm accumulates full 2D epix images at each incident
energy → (n_energy, 60, 300) stack, then reduce_detector_spatial
collapses the cross-dispersion axis → (n_energy, 300) RIXS plane.
mfx101609126_droplet_pershot_xes.yaml — per-shot XES using photon-
counting via droplet_reconstruction (epix100_0 fixed-length sparse
format), same downstream steps as pershot_xes.
mfx101609126_droplet_recon_test.yaml — 100-shot smoke test for
droplet_reconstruction; max_shots: 100 avoids materialising the full
~36k-shot array during development.
rotation_diagnostic — finds optimal detector tilt angle by running the pipeline on a subset of shots and inspecting the 2D sum image with overlaid rotation angle estimates from XSpectDetectorProcessor. Fixed angles (-2.0° / -1.6°) were determined here and hardcoded into YAMLs. pixel_patch_diagnostic — compares raw vs patched column profiles for epix100_0 and epix100_1, visualises the ASIC gap positions and the effect of auto_detect patching on the baseline ratio and z-score. seer_shot_browser — per-shot SEER image browser: scan forward/backward through individual shots to inspect photon distributions and confirm the ASIC gap correction is working correctly. seer_vs_xrt — quantifies SEER as an alternative incident-energy monitor for stochastic RIXS. Computes correlation between SEER and XRT per- shot total-intensity and center-of-mass energy. Conclusion: SEER tracks intensity well (r=0.90) but has low COM energy correlation (0.41) because it encodes sample transmission/emission, not purely the incident beam. Gap patching dramatically improves the spook AtA condition number. spook_rixs — stochastic RIXS extraction using the spook library: builds A (XRT hproj) and B (epix_spec per-shot spectra) matrices and solves for the RIXS cross-section X via regularised least squares. droplet_spook_rixs — same as spook_rixs but using photon-counted spectra from droplet_reconstruction instead of raw ADU integration. droplet_spook_transmission — stochastic RIXS using SEER (transmission geometry) as the B matrix; photon-counted for improved S/N. rixs_plane — static RIXS plane visualisation for the DCCM scan run.
regenerate_run0079.py — runs mfx101609126_pershot_xes.yaml (standard ADU integration) and saves per-shot spectra to examples/results/; used as the baseline to compare against droplet-reconstructed spectra. regenerate_droplet_pershot.py — runs mfx101609126_droplet_pershot_xes.yaml for runs 78-82 sequentially, saving one HDF5 per run to results/. Measures reconstruction throughput (shots/sec) and reports memory usage. droplet_reconstruct.py — standalone utility script that reconstructs a single run's sparse photon data without the full pipeline framework; used for rapid iteration and debugging of the reconstruction step. droplet_reconstruction.ipynb — step-by-step walkthrough of the sparse format: reads raw HDF5, shows fixed-length vs variable-length layouts, runs _scatter and _scatter_roi helpers manually, compares output images against the pre-processed ROI_area dataset for bit-identity verification. droplet_recon_pipeline_eval.ipynb — end-to-end comparison of ADU integration vs photon-counting via the pipeline API. Runs both YAMLs on run 79, overlays 1D emission spectra, and evaluates peak S/N ratios to quantify the photon-counting improvement for spook inputs.
…c assets mfx101080524_static_xes.yaml: remove hitfinding step. The hit-finder was removing shots that had low per-frame median signal, which is expected for single-photon-level XES data. Removing it recovers the correct shot count and avoids discarding weak-signal runs. seer_full_detector_run78.png: full-panel SEER detector image from run 78 showing the spatial distribution of signal across all ASIC tiles; used in rotation_diagnostic.ipynb and seer_shot_browser.ipynb to orient the cross-dispersion ROI selection. c.ipynb: scratch notebook for static RIXS pipeline diagnostics (run 97); shows RIXS plane shape, energy range, and 2D sum-image ROI inspection. xcs101591326.ipynb, Untitled.ipynb: updated experiment analysis notebooks.
…MFX pipelines Droplet reconstruction, patch_pixels auto-detect, XAS fixes, and XCS/MFX pipelines
…, and analysis notebooks
Adds the full mfx102101026 (ferricyanide) static XES analysis workflow and the
supporting XSpect engine changes needed to run it at scale, plus the earlier
mfx100895324 static-XES artifacts.
XSpect engine changes
----------------------
* model/run.py, controller/{config_parser,pipeline,batch_manager}.py:
Add an import-time row-range crop for detector keys (`row_range: [start, end]`
in the YAML `detector_keys` block). The slice is applied at HDF5 read time in
both the single-run (`load_run_key_delayed`) and batched (`_load_batch_data`)
paths, so the full detector frame is never materialized. For epix100_0 this
cuts per-batch memory ~5-7x and was necessary to stop the pipeline dying in
silent OOM. The crop origin is recorded on the run (`run._row_offset`) so that
downstream ROIs stay in absolute (full-frame) coordinates.
* analysis/spectroscopy.py:
- reduce_detector_spatial now auto-translates absolute ROI coordinates into
the cropped-array frame using the recorded row_range offset (stripping
_reduced/_ROI_* suffixes so derived keys inherit the parent offset). This
fixes ROIs silently reading out of bounds (Ka summing to zero) after a crop.
- hitfinding gains an absolute `min_sum` mode to reject genuinely dark shots
(sum == 0 after ADU threshold); the old relative median-cutoff mode is kept
as a fallback for signal-dominated data.
* controller/pipeline.py, pipeline_runner.py, __init__.py:
Add a real `logging`-based progress system on the "XSpect" logger, exposed via
`enable_logging(level, log_file=None)` (stderr and/or file). The batched path
now reports per-batch completion via imap_unordered and raises an explicit
OOM-hint error if a worker dies, replacing the previous silent hang. Per-step
debug logging and full tracebacks are emitted from run_pipeline.
mfx102101026 experiment (experiments/mfx102101026/)
---------------------------------------------------
* mfx102101026_static_xes.yaml: static Fe Ka/Kb pipeline on epix100_0
(transpose=true, row_range crop, hitfinding min_sum, ferricyanide ROIs).
* mfx102101026_static_xes_visualization.ipynb: per-run pipeline + diagnostics.
* mfx102101026_aggregate_xes.ipynb: cross-run aggregation from saved results.
* mfx102101026_ferricyanide_reference.ipynb: aggregates runs 9+10, minimum
subtraction then area normalization, saves the ferric (Fe3+) reference.
* mfx102101026_run9_baseline_speciation.ipynb: crop -> baseline -> normalize ->
speciation on the fixed windows (Ka 6380-6410, Kb 7031-7075 eV); baseline
subtraction currently disabled (passthrough) pending a robust routine.
* mfx102101026_speciation_simple.ipynb: non-expert runner. User lists runs;
runs are XSpect-processed only if not already cached (per-run HDF5), then
combined -> min-subtract -> normalize -> speciation vs the ferric reference,
with a difference spectrum and an eLog beamline-summary section (compiles the
figures, labeled by run, into stats/summary/XSpect_speciation via SummaryPost).
* mfx102101026_droplet_visualization.ipynb: droplet sparse-data viewer. Reads
only ROI droplets by streaming contiguous slabs of the first N sparse columns
(avoids the 28 GB full-array read that hung the notebook); documents that
droplet coordinates are in the raw, non-transposed detector frame.
* runs.csv: run bookkeeping; runs 4/5/8 flagged as no-beam, run 9 first 30 Hz.
* results/*.h5: cached per-run and aggregate spectra, ferric reference.
* FeIIICN6_*.dat: exported Ka/Kb reference spectra.
mfx100895324 (experiments/mfx100895324/) and examples/
------------------------------------------------------
Adds the static-XES notebooks, YAML, runs.csv, manifest/README, and saved
results that the mfx102101026 work was derived from, plus a spike-finder
example notebook.
Note: the LUTE smalldata configs and epix common-mode diagnosis for
mfx102101026 live in the experiment data area (not this repo); summary_post.py
lives alongside the notebooks in results/lbgee/ for the same reason.
…r mfx102101026 Adds a repository skill documenting how to stand up a new XES experiment end-to-end, plus the droplet-reconstruction pieces it references. skills/XSpect-setup-XES/SKILL.md - Phased guide (experiment-type triage -> LUTE smalldata pipeline -> geometry discovery -> XSpect YAML -> diagnostic/analysis notebooks -> issue tracking). - Decision questions to classify the analysis: static, time-resolved pump-probe, CCM-scanned/RIXS, or droplet/photon-counted XES. - YAML templates for each type, memory (row_range) guidance, the ePix100 common-mode gotcha, droplet vs droplet2photon distinction, and the LUTE droplet2photon template gap (issue #97). - Guardrails: never invent step names, verify units (ADU vs keV) and the transpose/ROI coordinate frame, confirm beam-on before debugging. - README links to the skill and the YAML pipeline guide. experiments/mfx102101026/ - mfx102101026_droplet_recon_xes.yaml: native photon-counted workflow using the droplet_reconstruction step (ROI applied on sparse arrays) -> union -> sum -> rotate -> reduce, modeled on the working mfx101609126 droplet pipeline. - droplet_visualization.ipynb: Section 7 reconstruction now uses the native XSpect droplet_reconstruction step instead of an external helper script.
…ing conventions Adds Phase 0.5 (Identify the HDF5 keys) and strengthens the guardrails so the skill enumerates datasets and asks the user when a role is ambiguous rather than guessing paths. - Role -> naming conventions: emission = ePix100 (epix100_0/1, epix_0/1 and variants); scattering = epix10k2m / jungfrau16m (usually not the XES signal); I0 monitor = ipm<N>/sum in standard hutches but MfxDg*BmMon/... at MFX; lightStatus masks, timing keys, CCM scan variable, von Hamos geometry PVs. - Ambiguity protocol: 0 candidates -> ask / offer to skip; >1 -> list with shapes and ask; always echo the final key->role mapping for confirmation. - Notes that with two ePix100s (spectrometer + SEER) the agent must ask which is which instead of assuming _0 is the spectrometer.
…ring Zeros low-variance detector pixels via sklearn VarianceThreshold instead of a hand-tuned ADU cutoff. Pixels that barely change across shots (dead, static hot, constant background) are dropped; signal-bearing pixels are kept. Writes back to the same detector key and stores the retained mask as <key>_variance_mask. Handles 3D and 2D detectors. Closes #36
…itecture Brings in the mfx102101026 static + droplet Fe Kα/Kβ XES analysis and the XSpect engine features developed for it: - Import-time detector row_range crop with automatic ROI offset translation (model/run.py, controller/batch_manager.py, analysis/spectroscopy.py) — the memory fix that prevents OOM in batched runs. - hitfinding min_sum absolute mode to reject dark shots. - logging system with enable_logging() and per-batch progress (controller). - experiments/mfx102101026/: static XES YAML, droplet recon YAML, and visualization/aggregate/reference/speciation/droplet notebooks. - experiments/mfx100895324/: predecessor static-XES artifacts. - skills/XSpect-setup-XES/: repository skill for standing up a new XES experiment (LUTE + XSpect YAML + diagnostics), with HDF5 key identification. Run 26 tracking: experiment #98, tracked under #99; open dev items #97, #96, #70, #55, #45.
…ne-architecture # Conflicts: # XSpect_XES_mfx101080524.ipynb
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the monolithic controller workflow with a YAML-driven pipeline. Analysis is now defined declaratively: a config file lists ordered steps, and a registry maps step names to stateless functions. Model (run/data), controller (batch manager, pipeline runner, config parser), and analysis steps are separated so the same steps run across XES, XAS, RIXS, and droplet workflows.
Merges
origin/master(XAS foil-scan / derivative-analyzer work, PR #90) into the branch, so this PR merges cleanly.What's in it
XSpect/analysis/registry.py), config parser, batch manager with multiprocessing, pipeline runner. Issue Architecture overhaul: YAML-driven pipeline with MVC separation #82.droplet_reconstructionfor sparse photon-counting HDF5,patch_pixelswith auto-detection,find_rotation_angle/rotate_detector, andfilter_detector_variance(sklearn VarianceThreshold as a data-driven alternative to ADU filtering, Feature: variance thresholding from sklearn as an alternative to Adu filtering. #36).docs/YAML_PIPELINE_GUIDE.md. NewXSpect-setup-XESskill.Backward compatible: old imports (
from XSpect.XSpect_Analysis import spectroscopy_run) still work.Reviewer notes
XSpect_XES_mfx101080524.ipynbwas deleted (master's reorg); the canonical copy lives atexamples/XSpect_XES_mfx101080524.ipynb.XSpect/XSpect_Analysis.pyauto-merged — worth a look since both sides touched it.Test plan
pytest tests/test_xes_steps.py tests/test_spectroscopy_steps.py(32 pass locally)examples/mfx101080524_static_xes.yaml)test_xas_steps.py(CCM off-by-one) andtest_batch_manager.pyexist independent of this branch's merge