Add common_mode_correction and subtract_polynomial_background steps#103
Merged
Conversation
common_mode_correction subtracts per-row, per-column, or per-bank electronic baseline estimated from a signal-free reference band, on 3D detector data before shot reduction, preserving shape. subtract_polynomial_background fits a low-order polynomial to signal-free regions along the spatial axis and subtracts it, writing a non-destructive <key>_bkgsub. peak_mask accepts multiple ranges so several emission lines dispersed on one detector can be masked at once. Per-row weighting excludes NaNs from the fit. Reuses the vectorized weighted-polyfit approach from patch_pixels. Adds unit tests for both steps and documents them in the YAML guide and README step lists. Closes #102
There was a problem hiding this comment.
Pull request overview
Adds two new, registered spectroscopy pipeline steps to support detector baseline correction and spatial polynomial background subtraction (Issue #102), along with documentation and unit tests so they can be configured via YAML and verified in isolation.
Changes:
- Added
common_mode_correctionstep to subtract per-row/column/bank common-mode offsets using a signal-free reference band. - Added
subtract_polynomial_backgroundstep to fit/subtract a low-order polynomial baseline along a spatial axis, writing<on>_bkgsuband supporting multi-range peak masking. - Extended docs/README step listings and added unit tests covering key behaviors and edge cases.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
XSpect/analysis/spectroscopy.py |
Implements the two new registered steps (common_mode_correction, subtract_polynomial_background). |
tests/test_spectroscopy_steps.py |
Adds unit tests validating correctness, shape preservation, NaN handling, and non-destructive output behavior. |
README.md |
Updates the “Registered steps” list to include the two new steps. |
docs/YAML_PIPELINE_GUIDE.md |
Documents the new steps, their parameters, and YAML configuration examples. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ) | ||
| return | ||
|
|
||
| reducer = np.nanmedian if method == "median" else np.nanmean |
| band = working | ||
| n_cols = working.shape[2] | ||
| corrected = working.copy() | ||
| for start in range(0, n_cols, bank_size): |
Comment on lines
+280
to
+283
| Works on a 1D spectrum or a 2D array (bins x pixels). The fit reuses the | ||
| vectorized weighted-polynomial projection from patch_pixels: the offset | ||
| vector depends only on the sample positions and weights, so it is built | ||
| once and applied to every row with a single matrix multiply. |
Comment on lines
+319
to
+323
| if background is not None: | ||
| weights[:] = 0.0 | ||
| for rng in background: | ||
| weights[rng[0] : rng[1]] = 1.0 | ||
| elif peak_mask is not None: |
Comment on lines
+352
to
+356
| # A[r] = V^T diag(w_r) V ; b[r] = V^T diag(w_r) flat_r, batched over rows r. | ||
| A = np.einsum("pk,pr,pl->rkl", V, w_full, V) # (N_rows, order+1, order+1) | ||
| b = np.einsum("pk,pr->rk", V, w_full * flat) # (N_rows, order+1) | ||
| coeffs = np.linalg.solve(A, b) # (N_rows, order+1) | ||
| baseline = (V @ coeffs.T) # (n_pixels, N_rows) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the two steps from #102.
What changed
common_mode_correction(XSpect/analysis/spectroscopy.py): subtracts a per-row, per-column, or per-bank electronic baseline estimated from a signal-freereferenceband. Runs on 3D detector data (shots x rows x cols) before shot reduction; also handles 2D. Preserves input shape.methodismedian(default) ormean;axisisrow/column/bank;bank_sizedefaults to 128 (ePix100).subtract_polynomial_background(same file): fits a low-order polynomial to signal-free regions along the spatial axis and subtracts it. Non-destructive, writes<key>_bkgsub. Fit regions named either bybackgroundranges or by excludingpeak_mask.peak_masktakes a single range or a list of ranges, so multiple emission lines on one detector can be masked at once. Per-row weighting drops NaNs from each row's fit and leaves them NaN in the output. Reuses the vectorized weighted-polyfit (Vandermonde + batched normal-equation solve) frompatch_pixels.Reviewer notes
einsumto build per-row normal equations,np.linalg.solveover the batch). The batch dimension is small (order+1 = 2-3), so cost is dominated by the data-sizedeinsum, same order as the single-solve version.common_mode_correctionwas placed in the Shot Filtering table;subtract_polynomial_backgroundin the XES-Specific table.How to test
31 passing (7 common-mode, 8 background-subtraction, rest pre-existing). The 11 failures in the full suite (batch_manager, xas, regression) are pre-existing on master and unrelated to this change.
Closes #102