SpyDE 0.3.0 — fitting, EELS/EDS, EBSD, atom mapping, and the data module - #89
Merged
Conversation
Five waves, each independently shippable behind its own optional extra. Tracked in #48 with a sub-issue per task. Decisions recorded up front, because each one shapes the code: - spyde/models/ is already the neural disk detector, so the HyperSpy model port lives in spyde/fitting/. Unrelated meanings of "model". - exspy/kikuchipy/atomap ship as optional extras gated by a new requires_package key, mirroring requires_vectors. - The fitting engine is a batched GPU Levenberg-Marquardt over the whole nav grid (the vector_orientation_gpu.py playbook), with SAMFire's neighbour-seeding kept as the seed source rather than the scheduler. Measured baseline that motivates it: multifit on 1024 spectra x 1024 channels runs 10.83 s = 95 spectra/s single-threaded, which extrapolates to 11.6 min for a 256x256 SI. Parity with multifit is the acceptance test, not convergence. Also found while measuring: numexpr is not installed, so every hyperspy Expression component silently falls back to numpy.
Waves 1-4 need something loadable per modality from day one, before the real datasets land (#80). These are also the better thing to test against: the ground truth is known exactly, so a fit or an indexing run is scored against the numbers the data was built from rather than a golden file that quietly encodes yesterday's bug. eels_si power law + C/N/O K edges at real onsets, intensities following known concentration maps eds_si bremsstrahlung + Fe/Ni/Cu K families at real energies, with Fe-Kb/Ni-Ka overlapping ON PURPOSE so family-aware fitting is actually exercised rather than assumed ebsd_patterns real gnomonic Kikuchi-band geometry for a cubic crystal over a two-grain orientation field; the exact Euler angles are stamped on the signal, so dictionary indexing can be checked against truth instead of against a fixture of its own output Every spatial map is distinguishable from its transpose and both mirrors, so a flipped axis fails a test instead of looking fine — the si_grains/movie precedent. Ground truth is read with spyde.data.ground_truth(), not straight off the metadata: hyperspy turns nested metadata dicts into DictionaryTreeBrowser nodes, which have no .values(). None of this needs exspy or kikuchipy. set_signal_type is attempted and its "not understood" warning demoted to debug, so a user without the extras gets usable data and no scary warning. Reachable as load_test_data_eels / _eds / _ebsd.
Closes the framework half of #49. requires_package: hides a toolbar action until its optional extra is importable, mirroring requires_vectors — so a user without spyde[eels] never sees a button that would raise ImportError on click. Resolved with find_spec, so it never imports the package and never pays its import cost; cached, since an extra cannot appear mid-session. Applied in BOTH filter paths. get_toolbar_actions_for_plot and _action_matches_plot duplicate the gate list, and adding a key to one renders a button that never dispatches. A test asserts both contain it. Extras: eels (exspy), ebsd (kikuchipy), atoms (atomap), all. Deliberately not core deps — each pulls a large tree and the frozen app must stay installable for someone who only does 4D-STEM. A CI job installs spyde[all] so the gated code actually executes; the default job proves the app still works with none of it. NOT adding numexpr, and pyproject now says why. Hyperspy warns "Numexpr is not installed ... which is slower to calculate model" on every model build, but at spectrum sizes that advice is backwards. Measured on a 1024-channel multifit: absent 110 spectra/s numexpr, 1 thr 77 numexpr, 4 thr 75 numexpr, 48 thr 70 numexpr's per-call setup and thread dispatch dominate arrays of ~1k elements. The warning will keep inviting someone to add it, hence the comment.
Ports hyperspy's model concept and makes it fast. #51, #52, #53. Lives in spyde/fitting/, NOT spyde/models/ — that name is already the neural disk detector. Unrelated meanings of "model". spec.py ModelSpec: serialisable, round-trips against hyperspy's BaseModel.as_dictionary(), so a model built here opens in plain hyperspy. Packed parameter order IS the API — column j always means the same parameter. components.py batched autograd-differentiable ports of hyperspy's components; no Python loop over pixels anywhere engine.py batched Levenberg-Marquardt over the whole nav grid Why it works: n (free parameters) is tiny even when P and C are large, so the normal equations are (P, n, n) — a batch of small solves torch.linalg does natively. The one large object is the (P, C, n) Jacobian, so P is chunked. Jacobian by jacfwd not jacrev: forward-mode costs one pass per INPUT (n~10), reverse one per OUTPUT (C~2000). One implementation, not two. The "CPU fallback" is this code with device="cpu"; a separate scipy path would be a second thing to keep in agreement with hyperspy, and hyperspy already IS the reference — the acceptance test asserts parameter parity with multifit directly, because "it converged" proves nothing when you are replacing a reference implementation. Measured, 1024 spectra x 1024 channels, PowerLaw + 3 Erf (11 free params): hyperspy multifit 214.80 s 4.8 spectra/s batched cpu 27.02 s 37.9 spectra/s 7.9x batched cuda 3.97 s 257.8 spectra/s 54.1x Two numerical traps found by measurement, both documented at the code: - Column scaling is load-bearing, not a refinement. A PowerLaw fits A ~ 1e6 alongside r ~ 3, so cond(JtJ) ~ 1e12 and float64 Cholesky loses nearly every digit in the small-eigenvalue direction. Unscaled: 247 iterations for a TWO parameter fit, stopping 1.5-3.5% short of multifit. Scaled: 46 iterations to chisq ~1e-34. - Convergence must be a GRADIENT test. Testing step size alone reads "stuck against a wall with large lambda" as converged; requiring an ACCEPTED step instead never fires at all, because at the optimum nothing improves. Convergence on hard many-parameter models is still seed-limited (hyperspy hits maxfev on the same model) — that is what seeded propagation (#54) is for.
Answers "is PyTorch really the right way to fit?" — the honest answer was
"yes as a vehicle, no as I was using it". Profiling the solver:
residual evaluation 3.4 ms
jacfwd Jacobian 51.6 ms <- 94% of the solver
Forward-mode AD costs one forward pass per parameter, and these components have
closed-form derivatives that mostly reuse the value already computed
(df/dA IS the shape for every linear amplitude). So components now carry
analytic gradients and the engine builds the Jacobian directly; autodiff stays
as the fallback for any component without one.
batched cpu 37.9 -> 105.9 spectra/s (2.8x)
batched cuda 257.8 -> 361.6 spectra/s (1.4x)
The GPU gains less because it is now dispatch-bound, not compute-bound — the
remaining lever there is kernel fusion, not more math.
Also: the chunk heuristic was starving the GPU. Capping the Jacobian at 2**22
elements chopped a 1024-spectrum batch into pieces of 372 and cost a third of
the throughput (178 vs 267 spectra/s). Raised to 2**26 so typical scans run
unchunked.
Correctness is unchanged and still defined by hyperspy: every analytic gradient
is asserted equal to autodiff's, which is the ideal oracle since it is derived
mechanically from the value function and cannot share a mistake with a
hand-written formula.
seeding.py implements #54 — coarse strided fit, propagate to nearest neighbour,
one batched refine. SAMFire's insight (a neighbour's result is the best start)
without SAMFire's per-pixel scheduler. Only CONVERGED coarse fits propagate: a
failed one has wandered somewhere unphysical and seeding from it spreads the
failure rather than the answer.
Two benchmark corrections, both of which had been quietly measuring the wrong
thing:
- With exspy installed, create_model() returns an EELSModel that ALREADY has a
background, so the reference model had TWO PowerLaws while the batched side
had one. Both sides must fit the same model or the comparison means nothing.
- The EELS benchmark model cannot represent a hydrogenic edge, so neither
engine converges and it was timing the iteration cap. Added a well-posed EDS
case (gaussian peaks, which the library expresses exactly) as the default;
EELS stays as the deliberately-hard timing-only case, documented as such.
On the well-posed case: 57.6% converged, 361 spectra/s, 11.6x multifit.
The reason Wave 3 exists. Normalised cross-correlation between an experimental and a simulated pattern is, once both are zero-mean and unit-norm, EXACTLY a dot product — so indexing P patterns against D dictionary entries is one matmul plus a top-k, which is the operation a GPU is built for. The (P, D) similarity must never exist (65k x 100k float32 = 26 GB), so both axes are tiled with a RUNNING top-k: each tile is reduced against the best-so-far and discarded. A test asserts tiled == untiled, because otherwise results would quietly depend on chunk size. preprocess.py adds static/dynamic background removal and the average dot-product map, all batched over the whole scan. Why background removal is not cosmetic, measured here: NCC is invariant to gain and offset but NOT to a spatial GRADIENT. The detector background is in every experimental pattern and in no simulated one, so it dilutes every score. An exact-orientation match scores ~0.92 with it left in and >0.99 once removed — and the correction must be applied to BOTH sides, or the dictionary keeps low-frequency content its counterpart no longer has. Correctness comes from ground truth, not from a fixture: the synthetic EBSD data stamps the exact Euler angles it was generated from, so the tests assert that indexing recovers them, that the two grains index to disjoint entries (a blank map would pass every score check), that a uniform grain indexes consistently, and that a finer dictionary matches monotonically better. One real bug caught by its own test: the shape guard compared pixel counts AFTER reshaping the experimental stack to the dictionary's pixel count, which silently succeeds whenever the sizes share a factor — a 40x40 scan against a 20x20 dictionary reshaped to 4x as many "patterns" and indexed garbage rather than raising. Nothing here needs the ebsd extra; indexing is plain torch + numpy. Only the IO/geometry/master-pattern layer (#69) will, and that is requires_package-gated.
Dictionary indexing can only ever return a dictionary ENTRY, so its accuracy is capped by the sampling step — a 5-degree dictionary gives 5-degree answers. Refinement lifts that cap by optimising each orientation continuously from its indexed match. Same shape of problem as actions/vector_orientation_gpu.py, solved the same way: the whole field at once. Every pattern's three Euler angles are one row of a (P, 3) tensor, the simulated patterns are one batched forward pass, and one Adam optimiser walks all P orientations together. No dask, no per-pattern loop. Two things carried over from the vector-orientation work rather than rediscovered (CLAUDE.md, GPU Computing): - backward() segfaults off the main thread under CUDA on Windows, so the loop runs INLINE with an on_yield callback instead of being pushed to a worker. - Yield INSIDE the step loop, not between stages, or a UI freezes for seconds while the progress bar looks stuck. A test asserts on_yield fires during the optimisation, not just around it. Refinement never returns a WORSE orientation than it was given: it is an improvement step on top of indexing, so if Adam wanders off (bad start, too large an lr) the indexed answer is still the better estimate and is what comes back. Tested by driving it to diverge on purpose with lr=5. The simulator is injectable and the built-in one renders the same band geometry as the synthetic data — asserted equal to the numpy generator, since refining against a DIFFERENT forward model than the one that made the data would be fitting the wrong thing. #69 swaps in kikuchipy master patterns without touching the optimiser. Orientations are compared by the PATTERNS they produce, never by Euler distance: Euler angles are not a metric space and cubic symmetry means different triples describe the same crystal.
The display half of Wave 3, and it is nearly free by design. SpyDE already depends on orix and already has IPF views, orientation maps and a 3D IPF toolbar from the 4D-STEM work, so the job is to hand the EBSD result over in the form that machinery already speaks — not to build a second orientation display beside the first. ipf_colors returns (H, W, 3) RGB, which commit_result_tree already accepts as an RGB primary. orientation_similarity_map is the one piece with non-obvious semantics, and it earns its place: it measures how much each position's RANKED match list agrees with its neighbours', which exposes indexing that is confidently WRONG. A raw score map cannot — a pattern that matched something strongly scores just as highly whether or not the answer is right. Tested on exactly that case: one position indexed to a completely different entry from its neighbourhood reads 0.0 while the rest of the map reads > 0.9. merge_phases combines per-phase runs by score, since multi-phase indexing runs one dictionary per phase. This commit is COMPUTE ONLY. Putting these on screen is UI work and gets verified by running the app and looking at pixels (CLAUDE.md), not by these tests. The closest headless proxy is asserted instead: index the synthetic scan, colour it, and require the two grains to come out distinguishable — identical colours would mean the map renders as a flat block and would still pass every range check. Two small fixes while writing the tests: - merge_phases validated coverage AFTER np.stack had already raised "all input arrays must have the same shape", which says nothing about which phase disagreed. Validate first, and name the sizes. - numpy's assert_allclose no longer broadcasts shapes, so comparing (N, 3) against row 0 fails on shape even when every row is identical.
Gaussian2D through the SAME engine, not a second code path. An image is flattened to H*W sample points exactly as a spectrum is C channels; the only difference is that the "axis" carries (x, y) PAIRS instead of scalars, handled by one branch in TorchComponent._split. Packing, the Jacobian, bounds, fixed parameters and the LM solve are all the 1-D machinery unchanged — and the tests assert that rather than assume it (bounds and fixed-parameter tests are repeated in 2-D precisely because a different shaping path is where they would break). image_coordinates() builds the coordinate grid, row-major so it lines up with a .ravel()'d image. The test image is deliberately NON-SQUARE (24x28): a transposed coordinate grid would run happily on a square one and put every peak in the wrong place. As in 1-D, A is the VOLUME under the surface rather than the peak height — transcribed from hyperspy's expression, with parity, batching and analytic-vs-autodiff gradient tests to keep it honest. This also unblocks Wave 4: refining atom positions IS a batched 2-D gaussian fit (#77).
Division of labour, and why this wave is small: atomap owns the STRUCTURE (peak finding, sublattices, neighbours, zone axes, dumbbell pairing) because reimplementing it would diverge from published atomap results. SpyDE owns the REFINEMENT, because refining atom positions IS a batched 2-D gaussian fit and spyde.fitting already does the whole field in one Levenberg-Marquardt where atomap fits one atom at a time with scipy. Only find_atoms needs the atoms extra; refinement and the property maps are plain numpy/torch, so they are testable and usable without it. Correctness comes from ground truth: the new atom_lattice() generator knows exactly where every atom is, so refinement is scored against those positions rather than against atomap's answer. The full pipeline (find -> COM -> gaussian) recovers them to a MEDIAN error of 0.2 px, and refinement alone to better than 0.1 px worst-case. Tolerances are tight deliberately — a test accepting +/-1 px would pass with the refinement removed entirely. Details that are easy to get wrong and are pinned by tests: - A pinned atom (the red/green toggle, #76) is EXCLUDED from the fit, not fitted and discarded — otherwise the work is wasted and a diverging neighbour can still perturb the shared solve. - A fit that leaves its own box keeps its INPUT position. A wild coordinate is worse than an unrefined one. - Gaussian A is a VOLUME, so the amplitude seed is scaled by 2*pi*sigma^2; seeding it with the peak height starts every fit an order of magnitude low. - Ellipticity is max/min sigma, never sigma_x/sigma_y: the latter drops below 1 for an atom elongated along y and puts a spurious boundary wherever the elongation direction changes. - Displacement uses a LOCAL reference (the centroid of k neighbours), so it measures real distortion and is immune to drift or a tilted scan, which a globally-fitted ideal lattice would report as displacement. atom_lattice() also grows dumbbell / displacement / ellipticity knobs so the downstream maps have something real to recover instead of a uniform zero.
Both shaped their output with `expand_as` / `expand`, which forces the
parameter block's P down to the axis's leading 1 and raises for P > 1:
RuntimeError: The expanded size of the tensor (1) must match the
existing size (4) at non-singleton dimension 0
Plain broadcasting (`p + 0.0 * x`) gives (P, C) for both a shared axis and a
per-position one.
Found by fitting a real EDS composition model, whose background is a
Polynomial — not by the component tests, because every one of them used a
SINGLE spectrum, where expand_as happens to work. That is the actual defect
here: a batched component library whose tests never batch.
Closed with TestEveryComponentBatches, which runs every component (and
Polynomial) with P=4 and asserts the value shape, the gradient shape, and that
row 1 reflects ITS OWN parameters — a broadcasting slip that returns row 0 for
everything would still have the right shape.
Wave 2's headline: type which elements are present, get a populated model. exspy owns the spectroscopy — edge and line energies, GOS edge shapes, family relationships — because reimplementing it would mean diverging from the numbers the community publishes. This module owns the plumbing: - one call for EELS and EDS alike returning a ModelSpec, so the result feeds the batched engine, the wizard and the renderer through one type; - pruning to the measured range. A component with no data under it is NOT harmless: its amplitude is unconstrained, so the optimiser trades it against everything else and degrades the parameters that ARE measurable. exspy will happily add a Cu-L line at 0.93 keV to a spectrum whose useful range starts at 2 keV; - an honest report of whether the batched engine can fit the result. That last point is asymmetric today and worth knowing: EDS -> Polynomial + Gaussian -> fully supported, gets the GPU path NOW EELS -> EELSCLEdge (GOS table) -> no batched port yet (#63), falls back to hyperspy, which is correct but slower Reported in the returned info rather than left as a surprise. Two exspy behaviours that do not do what their names suggest: - `add_lines` APPENDS, so restricting needs `set_lines`. - Even then, the EDS model expands each ELEMENT into its whole family (selecting Fe_Ka still builds Fe_Kb, Fe_La, Fe_Ln, ...) regardless of metadata.Sample.xray_lines — so honouring only_lines means filtering the built spec by name. The source signal is never mutated; a user may be exploring several compositions.
Two bugs of the same shape, found by round-tripping a composition model. to_model() rebuilt every component BARE, which breaks for anything that takes constructor arguments: - EELSCLEdge raises outright (element_subshell is required), so no EELS model could be turned back into hyperspy at all. - Polynomial is worse because it does NOT raise. Rebuilt bare it comes back at its DEFAULT order, so exspy's order-6 EDS background silently lost a3..a6 — to_model skipped the parameters that no longer existed and returned a different model that still fits and still looks plausible. ComponentSpec now carries init_args, captured by from_model via a small _INIT_ARGS table and used when rebuilding. An unrebuildable component gets an error naming the table to add to, rather than a bare TypeError from inside hyperspy. The original round-trip tests missed both because they only ever went model -> spec, never spec -> model with a component of this kind. Fixing that surfaced a third: EELSCLEdge.fine_structure_coeff is a VECTOR parameter with 8 elements, and ParameterSpec stored a single float, so hyperspy rejected the assignment on length. Vector parameters now round-trip faithfully (value becomes a list) but are excluded from the PACKED vector via ComponentSpec.scalar_parameters — the solver puts one scalar per column, and no component with such a parameter is fittable by the batched engine anyway (components.supports already says no). components.py uses the same scalar view so the two cannot drift apart.
…oo (#63) An EELS core-loss edge is not a formula — EELSCLEdge looks its shape up in a GOS table, which is why it had no batched port and why an EELS model fell back to hyperspy's one-pixel-at-a-time fitting while an EDS model (gaussians) got the GPU. The way out is to notice what is actually being fitted. Across a spectrum image the edge SHAPE is the same everywhere — atomic physics, fixed by the element and the microscope. What varies pixel to pixel is HOW MUCH of the element there is and exactly WHERE its edge starts. So the edge is sampled once on the signal axis and replaced by a tabulated component fitting intensity (linear) and onset_shift, batched like everything else. The approximation, stated rather than buried: fine structure and effective angle are FROZEN at the values the table was sampled with. They are what make an edge a lookup in the first place, and refitting them would mean re-sampling the table every iteration — the per-item Python work the batched engine exists to avoid. tabulate_model() reports what it froze. Right trade for quantification; wrong one for fine-structure analysis, which is why it is an explicit call and not automatic. Two deliberate choices in the component: - Outside the table the value HOLDS at the end sample rather than extrapolating. Extrapolating a GOS tail would invent signal where the measurement has none. - The onset-shift derivative is a central difference over one channel, not the exact piecewise-linear slope. The exact one is a step function that jumps at every segment boundary, so LM chatters as the shift crosses a channel. It differs from autodiff by ~1.5% of the gradient's own scale, only within a channel of a kink, and the test states that as the property rather than asserting an equality that is not true. Tested end to end: the table reproduces hyperspy's edge to 1e-9, a fit recovers both a known intensity and a known 12 eV onset shift, and on the synthetic SI the fitted O_K intensity tracks the concentration map the data was built from.
Two steps, kept separate because they fail differently. element_intensity_maps() is the BRIDGE: fitted parameters -> one intensity map per element, reading the component names the model was built with (Fe_Ka, O_K) and summing a family's lines. Pure array work, no physics, no optional extra. One bridge serves EDS and EELS because the naming shape is the same and the intensity parameter is looked up per component kind (Gaussian's A, TabulatedShape's intensity) — otherwise the EELS path silently produces no maps. quantify() is the PHYSICS: relative, Cliff-Lorimer, or EELS cross-sections. The split matters because the bridge is where a naming or ordering mistake sends iron's intensity into copper's map, and that is SILENT — every downstream number stays plausible. A wrong k-factor is at least a number a microscopist can sanity-check. They deserve separate tests, and the bridge gets more of them. Three things that would otherwise be quiet: - Negative fitted intensities are clamped at SOURCE. A fit can drive a line slightly negative on noise, and a negative "amount of an element" is not just unphysical for that element — it corrupts the normalisation for every other one. - Missing k-factors default to 1.0 and are REPORTED, because defaulting silently makes an uncalibrated number look calibrated. - The result records that composition is RELATIVE to the fitted elements. Cliff-Lorimer cannot know about an element you did not fit, so leaving one out does not cause a small error — it redistributes that element's share across everything else. End to end on the synthetic EDS SI: composition -> model -> batched fit -> quantify recovers the generator's concentration maps at r > 0.9 per element. "Quantification works" means it recovers what we put in, not that the numbers sum to 1.
Many structures image as PAIRS of closely-spaced atoms. Treating each pair as two independent atoms throws away what the pair actually measures — the separation and orientation that carry polarisation, tilt and local strain. Follows atomap's workflow: estimate the dumbbell vector, pair the atoms, then measure. The vector is estimated from the data rather than typed in, since it depends on the projected structure and the scan rotation. The part worth reading is refine_pairs, which exists because writing the tests exposed a systematic error rather than a bug. Fitting each atom of a dumbbell INDEPENDENTLY biases its centre TOWARDS its partner: the partner's tail is signal a single gaussian can only explain by moving. The separation therefore comes out too small — and uniformly so, which makes it look like a calibration rather than an error. Measured on a synthetic dumbbell of 6.0 px with sigma 2.0, independent fits return 4.21 px, a 30% underestimate. Fitting both atoms in ONE box as one two-gaussian model recovers 6.0, because each atom now explains the other's tail instead of absorbing it. Every pair is still one batched call — the engine does not care that the model gained a component. This is a good advert for the engine: the fix is a different MODEL, not different code. Three tests pin it: that independent fitting shows the bias (and says so if it ever stops), that joint fitting recovers the truth, and that joint beats independent by at least 5x — so a regression in either path shows up as the gap closing from the wrong side. Also pinned, because each is silently plausible if wrong: - The vector estimate must not CANCEL. Each pair contributes +v from one atom and -v from the other, so averaging without folding onto a half-plane gives exactly zero, which looks like a legitimate answer. - Pairing must not claim an atom twice, or there are more pairs than atoms. - The angle is folded to a half-turn, or two identical dumbbells differ by pi depending on which atom was listed first and an angle map shows a boundary that is not there.
…55, #56, #58) The first UI on top of spyde.fitting. Two tabs, matching the Orientation caret's shape: Model builds the model component by component, Run fits it and commits the maps. Layout is horizontal by preference — a component's parameters sit side by side so the caret grows wider rather than taller. #56, the picker: components are added from a "+ Component" POPUP that shows each one's SHAPE, not just its name. The backend samples every offerable component at defaults over the CURRENT signal axis and sends a normalised polyline the renderer draws as a sparkline. Seeding matters — a default Gaussian sits at 0 with sigma 1, which is off-screen on a 200-800 eV axis, so every peak shape would otherwise preview as an identical flat line. #58, the maps: Commit ships one integrated-area map per component through commit_result_tree, which already gives the strain-style toggle (click one, cmd-click to tile). No new display code. The preview is ONE spectrum, not the grid: a model edit has to feel instant, and fitting 65k positions per keystroke would not. fit_run is the only thing that touches the whole scan. The caret rebuilds itself from `fit_state` and never edits a local copy — the backend owns the model, and a caret that mutated its own would drift out of step the moment an edit was rejected, then show a model that is not the one being fitted. TWO REAL BUGS that only running the app could find; both handler-test suites pass either way: 1. A click on a caret control was SILENTLY LOST. Capture-phase listeners showed `mousedown -> the button's span, mouseup -> mdi-area`: the mousedown bubbled to the window, re-rendered the subwindow subtree, and replaced the control between down and up, so React never saw a click. The add simply did not happen, with no error anywhere. Fixed by keeping mousedown inside the caret. (dispatchEvent worked throughout, which is exactly what makes this look like a test problem rather than a UI one.) 2. `emit` was imported as `from ipc import emit`, which binds the original at import time — the test fixture patches `ipc.emit`, so every message assertion would have silently observed nothing. Imported as a module now. Backend: 26 handler tests (including the StrictMode open/close/open contract). UI: fit_wizard.spec.ts drives the real app end to end — toolbar button, palette sparklines, model building, a parameter edit echoed back from the backend, the fit, teardown/reopen, and Commit opening a window with a chip per component. Screenshots at every stage.
#57 — every positioned component gets on-plot handles: a POINT handle driving its centre and height, and a RANGE band driving its width. Dragging writes straight back into the model and redraws, and the numeric fields refresh from the same state, so both directions agree. Ported from anyplotlib's interactive-fitting example. The conversions are the substance. For most components the fitted amplitude is an AREA, not a height, so storing a dragged y straight into `A` would jump the curve by sigma*sqrt(2*pi) the instant it was touched. A width drag likewise holds the peak HEIGHT fixed while changing sigma — otherwise the curve moves away under the cursor, which reads as it fighting the drag. PERSISTENCE: the model and the fit now live on the TREE (`tree.fit_spec`, `tree.fit_result`), not on the caret controller. They are results, and the ownership map in actions/README.md §3 puts results on the tree. A model costs real effort to build and a fit costs minutes; closing the caret to get it out of the way must not discard either. tree.close() still disposes both. THREE MORE BUGS THE RUNNING APP FOUND, all invisible to the handler tests: 1. The live preview NEVER DREW. anyplotlib's add_line takes `label`, not `name`, and the TypeError was swallowed by this method's own except. The previous spec passed end to end while the plot showed nothing — I only caught it by looking at the screenshot. 2. Preview lines ACCUMULATED: remove_line takes an id or a Line1D handle, not a label, so every redraw stacked another line and the legend filled with `fit_preview` entries. The handle is kept now. 3. A new component arrived with amplitude 1 against counts of 1e5 — the curve was a flat line on the axis, the handles sat at y=0, and the fit started five orders of magnitude from its answer. All three read as "the model does nothing". scale_to_data() now sets the linear amplitude from the spectrum itself, generically: evaluate at unit amplitude, scale the peak to a fraction of the data's range. 39 backend tests; the e2e now also asserts the model survives close/reopen, and its screenshot is where the preview curve and the handles are actually checked — nothing else in the suite can see them.
…background Three things, two of them from using it. "FIT SPECTRUM" (requested): fits only the spectrum on screen and writes the result back into the model, so the next nudge starts from a fitted position. Building a model is a loop — place a component, look, adjust — and fitting the whole scan to check one guess is the wrong unit of work: it costs seconds to minutes and answers a question about one pixel. Same engine, same spec, one row of data. It deliberately does NOT set a scan result: one spectrum is not a map, and offering Commit after it would export a component map built from a single pixel repeated everywhere. The other button is now "Run scan", so the two read as what they are. HANDLES MOVE IN PLACE instead of being rebuilt (the "gaussians don't move" report). fit_set_param used to call sync_widgets(), which tears down and recreates every widget — on every keystroke, several times a second. That makes the handles flicker and can pull one out from under the cursor mid-reach. update_widgets() now moves them via the widget's own set(), and rebuilding is reserved for a changed component LIST. A drag skips the handle being dragged, because writing a position back to the widget under the user's finger fights the drag. I could not reproduce a parameter edit failing to move the CURVE — in the app, typing a new centre moves both curve and handles — so the churn above is my best explanation for what was seen. If it persists, which parameter and how many components would narrow it. #65: fit_from_composition — type the elements, get a populated model, then drag the lines. EELS models are tabulated on the way in (#63), or the model is correct but falls back to hyperspy's per-pixel fitting, which is the difference between seconds and minutes on a real scan. #64: background_action — drag a window over a background-only region; the model is fitted to THAT SPAN ONLY and extrapolated across the whole axis, which is the entire point of a background model (it says what the background would have been under the peaks, where it cannot be measured). Fitted per navigation position, because a spectrum image has a different background per pixel and subtracting one pixel's fit everywhere would imprint a thickness map on the result. Seeded from the data INSIDE the window: a pre-edge window sits far below the peak, so scaling to the global max starts it an order of magnitude high. 46 backend tests; the e2e passes end to end.
…erly The drag did nothing to the model, and the reason is embarrassing: anyplotlib hands the dragged widget back on `event.source`, so the position is `event.source.x`. I read `event.x`, which is None — so every drag silently updated nothing while the handle still moved on screen, because the widget draws itself. That is exactly why it looked like "the curves don't move". Rewritten to follow the interactive-fitting example rather than approximate it: - Lines are updated with Line1D.set_data IN PLACE. Removing and re-adding a line per drag frame is heavy and does not repaint mid-drag. - ONE LINE PER COMPONENT plus a dashed sum, instead of a single summed curve. With several overlapping peaks, a sum tells you the total is wrong but not which component to grab. - The `_syncing` guard, because moving the handles in response to a drag re-enters the handler and the widget and model chase each other. - The range widget uses style="fwhm" at y=height/2, and the point widget drops its crosshair — the example's geometry, which reads as a peak handle and a width band rather than two unrelated markers. Verified with a REAL pointer drag in the app, not a synthesised event: grabbing the handle and dragging left moves centre 500 -> 296 eV, and the curve, both handles and the caret's fields all follow. A screenshot confirms it. "Fit spectrum" moved to the Model tab (requested). That is where the loop is — add, look, nudge, fit this one, look again. Run scan stays on its own tab as the separate expensive thing you do once the model is right. The drag test now pins the `event.source` contract directly, so this specific mistake cannot come back silently.
…d spectrum Reported as "Fit spectrum doesn't work at all", and the way it failed is worse than not working: it said CONVERGED and drew a model at about half the height of the data. current_spectrum() reconstructed the on-screen spectrum from signal.data plus a navigator index. The index was not where it expected, so it fell through to the mean over navigation — and a fit against the mean converges perfectly well. The status was true; the answer was about a spectrum nobody was looking at. plot.current_data is the authority: it IS the array the plot is displaying, already resolved through whatever navigator, region-integration or derived-view path produced it. Read that. The navigation mean survives only as a stand-in for the PREVIEW before the first frame lands, and it logs when it is used. Why nothing caught this: the handler tests build their own signal and never have a navigator, so the fallback WAS the displayed spectrum for them. It needed the real app, a real navigator position, and an assertion about the fitted VALUES rather than about a status string. fit_two_gaussians.spec.ts is that test, on the dataset this is meant for (tutorial_spectroscopy = hyperspy's two_gaussians). It asserts the model's peak height, computed from the fitted parameters — a HyperSpy Gaussian's A is an AREA, so the height is A/(sigma*sqrt(2*pi)) summed over components. Fitting the navigation mean gave ~460 against data peaking at ~1050; the fix gives ~1050. The threshold separates those two outcomes without pinning an exact fitted value, and a second assertion stops a degenerate fit that zeroes one component from passing on the other's back. Four backend tests pin the same contract directly, including that a painted spectrum always beats the mean and that a stale/odd-shaped current_data (a dask Future mid-transition) is ignored rather than fitted.
… icon PER-POSITION MEMORY. Each navigator position now remembers its own fitted parameters, on the tree beside the model. Scrubbing back to a pixel shows what was found THERE rather than whatever the last pixel left behind. The store is sparse — a dict keyed by position, not a (P, n) array — because it fills in as the user explores, and a full allocation would claim every position had been fitted. It is cleared whenever the component list changes: the vectors are POSITIONAL, so after an add or remove a stored sigma would arrive as an amplitude and nothing would look obviously wrong. ADAPTIVE FITTING (an "Adaptive" toggle). Moving the navigator recalls a stored fit if there is one, and otherwise — with adaptive on — fits the new spectrum seeded from the model as it stands, which is a neighbouring position's answer and therefore a good starting point (the same reason seeded propagation works for a whole scan, #54). Recall wins over refit: same computation, already done. Driven from pointer_UP on the navigator, not pointer_move, or it would queue fits faster than they complete. This also fixes the thing that made the earlier "Fit spectrum" bug possible: current_indices lives on the navigation SELECTOR, not on the Plot. Reading it off the Plot always returned None. There is now a real accessor and a test that exercises it against a REAL session rather than only a fake. SMOOTHER DRAGS. pointer_move and pointer_up are wired to separate handlers because they should do different amounts of work. A move now redraws the curve and nothing else; moving the partner handle and re-sending the whole model both cross the IPC boundary, and doing either at pointer rate is what made the curve lag the cursor. Both happen once, on release. Remove Background gets its OWN icon — it was borrowing Fit's, so the two buttons were indistinguishable. Drawn for 20 px rather than for the artboard: only about three strokes survive at button size, so it spends them on a straight baseline and a down-arrow, a different silhouette from Fit's peak. The first attempt was a dashed background curve — handsome at 150 px, mush at 20, which was the whole complaint. Checked by rendering at 20/28/150 px on the toolbar's own background before committing. 66 backend tests, both e2e specs green.
THE DRAG REGRESSION WAS MINE. Splitting the preview into one line per component went from ONE line update per pointer frame to N+1, and Line1D.set_data is not cheap bookkeeping: it recomputes the axis range and pushes the WHOLE plot state to the renderer. Two gaussians therefore meant three full state pushes per frame where there had been one. That is why it got worse after the last change, and the "smoothness" work in that commit did not touch it. refresh_lines now writes every line's data and pushes ONCE, so it is one push per frame regardless of component count — fewer than before the per-component rewrite, not merely back to parity. It reaches into anyplotlib's line entries because there is no public batched update; the public per-line path stays as the fallback, so an upstream change costs speed rather than correctness. FIT COVERAGE, so the store is visible: the caret shows "N/M fitted" with a filled marker when the CURRENT position has an answer. Without it there is no way to tell a position you skipped from one you fitted, and "why didn't adaptive fill this in" has no answer on screen. WHAT I COULD NOT MEASURE, stated plainly. I tried to quantify drag smoothness through Playwright and produced a number that was mostly harness: a bare page.mouse.move costs ~16 ms here, so 30 synthetic moves take ~600 ms with NO caret open at all, against ~850 ms with one. My earlier "~28 ms per move" was therefore ~70% Playwright. On that bad number I added a redraw coalescer; it showed no improvement, and I have removed it rather than ship an unverified optimisation that also drops frames. The batched push stays because it is a strict reduction in work that is true independent of any timing. Measured for real, on the side: a single-spectrum fit is 33 ms (14 LM iterations). float32 is SLOWER than float64 here (203 vs 109 ms) — worth recording so nobody tries it again. The 109 ms in that comparison was dominated by BUILDING the spec, which goes through sympy in hyperspy's Expression components; reusing a spec is 33 ms.
Two things the caret got wrong, both reported from the app. **Specs were rebuilt constantly.** Constructing a hyperspy component costs 20-110 ms — Expression components lambdify their formula through sympy on every instantiation, with no caching upstream — and the picker, "add component" and the background wizard each built their own. The palette therefore cost 6.2 s on first open and ~130 ms on every open after, which is what "the fitting is slow" actually was: the caret opening, not the fit. One prototype per kind now, one ComponentSpec read off it, deep-copied per use (`new_component_spec`), and the sampled palette cached per axis. With imports already paid, as the app pays them: first open 681 -> 3 ms on a new dataset, 0.04 ms reopening the same one. The remaining first-open cost is sampled off the main thread, so the caret appears immediately and the sparklines fill in. **Handles came off the component.** Backgrounds had none at all — a PowerLaw or an Offset was editable only through the caret's number boxes. They now carry ANCHOR points, and dragging one solves the component through it in closed form, so the curve passes exactly under the cursor (the other anchors are held, so it does not pivot out from under them). And the partner handle is moved every pointer frame instead of only on release: a widget `set` is a targeted push of a few fields, unlike `fit_state`, which stays on release. Two real bugs surfaced while verifying the anchors in the app, neither visible to the handler tests: * A PowerLaw's `origin` was seeded to the axis midpoint, because the rule that centres a gaussian also matched the name. The background was then identically zero over the left half of the spectrum, with its anchors outside its own domain. * Scaling a background by its PEAK matches a power law's singular left edge, leaving ~3e-5 across the spectrum; by its median LEVEL it puts 3.7e10 at that same edge. Neither is a scaling problem — a background needs its shape solved from the data, which `seed_background` now does through the same closed form, sampling a low percentile over a wide window so it lands on the low envelope rather than on whatever peak sits under an anchor. Verified in the app: `fit_handles.spec.ts` reads each handle out of the panel state and requires it to sit on its component's own line, drives a live pointer_move and asserts the width band tracked it, and drags a background anchor. Adds a `_spyde_test_panel_json` hook so a spec can read the lines as well as the widgets — a screenshot cannot tell "on the curve" from "a few pixels off it".
`exspy`, `kikuchipy` and `atomap` (plus `ase`, which atomap pulls in) are declared as optional extras in pyproject but were never resolved into the lockfile, so `uv sync --extra ebsd` had to re-resolve them every time.
…ehind Three things wrong in `fit_run`'s completion, all of the same shape as the handle drift: the result landed in the model but nothing else was told. * The store was never filled, so the caret said "0/1024 fitted" immediately after fitting all 1024, and navigating afterwards refit positions it had just solved. `record_run` now records every CONVERGED position — a failed fit is not an answer to remember, and leaving it out lets a later adaptive pass retry from a better neighbouring seed. * The handles stayed put while the curves jumped to the fitted values. * The post-run preview read `current_indices` off the PLOT, where it does not live, so `idx` was always None and it silently showed position 0's parameters instead of the displayed one. Same mistake that made "Fit spectrum" fit the navigation mean. `position_of` is the exact inverse of the `ravel_multi_index(reversed(idx))` the run path uses; it is tested on a NON-square scan, because a transposed key is invisible on a square one. `record_run` is a method rather than inline in `_done` so it is reachable without the worker marshal, which does not turn under pytest — the existing run test sidesteps the same way. Verified in the app: the spec now runs the scan and asserts the coverage readout moved off zero and that the handle positions changed (347/1024 on the tutorial spectroscopy dataset, handles on the fitted curves).
…t separate Every component is seeded mid-axis, so a second gaussian arrived with the same centre AND the same sigma as the first: two IDENTICAL Jacobian columns (measured corr(centre1, centre2) = 1.000000, cond(J) = 1e16). A perfectly degenerate pair can never separate — the solver moves both the same way forever — so it grows one and shrinks the other. On two genuinely separated peaks it was worse than cosmetic: the pair ran away together to sigma = 170 on a 100-wide axis, chisq 6.3e7, with neither peak found. Two changes, and the measurement says both are needed: * `spread_repeats` places repeats of a kind symmetrically about the seed, spaced by ONE of their own widths. Symmetric so a genuinely co-centred pair still starts co-centred; one width because wider is worse, not better (at two widths the co-centred case fails outright, chisq 6.3e5 against 1.3e-21). * `clamp_to_axis` bounds a peak's position to the axis and its width to the axis span. A component centred off the data or wider than all of it is not modelling anything the user can see, and those are exactly the runaway directions. Backgrounds are exempt: a PowerLaw's `origin` is a reference point deliberately placed outside the axis. Neither alone is enough. Bounds without the spread only pin the degenerate pair at the cap (chisq 6.3e7, unchanged). The spread without bounds still diverges on a noisy near-overlapping pair (chisq 1.1e8 against 127). Measured over six synthetic arrangements plus two positions of hyperspy's real `two_gaussians`: three catastrophic failures fixed, and the two real positions reproduce their previous fits EXACTLY (chisq 5.245e5 and 1.795e5, unchanged) — which is the point of spacing by one width rather than relocating. No seed is universally right on a non-convex problem; a wider spread trades one failure for another. That is what the drag handles are for.
"Once you try to fit a spectrum then you can't move either component."
`spyde:figure_event` carries `{figId, event}` and NO window_id — SpyDEContext's
re-broadcast never had one. The Fit caret's navigator listener guarded with
`detail.window_id === windowId`, which compared `undefined` and so never fired.
Every pointer_up on the caret's OWN plot therefore sent `fit_navigated`,
including the ones from its own drag handles.
Before a fit that was invisible: `fit_navigated` finds nothing stored and just
redraws. After one, the position IS stored, so it RECALLED the fitted values
and overwrote the drag the instant it landed — the handle snapped straight
back and the component could not be moved at all. Which also made the fit
un-improvable, since nudging a component and refitting is the entire workflow.
Filtered by figId instead, against this window's own figures.
Also reorders `_on_widget_drag`: handles first, then lines. `refresh_lines`
ends in a FULL panel push that serialises every widget's geometry, while
`update_widgets` issues TARGETED per-widget pushes that travel on the event
channel. Lines-first meant the frame's full push carried the OLD partner
position and the new one went out targeted-only. Handles-first is one push
per frame instead of one plus one per moved handle, and it is what lets a test
see the partner track at all.
Verified in the app by fit_drag_after_fit.spec.ts, which drags a component
before a fit (the control), fits, drags again, and requires the model to both
move AND stay moved.
Two gaussians are exchangeable — nothing in the model says which is the broad one — so fitted position by position they land in whichever slot each fit happens to pick. Measured on hyperspy's two_gaussians: the first component was the broad peak at only 43% of positions, its sigma ranging 2.2 to 25.7 across the scan. Which is what "it's literally suppressing one component to 0" was. Scrubbing the navigator made the SAME named component jump between the broad peak and the narrow one, so its amplitude dropped by an order of magnitude at every swap (10259 -> 1509 between two positions, verified in the app). And a committed component map would have been a checkerboard of two different peaks, not a map of anything. The fit itself was never the problem: 97% of positions already reach the noise floor, median chisq 0.991x what the TRUE generating model achieves on the same noise. Only the labelling was wrong. `relabel_scan` decides the ordering ONCE for the whole scan and applies it everywhere: find the parameter that genuinely separates the components (here sigma, 2.5 against 25.5) and sort by it. Wired into both the cold batched path and `fit_seeded` — the latter also relabels its COARSE grid before propagating, or each coarse cell hands out its own arrangement and undoes the seeding. Two things I tried first and rejected on the measurement: * Serpentine neighbour-matching — walk the grid in boustrophedon order and keep the arrangement closest to the position you just came from. It drifts: matching is chained, so one bad match (a position where the components are genuinely similar) flips the convention for everything after it. 425 of 1024 positions were permuted correctly and agreement went DOWN, 43% to 28%. * Scoring separability by how far apart the sorted slots' means are. Sorting ALWAYS separates: two components drawn from the same distribution still give a "smaller" slot and a "larger" one. The score now measures DISJOINTNESS (5th percentile of the upper slot minus the 95th of the lower), and refuses to reorder when they overlap — an invented identity is worse than an inconsistent one, because the map then looks meaningful when it is not. Relabelling changes no residual: median chisq is identical before and after (216748), and a test asserts the summed model curve is unchanged to 1e-12. Verified in the app by fit_navigate.spec.ts: fit the scan, move the navigator between two positions with different answers, and require both that the model follows the navigator AND that each named component is still the same peak.
…per fixture The dataset fixtures loaded a signal and then slept a flat 0.8 seconds to "let selector debounce timers fire". They are used ~930 times across the suite — roughly 12 minutes of the run spent sleeping. The timer being waited on is armed for max(0.12, live_delay + 0.1) (base_selector._arm_settle), so 0.8s was about 6x the margin needed. Wait for the actual state instead: nothing pending on the nav dispatcher, no selector holding a live settle timer, and every signal plot painted. The floor matters as much as the condition. At the instant _add_signal returns, the timer may not be armed yet and no plot has painted, so a pure condition poll would return immediately and hand the test a session that had not settled at all — never return before one full settle period has passed. It falls through after a 3s timeout rather than hanging, so a fixture that genuinely never settles still fails loudly in its test. Measured: 213 ms for the 4D-STEM fixture and 157 ms for the 2D one, the 0.12s timer now being the floor. Same 25 files that took 78.6s take 59.6s, with the same 340 tests passing. No xdist — 4x box load on a suite this thread-sensitive traded flakes for speed, and three races surfaced this week already.
…lector test_run_update_fires_hook_only_on_change failed on ubuntu-py3.10 with `assert 2 == 1`: it installs a NAV_CHANGE_HOOKS counter immediately after _add_signal, while the settle timer that load armed is still pending. The timer then fires on the dispatcher thread, commits a position and trips the hook — so the test's own single _run_update() looks like two. This file is the empirically hot one: the same race produced a flake under xdist, another on macos-py3.11 (test_no_selector_falls_back_to_frame_zero fighting a selector that kept re-arming), and now this. Every test in it drives the nav cursor, so settle after all seven loads rather than only the one that happened to fail. Costs ~1s across the file. The fixtures already do this; these tests call _add_signal themselves and so never got it.
Both read the model curve straight after a navigator move. goTo ends in a flat waitForTimeout (2.5s / 1.2s) and repainting the overlay is a backend round trip, so on a loaded runner the read scores the PREVIOUS position's curve. fit_navigate saw it directly: peak 404.84 / argmax 486 at both stops, so "the overlaid model curve did not change between positions" fired on two reads of the same stale curve. It now polls until the curve changes, which is the property the test is named for — and is faster than the fixed sleep whenever the repaint lands early. fit_quality hid it behind a tolerance. Its sweep logged worst misfits of 47%, 76%, 83% and 191% across CI runs, while re-measuring that very position moments later gave 2-12% — so "worst" was really "where the read was most stale", and comparing that noise to a dedicated refit is what tripped `worstBefore < worstAfter * 1.5`. Loosening the ratio would have buried a real measurement bug. Instead every misfit now samples until two consecutive reads agree; unlike waiting for a CHANGE it assumes nothing about neighbouring positions differing. The measurement is what improved, not the threshold: the sweep's worst goes 47-191% -> 3.9%, median 3.6-3.8% -> 2.9%, and the worst position now reads 3.9% -> 3.9% after a dedicated refit — i.e. the whole-scan fit did leave it at its own answer, which is exactly what the assertion claims.
…rted test_thumbnail_reflects_cursor failed on windows-py3.11 with "cursor never settled at (1, 2)" — _settle_cursor burning its entire 30s budget. _set_indices assigned current_indices only. These tests never move a real widget, and _run_update does `self.current_indices = self.get_selected_indices()` on the dispatcher thread, so the committed value was reverted to the widget's own unmoved position every time a settle timer fired. _settle_cursor re-applied, got reverted, re-applied — a coin flip every 50ms that it can lose for 30s straight under load. Pin get_selected_indices as well as current_indices: the state is then self-consistent and nothing has anything to revert. That is the technique the NAV_CHANGE_HOOKS test in this same file already uses, and _run_update is safe with it (array_equal short-circuits, and the settle's force=True path just re-commits the same value). 36 passed three times running; 83 passed across the neighbouring selector suites.
fit_quality still flaked once after the settled-read fix. The retry's own log gave it away: [2,2] — the first sweep stop after the tab switch — still read 191.6% while re-measuring that position later gave 3.7%. "Two consecutive reads agree" can agree on the OLD value when the move has not been applied yet, so the stability check settled on the stale curve. Wait for the SPECTRUM to become this position's before judging the model against it, and require three consecutive agreements rather than two so one repeat cannot land in a lull while the overlay catches up. The sweep is now deterministic: three runs in a row give median 2.9%, worst 3.9% at [30,10], worst position 3.9% -> 3.9% after a dedicated refit. Before, the worst position and value moved every run — 191.6% at [2,2], 82.6% at [30,2], 76.3% at [24,16], 46.9% at [30,10]. That variance WAS the staleness. Also faster: goTo waits for the real change instead of a flat 1.2s, so the spec runs in 1.0-1.1m rather than 1.3m.
Third pass at this one, each driven by what CI actually logged. A flat 1.2s: [2,2] read 191.6% where re-measuring gave 3.7%. Sampling until two reads agree: a stale overlay is perfectly stable, so stability cannot tell "finished updating" from "has not started" — [2,2] still read 191.6%. Waiting for the DATA to change: the spectrum arrives first and the model follows as a separate push, so on a loaded runner [16,2] still scored 28.3% mid-sweep and the run went median 24.8%. The MODEL changing is the signal that the overlay belongs to this position — the same thing that made fit_navigate stable. goTo now waits for both curves, and falls through on timeout rather than hanging, with settledMisfit still the final guard. Two runs back to back, identical: [2,2] 3.7%, median 2.9%, worst 3.9% at [30,10], worst position 3.9% -> 3.9%. And faster again — 52.5s against 1.3m for the original flat-sleep version, because it waits for the condition instead of guessing at it.
fit_quality came back clean this run; fit_navigate took its place, and the message was mine: "the model stayed IDENTICAL at two different navigator positions", the 60s poll expiring because the caret never showed the new position. That is a moved-too-early failure, not a stale-fit one. goTo posted a pointer_up and slept a flat 2.5s; when that was not enough the navigator had not landed, so both reads described the same position and no amount of polling afterwards could separate them. Wait for the SPECTRUM to change — the same gate that fixed fit_quality's sweep. Two runs back to back give identical parameters at both stops, and it is quicker too: 28.8s / 31.6s against 38.9s for the flat-sleep version. Note the sweep's per-position log in fit_quality can still print a stale figure on a loaded runner. It no longer decides anything: the worst position is re-measured with settledMisfit before the assertion, which is why that run passed with worst position 3.8% -> 3.8% despite the sweep printing 82.6%.
CSSFrancis
added a commit
that referenced
this pull request
Jul 29, 2026
PR #90 targets feat/0.3.0 and had exactly ONE check — the docs Build. The whole test workflow triggers on `pull_request: branches: [main]`, so a PR into the branch that collects an entire release ran no Python matrix, no extras and no e2e. It reported MERGEABLE / CLEAN because nothing failed, because nothing ran. That is not specific to #90: #82 through #88 each landed a 0.3.0 wave that way, and the matrix first saw any of them when 0.3.0 itself went at main (#89) — which is where the timeouts and the four latent races surfaced all at once. `feat/[0-9]*` matches the release integration branches (feat/0.3.0) and not ordinary topic branches, so this costs a matrix per PR into a release line and nothing elsewhere.
This was referenced Jul 29, 2026
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrates the five 0.3.0 waves plus the Examples overhaul. Closes the code side of #48.
96 files, ~20k insertions. Each wave landed on this branch through its own PR (#82, #84, #86, #85, #83) rather than straight to
main, so the history stays reviewable wave by wave.What lands
requires_packagetoolbar gate, bundled synthetic datasetsspyde/fitting/spyde/fitting/eelsspyde/ebsd/ebsdspyde/atoms/atomsspyde/data/The fitting engine replaces
multifit's one-pixel-at-a-time loop (95 spectra/s, 11.6 min for a 256² SI) with a batched GPU Levenberg–Marquardt, keeping SAMFire's insight — a neighbour's result is the best starting point — as the seed source rather than the scheduler.EBSD is the 4D-STEM orientation wizard stage for stage, because it is the same job on a different signal: pick a crystal, build a library, check the match under the crosshair, run the field. The difference is what the Refine stage draws — a template match is a set of spots, an EBSD match is a set of bands, and a band centre is a straight line on a flat detector. Downstream nothing is new: the result becomes a
SpyDEOrientationMap, so the IPF colouring, 3-D explorer, point selector and direction toggle all work untouched. 1.8° misorientation from a 5° dictionary, 0.09° after refinement, 1.5 ms per live match.The Examples menu now comes from em-database — one curated, checksummed, cited collection instead of each library's own fixtures. Submenu per technique, each row carrying shape, size and an on-disk marker, with a themed hover card. Dummy Data is its own submenu; "Show Example Data Directory" added.
Two bugs worth calling out
Both were invisible to a green test suite, and both are the kind that produce plausible wrong answers:
map_blocksfor the per-pattern work,map_overlapwith a halo for the ADP map). Measured 12.5% of the scan resident at peak across 8 worker threads. A guard test now fails if anything computes a dask array the shape of the full dataset — it caught two further bugs while being written.Verification
main); 25 new test filesKnown issues going in
fit_wizard.spec.tsfails: committing makes two "Fit components" windows. Pre-existing onfeat/0.3.0-fit-wizard— verified against that branch alone, not caused by the integration — but it ships with this merge.EBSDDetector) is not started: no kikuchipy.h5/.angreaders, so real vendor files cannot be loaded yet. The wizard uses its own band geometry and a fractional projection centre.python_version >= '3.12'marker: it declaresrequires-python >=3.12while SpyDE supports >=3.10, and depending on it unconditionally made the project unresolvable there — the backend refused to start at all. Below 3.12 the Examples menu falls back to Dummy Data. Relaxing the floor upstream would let the marker go.