Many-grain density support (dense/streaky frames)#7
Merged
Conversation
…eiling Enables indexing of dense, streaky, many-grain frames (hundreds of overlapping grains per image) that the pipeline previously mishandled. Validated against multi-grain synthetic ground truth during the July 2026 34-ID-E beam time. Three coupled changes: 1. laue_index/preprocess.py — robust spot spacing. The auto blur sigma was sized from the ABSOLUTE minimum neighbour distance, which collapses on dense frames: fragments of a single streak sit a few px apart, shrinking sigma to ~1 px and leaving the 0.4-deg orientation grid unable to reach spots up to ~18 px from a grid seed (recovery dropped to ~15% on a 150-grain synthetic frame). Fragments of the same streak SHOULD merge — they are one grain's reflection — so size the blur from the 10th percentile of neighbour distances instead of the minimum. This changes the default preprocessed output; the golden fixtures are updated to match. 2. GaussSigmaMax parameter (laue_stream_utils.py + preprocess.py) — optional cap on the auto sigma. On dense frames a large blur lights so much of the detector that chance matches flood the coarse gate and bury true grains; iterative-peel drivers start tight and widen per pass. Default 0.0 = no cap. 3. MAX_MATCHES 100k -> 4M (LaueMatchingGPU.cu, LaueMatchingGPUStream.cu). The kernel appends matches in arrival order, so overflow discards candidates ARBITRARILY — true grains included — before the top-score merge can rank them. 100k overflowed on many-grain frames. 4M x 12 B x MAX_STREAMS is < 200 MB on device and pinned host. Also in laue_stream_utils.py: read_solutions tolerates torn/truncated rows (daemon terminated mid-write on a dense frame), dropping malformed rows with a warning rather than failing the whole parse. Char suite and the streaming regression tests pass with these changes.
Add calc_recip_array / orientations_to_recips and accept a list of orientations, so a single frame can superpose many grains' patterns. This is the synthetic ground truth used to validate the many-grain density work — known orientations, known count, to measure recovery.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables indexing of dense, streaky, many-grain frames — hundreds of overlapping grains
per image, no depth resolution — which the pipeline previously mishandled. This is the
core capability behind the July 2026 34-ID-E Ti-6Al-4V campaign (13 scans, ~150k frames).
Validated against multi-grain synthetic ground truth.
1. Robust blur sizing (
laue_index/preprocess.py)The auto matching-blur sigma was sized from the absolute minimum neighbour distance,
which collapses on dense frames: fragments of a single streak sit a few px apart, shrinking
sigma to ~1 px and leaving the 0.4° orientation grid unable to reach spots up to ~18 px from
a grid seed. Recovery dropped to ~15% on a 150-grain synthetic frame. Fragments of one
streak should merge — they are one grain's reflection — so size from the 10th percentile
of neighbour distances. This changes the default preprocessed output; golden fixtures updated
to match.
2.
GaussSigmaMaxparameter (laue_stream_utils.py+preprocess.py)Optional cap on the auto sigma. On dense frames a large blur lights so much of the detector
that chance matches flood the coarse gate and bury true grains; iterative-peel drivers start
tight and widen per pass. Default
0.0= no cap, so existing behaviour is unchanged unlessthe param is set.
3.
MAX_MATCHES100k → 4M (LaueMatchingGPU.cu,LaueMatchingGPUStream.cu)The kernel appends matches in arrival order, so overflow discards candidates arbitrarily —
true grains included — before the top-score merge can rank them. 100k overflowed on
many-grain frames. 4M × 12 B × MAX_STREAMS is < 200 MB on device and pinned host.
4. Torn-row tolerance (
read_solutions)Tolerates truncated rows (daemon terminated mid-write on a dense frame), dropping malformed
rows with a warning rather than failing the whole parse.
Plus
GenerateSimulation.pygains multi-grain synthetic frames (orientations_to_recips) — theknown-orientation, known-count ground truth used to measure recovery above.
Tests
Char suite and the streaming regression tests (#5) pass. The
.cuchange needs the GPUbuild (
build-gpu) to verify — the CPU build does not exercise it.Deliberately not included: the
laue_torchREADME bullet and itstorch/midas-hklsrequirements, held until the associated publications go through.