Dev Image Culling Signal Analysis

Culling: Signal Quality and Genre Analysis

Reference material behind the culling performance and correctness pass. The what changed and what is outstanding view lives in Dev-Image-Culling-Implementation-Plan.md; this page is the reasoning underneath it — which signals cap ranking accuracy today, and what "best frame" actually means across photographic genres.

Line references point at the state of the code when this was written; treat them as signposts rather than exact addresses.

Where the quality ceiling is

Sharpness is global, and that is the wrong question — FIXED (region), OUTSTANDING (eye crop)

metrics.rs:196-211 takes Laplacian variance over the whole 512 px frame.

• Shallow DOF is punished. An f/1.4 portrait or a 400 mm wildlife frame has a razor-sharp subject and smooth background; global variance is lower than a mediocre f/8 shot. The sports preset's only answer is lowering a threshold (culling_config.rs:218).
• Content, not focus, dominates. Busy foliage beats a clean minimalist frame regardless of whether anything is in focus.
• The decisive detail is invisible at 512 px. Eye-sharp vs. nose-sharp — the most common portrait cull decision there is — is not resolvable at that scale.

Fix: compute sharpness on regions, at native resolution.

1. Tile the working image; keep max-over-tiles ("is anything in focus?") and the location of the sharp region alongside the global mean. The pair distinguishes "soft frame" from "correctly shallow".
2. Where faces exist, measure sharpness on a native-resolution eye-region crop, not the 2048 px working image. Face boxes are already available from YuNet.
3. Add a motion-blur direction estimate (anisotropy of the gradient orientation histogram) so directional camera shake is separable from defocus and from intentional panning.

Status. Points 1 and 3 shipped. Tiling accumulates per-tile Laplacian variance in the existing pass, giving cull_sharpness_peak, cull_focus_concentration and cull_sharp_region_{x,y}; ranking blends peak into effective sharpness at sharpness_peak_weight (0.70 sports, 0.55 portrait, 0.25 street). Point 3 uses the structure tensor rather than an orientation histogram — the coherence √((Jxx−Jyy)² + 4Jxy²)/(Jxx+Jyy) is the same quantity for two axes and costs three accumulators. It is not scored, only used to label a soft frame motion_blur instead of blurred, because directional content (architecture, horizons, rain) raises it identically. Point 2 — the native-resolution eye crop — is still outstanding.

Eye-openness: the aggregation is inverted for group shots — FIXED

face_aggregate.rs:68-135 rolls multiple faces up as eye_openness = max, blink_penalty = min. In a ten-person group shot where nine people blink and one has eyes open, the photo scores maximum eye-openness and minimum blink penalty. That is exactly backwards, and group shots are the highest-value case in event work.

Fix: aggregate eyes-open as a face-size-weighted min (worst prominent face decides), keep max only for prominence. Currently nothing is weighted by face size at all.

Blink detection itself is a ≤8 px gradient — OUTSTANDING

face_quality.rs:56-104: mean absolute vertical gradient in a patch of radius clamp(min(w,h) × 0.08, 2, 8) around YuNet's two eye keypoints.

• The EYE_PATCH_RADIUS_MAX: 8 clamp means the window stops scaling with face size — on a headshot where the face is 1200 px, the analysis window is a 17×17 px sliver.
• YuNet runs on a 640×640 letterbox (yunet.rs:31), so keypoints carry several pixels of error when scaled back, and small faces in group shots go undetected. The letterbox size was kept deliberately when the detector was swapped, so this limit is unchanged.
• Eyelashes, eyeliner, glasses frames and dark eyebrows all produce strong vertical gradients on a closed eye; squinting in sunlight reads as closed.
• blink_penalty is literally 1.0 - eye_openness (faces.rs:279) — it carries no independent information, yet reject_blink_penalty_threshold is tuned as if it did.

Fix: a small eye-state ONNX classifier on a native-resolution eye crop (24×24 or 32×32, 1–3 MB, sub-millisecond). YuNet's single point per eye cannot support an eye-aspect-ratio measure, so this genuinely needs a model. Remove blink_penalty as a separate field or derive it from the classifier's confidence.

occlusion does not measure occlusion — FIXED

face_quality.rs:107: 1 − (0.55·det_score + 0.20·center_proximity + 0.25·eye_openness). It never looks at the image beyond signals it already has, and center_proximity is a composition heuristic — a face near the frame edge is not occluded. The field is ~collinear with det_score, so reject_occlusion_threshold is effectively a second detector-confidence gate. Either replace it with a real measurement or delete it and stop pretending.

Status. Replaced, not deleted. occlusion_from_crop combines two measurements that fail on different inputs: the landmark-fit residual against the ArcFace template (a similarity transform absorbs translation, rotation and scale, so only the face's shape changing moves it — which is what a covered feature does to the detector's estimate) and mirror asymmetry on the aligned 112 px crop, normalised by the crop's own luminance spread so it is not just re-measuring contrast. Neither det_score nor center_proximity is an input any more.

The template is still ARCFACE_DST and the crop is still 112 px even though recognition moved from ArcFace to FaceNet: both are kept byte-for-byte (umeyama.rs) precisely so the detector/recognizer swap left every culling score untouched. FaceNet gets its own wider framing instead (FACENET_ZOOM_OUT), which is not what this measure runs on.

Two honest caveats. Strong out-of-plane rotation raises the geometry term, and hard side lighting raises the symmetry term; the field measures "obstructed or turned away", which is what the plan calls "occlusion / poor facial visibility" but is not pure occlusion. And a FACE_TABLE row written before this existed carries no substitute — face_aggregate now scores a missing value 0 rather than reconstructing it, because a reject nominated from a missing field is worse than a reject missed.

"Aesthetic" is contrast + colorfulness — FIXED

AESTHETIC_CONTRAST_WEIGHT 0.45 / COLORFULNESS 0.35 / EXPOSURE 0.20 (metrics.rs:125). This rewards punchy saturated frames and floors muted fine-art portraiture, fog, snow, and every desaturated editorial look.

Fix — and this is nearly free: the SigLIP2 embedding is already computed and stored.

• A LAION-aesthetic-style linear head over that embedding is a dot product.
• CLIP-IQA style antonym prompt pairs ("a sharp photo" vs "a blurry photo", "a well-composed photograph" vs "a badly composed photograph") need only embed_text, which already batches (siglip.rs:197), computed once and cached.

Both are ~zero marginal cost per photo and replace a heuristic that is actively wrong. They apply only when embeddings exist, so the fast cull path degrades to technical signals.

Status. The CLIP-IQA route shipped (lrg-ml/src/clip_iqa.rs); the LAION head did not, because it would mean shipping and versioning another weights file for a signal the prompt pairs already provide. Five pairs, embedded once per server lifetime and cached in AppState, then a two-way softmax over cosine similarity per pair. The logit scale of 100 is load bearing: raw cosines for a prompt pair differ by a few hundredths, which a plain softmax flattens to ~0.5 for every image. Blended against the heuristic at aesthetic_iqa_weight (0.8) rather than replacing it outright, and skipped silently when there is no embedding or the text tower will not load.

Note the prompts deliberately do not ask about sharpness or exposure, though the passage above suggests "a sharp photo" vs "a blurry photo". Those are measured directly from pixels far more reliably than a 1152-dim embedding can judge them, and asking twice only adds a noisy second opinion to a question already answered.

Exposure and noise are absolute, and they fight each other — FIXED (exposure), OUTSTANDING (noise)

EXPOSURE_TARGET: 0.5, EXPOSURE_TOLERANCE: 0.35 (metrics.rs:117) penalise low-key portraits, concert and stage work, night, silhouettes and high-key fashion for being correctly exposed for their genre. Meanwhile the noise estimate (metrics.rs:243-266) is a 3×3 box-blur residual — a high-pass measure — and fine detail is high-pass energy, so a sharper frame scores as noisier and 0.5·sharpness partly cancels 0.15·(1−noise).

Fix: score exposure and noise relative to the group (z-score within group, or rank), not against absolutes. Keep absolute clipping fractions, which are genuinely absolute. For noise, gate the residual on low-gradient regions only so texture stops registering as grain.

Correction after implementing it. Exposure was normalised against the group and benefits. Noise was not, and must not be. Relative scoring assumes the metric is at least pointing the right way; this one is not. Measured on two frames of one scene differing only in focus, the raw estimator scores the sharp frame 0.082 and the blurred one 0.012 — so rescaling to the group's range stretched the gap to 0.535, amplifying the error eightfold. The stated motivation does not hold either: a group shot entirely at high ISO is penalised equally, and a constant offset cannot change a within-group ranking. Only the second half of the fix above — gating the residual on low-gradient regions so texture stops registering as grain — actually addresses this, and it remains outstanding.

Ranking is absolute where it should be relative — FIXED

rank_group_records (grouping.rs:223) weight-sums absolute 0–1 metrics. Reason codes use within-group deltas, but the score does not. A group shot entirely in dim light gets uniformly crushed technical scores and the ordering falls to whichever metric happens to retain dynamic range. Normalise each metric within its group before weighting.

Intentional multi-frame sets are destroyed — FIXED

Nothing detects HDR brackets, focus stacks or panorama sequences. They are near-identical frames close in time — precisely the grouper's signature — so it nominates a winner and marks the rest reject candidates. This is the most damaging single failure mode for landscape, architecture and real-estate users.

Fix: detect before ranking, from EXIF the plugin already has:

• Bracket: same scene, monotone exposure ladder, exposureBias varying, aperture/ISO otherwise consistent → mark group_type = "bracket", suppress winner/reject entirely.
• Focus stack: same scene, static framing, sharp-region location migrating across frames.
• Panorama: sequential frames, consistent exposure, partial content overlap with a translational shift.

Correction after implementing it. "Same scene" cannot be verified by pHash for a bracket, and the first implementation's attempt to do so rejected every bracket it was meant to confirm. Measured against a real +2 EV frame of an otherwise identical scene, the Hamming distance to the base frame is 61 of 64 bits: clipping into the highlights flips which DCT coefficients sit above the median, so the hash comes back near-complemented. Changing the exposure is exactly what destroys a perceptual hash, which makes it useless on the one input that matters here.

Worse, the same failure meant the grouper never put the frames in one group to begin with — on the fast tasks=cull path there is no embedding, so pHash was its only similarity signal. Detection was running on groups that could never contain a bracket.

The replacement evidence is the exposure pattern itself: three or more evenly spaced stops spanning at least a full EV, shot within a few seconds. Only auto exposure bracketing produces that, and the false positive it needs to exclude — a photographer riding the compensation dial through a burst — lands on uneven steps. Grouping gained a matching bracket_edges rule that joins frames close in time whose exposure compensation differs, without asking pHash.

Focus-stack detection does still check framing, and correctly: there the exposure is constant, so the hash means what it says.

Dead config that silently did nothing — FIXED

Presets appeared tunable but largely were not. All four of these are now fixed; kept here because they explain why preset tuning historically had no effect:

• ImageMetricsConfig is entirely dead. Every field has zero references outside culling_config.rs; the live values are duplicated as consts in metrics.rs:112-127. No preset can tune any image metric.
• Most of FaceMetricsConfig is dead, duplicated as consts in face_quality.rs:6-13 and face_aggregate.rs:5-15. Only the five score_weight_* are read.
• grouping.time_window_default_seconds is dead — derive_grouping_thresholds takes the window from the request only (grouping.rs:74). The event (=2) and sports (=3) preset overrides do nothing.
• grouping.duplicate_distance_auto = 0.05 is dead, superseded by the min + normalized·span formula yielding 0.0294 at defaults.

Thread CullingConfig through culling_metrics and face_quality and delete the duplicated constants. Without this, every genre-tuning change in Part 3 is a no-op.

---

What "best frame" means, per genre

Presets today are default / event / portrait / sports / street (culling_config.rs:228), differ only in ranking weights, and the user picks one by hand.

Genre	Volume	What decides the pick	What breaks today
Wedding / event	2–5k/day	Every face eyes-open; key-person priority; the moment (kiss, ring, first dance); mixed dim light	max-aggregation hides nine blinkers behind one open-eyed face; blink proxy weak; absolute noise penalty misfires at high ISO
Portrait / headshot	low, high precision	Sharpness on the eye plane; catchlight; micro-expression; hair across face; half-blink	512 px global sharpness cannot resolve eye vs. nose focus; 17 px eye window
Sports / action	1–5k bursts	Subject sharp — background should be soft; peak action; subject not clipped by frame edge	Global sharpness penalises correct shallow DOF and panning
Wildlife / birds	large bursts	Animal eye sharp + catchlight; wing/limb position; background separation	YuNet is human-only; no animal-eye detection at all
Landscape / architecture	low	Corner-to-corner sharpness; horizon level; brackets/stacks/panos must survive intact	Intentional sets are culled
Street / documentary	low burst rate	Moment and gesture over technical; grain and motion blur are legitimate	Technical weights over-penalise the aesthetic
Real estate / product	tripod-locked repeats	Sharpest of identical frames; verticals; bracket sets	Bracket sets again
Family / kids / pets	chaotic	Multiple subjects eyes-open; expression; motion-blur tolerance	Same inverted eye aggregation
Concert / low light	high ISO	Noise is expected — penalise relatively; stage clipping is normal	Absolute exposure target and absolute noise penalty both misfire
Astro / night	low	Star trailing; intentional darkness	Exposure metric inverts the ranking

Two structural conclusions

Genre should be detected, not asked. SigLIP2 embeddings already exist for indexed photos; zero-shot classification against a fixed genre prompt set is a dot product against cached text embeddings — effectively free. Detect per group, expose it in the response as detected_genre + confidence, and let the user's explicit preset override it. Fall back to default when no embedding exists (the fast cull path) or confidence is low.

Genre changes which signals are trusted, not just their weights. A weights-only preset system cannot express:

• sports/wildlife: use subject-region sharpness, ignore background sharpness;
• landscape/real-estate: run bracket/stack/pano detection and suppress culling for those groups;
• event/family: aggregate eyes-open as size-weighted min, not max;
• concert/astro: switch exposure and noise to relative-only scoring.

So the preset struct needs behavioural fields (which sharpness estimator, which face aggregation, whether set-detection is active), not only more f64 weights.

Learning the user's taste

The strongest differentiator, and the infrastructure precedent already exists (TaskTrainFromEdits.lua, lrg-analysis/src/training.rs, the edit_training LanceDB table).

Every photographer's catalog is full of free labels: pick flags, star ratings, which frames got develop edits, which got exported. Train a small pairwise ranking head on [SigLIP2 embedding ‖ technical metrics] over within-group comparisons drawn from that history, and blend it as a weak term. This is what turns "technically correct" picks into "picks that look like yours". Ship it behind an explicit opt-in with a visible confidence, after the deterministic layers are solid.

---