Handle extreme outliers in 1D quantization by rebuilding histogram over trimmed range

[slimbuck]

Summary

• When extreme outliers stretch the histogram range so wide that most bins are empty (non-empty bins < k), the quantizer now rebuilds the histogram over a percentile-trimmed range (0.1% from each tail) to restore bin resolution
• Outlier values are clamped into edge bins rather than discarded, so no data is lost — they just no longer dominate the range and starve the codebook of useful centroids

Technical Details

The quantize1d function builds a 1024-bin histogram between dataMin and dataMax, then uses DP to find optimal centroids. When a handful of extreme outliers inflate the range, nearly all values cluster into a tiny fraction of the bins, leaving most bins empty. If nonEmpty < k, the DP cannot produce k distinct centroids, resulting in a degraded codebook.

The fix detects this condition after the initial histogram build and, when triggered:

1. Walks inward from both tails to find the bins containing the 0.1th percentile boundaries
2. Recomputes newMin/newMax from those bins
3. Rebuilds counts, sums, centers, weights, and all prefix-sum arrays over the tighter range
4. Recalculates nonEmpty before proceeding to the DP solver

Values outside the trimmed range are clamped to bin 0 or bin H-1, preserving their contribution without wasting resolution on near-empty stretches of the number line.

Test Plan

• Verify quantization quality on scenes with known extreme outlier splats (e.g., scale values spanning many orders of magnitude)
• Confirm no regression on clean scenes where outliers are absent (the rebuild path is skipped when nonEmpty >= k)
• Check that codebook centroids are well-distributed across the value range after the fix

[Copilot]

Pull request overview

This PR updates the 1D quantizer’s histogram pre-processing to better handle datasets with extreme outliers that otherwise make the histogram too sparse to support k distinct centroids.

Changes:

• Detects when the initial histogram has fewer non-empty bins than k, and rebuilds the histogram over a percentile-trimmed range.
• Clamps values outside the trimmed range into the edge bins so outliers still contribute without dominating histogram range.
• Recomputes bin centers, weights, and prefix sums after rebuilding before running the DP solver.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.