fix(readutil): preserve decimal scale in zonemap filters

[XuPeng-SH]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #26808
issue #26817

What this PR does / why we need it:

DECIMAL fast filters retained folded literal bytes but lost the literal's scale. Object and block zonemap comparisons then interpreted a lower-scale bound as though it used the persisted column scale, which could prune matching blocks and return wrong rows.

This change preserves the bound type/scale in a typed zonemap and uses scale-aware comparisons for object filters, block filters, sorted quick-break/seek, equality, BETWEEN, and in_range hints 0–3. Incompatible comparison metadata fails open. DECIMAL Bloom lookups skip unsafe raw-byte rejection when type or scale differs. It also makes CompareDecimal64WithScale promote to DECIMAL128 when scaling overflows or crosses the signed range.

It also makes unknown/uninitialized sorted zonemaps fail open instead of treating them as proof to stop scanning. This closes the NULL-first nullable CLUSTER BY false-pruning path reproduced with DECIMAL, BIGINT, and VARCHAR.

Validation includes:

• DECIMAL64/128 white-box matrices across lower/equal/higher scales, negative and extreme values
• <, <=, >, >=, =, BETWEEN, and in_range 0–3
• object/block filters, sorted break/seek, comparison-failure fail-open, and Bloom controls
• non-DECIMAL control and signed-overflow regression
• public SQL regression and controlled CGo package tests/build/vet
• independent 120,000-row oracle A/B before and after restart
• independent 60,000-row indexed UPDATE/flush matrix and direct hidden-table audits

No partitioned table is involved.

[XuPeng-SH]

Independent owner-review found a P1 before merge; this draft is being reworked.

The new scale-aware helpers currently return false for an uninitialized DECIMAL zonemap. On a nullable sorted/cluster key, an all-NULL block can legitimately have an uninitialized zonemap. In the <, <=, =, BETWEEN, and in_range sorted paths, that false can be converted into quick-break, causing the executor to stop before later initialized blocks that contain matching values.

Required closure:

• preserve unknown/uninitialized vs known-nonmatch semantics;
• never use unknown metadata as proof for quick-break or seek exclusion;
• add deterministic DECIMAL64/128 tests with an uninitialized first block and later matching blocks;
• cover comparison operators, BETWEEN, and in_range hints 0–3;
• rerun the real SQL/restart/physical-index acceptance after the fix.

PR remains draft and should not be accepted at 5c29d70244.

[XuPeng-SH]

Scope update: baseline black-box testing proves the uninitialized sorted-zonemap problem is generic, not DECIMAL-only. It reproduces with nullable BIGINT and VARCHAR cluster keys (</<= return 0 while expression oracles return 19,999/20,000; EXPLAIN ANALYZE inputBlocks=0). Filed #26817 and added it to this draft. Rework must fix both #26808 and #26817 before acceptance.

[XuPeng-SH]

Follow-up 7a5c1cc0e6 resolves the owner-review blocker and is independently accepted locally.

Validation:

• go test ./pkg/vm/engine/readutil -count=1 PASS.
• Added tri-state zonemap results; unknown/uninitialized/type-incompatible metadata fail open at object, block, quick-break, and sorted seek layers.
• Sorted seek now validates every block metadata entry before binary search, including a regression where the unknown block is deliberately not sampled by sort.Search.
• Fresh 55 binary: persisted NULL-first DECIMAL/BIGINT/VARCHAR < direct/oracle all match at 19,999 rows and checksum 599,970,000; DECIMAL <= direct/oracle both 20,000.
• Original 120k DECIMAL scale matrix (< <= > >= = BETWEEN, scale 0/1/4/6, negative and status combinations) matches every expression oracle.
• Restarted the candidate and reran all three persisted probes: still exact; pprof healthy.
• Four hidden indexes on idx_rewrite_parent: each 120,000 rows / 120,000 distinct PK mappings.
• Seek benchmark: 16 blocks ~110ns, 256 ~1.18us, 4096 ~18.7us, 0 allocs/op.

No partition scenario was used. CI is not being monitored.

[XuPeng-SH]

Five-PR live combination acceptance passed on 55:

• Exact heads: fix(plan): normalize decimal index range bounds #26806 c09d232b67, fix(readutil): preserve decimal scale in zonemap filters #26815 5c29d70244 + 7a5c1cc0e6, fix(plan): preserve NULL semantics for index upper bounds #26848 b4c2d1b0c9, perf(plan): cost encoded regular index access #26850 0278375310, fix(plan): preserve forced index access for ORDER BY #26851 18091e76cd.
• Integration commit/tree: 277890cf3ce3b8ff5879f4b62a449b8888442521 / 796cf343da9b0e3ac3270d7c98603ee2ba4fef2e.
• Round 36: 6k customers / 30k orders / 90k items, 13 DML epochs, 10 JOIN/EXISTS/UNION/CTE/window/aggregate query families. All 130 indexed-vs-shadow result digests passed; failures=0, perf_candidates=0. All 9 hidden index tables passed count/distinct-PK audits after every epoch.
• Round 38: 20k rows / 7 hidden indexes; rollback, unsupported SAVEPOINT followed by full rollback, failed compound-unique UPDATE, committed PK+multi-index UPDATE, second-unique ODKU conflict, flush and restart all passed.
• Round 31: 124,971 rows; nullable BIGINT/text/PK-leading direct predicates exactly matched expression oracles. Every one of 4 hidden tables was 124971 rows / 124971 distinct PK / 0 decoded-key mismatch after mixed UPDATE/DELETE/INSERT and flush.
• Restart controls: Round 32 persisted 80k/24-epoch state machine remained 0 failures; Round 38 and Round 39 physical audits passed, including per-column serial_extract checks for unique, nullable, DECIMAL and PK-leading keys.

Important interaction found during staging: the four-way candidate without #26806 made the base/shadow DECIMAL path correct via #26815 while the secondary-index range path remained wrong (BETWEEN: 0 vs 13,300; Round 38: 20 vs 1,925). The older triple had appeared to pass Round 38 because the base zonemap and secondary-index range defects canceled each other. Adding #26806 removed that false agreement and produced the all-green results above.

[XuPeng-SH]

Follow-up deep review completed on fb3879c58df after rebasing the full PR onto latest main at db2ef4dd03.

The prior P1 is closed. The fail-open contract now covers scalar comparisons, equality, BETWEEN/in_range, IN, and all prefix variants at object, block, quick-break, and sorted-seek layers. Unknown/uninitialized or type-incompatible zonemaps cannot exclude data. Decimal IN membership also requires compatible scale before zonemap/Bloom rejection, while typed min/max bounds retain safe cross-scale quick-break/seek. Sorted seek validates the object zonemap and every block zonemap before binary search.

The follow-up review also closed two adjacent prefix-vector unhappy paths: malformed or non-VARCHAR payloads do not compile, and nullable prefix vectors conservatively skip zonemap pruning/seek rather than entering the non-null sorted-vector fast path.

Regression coverage now includes DECIMAL64/128, BIGINT, VARCHAR, IN, prefix_eq/prefix_between/prefix_in_range hints 0-3/prefix_in, unknown leading blocks, incompatible metadata, scale-mismatched vectors, nullable prefix vectors, malformed payloads, and an unsampled unknown block that would otherwise make binary search unsafe.

Validation on the rebased tree:

• controlled CGo full tests: ./pkg/container/types ./pkg/vm/engine/readutil PASS
• focused fail-open/type/null regressions PASS
• go build -mod=readonly and go vet -mod=readonly for both packages PASS
• git diff --check PASS
• seek benchmark remains 0 allocs/op; 256 blocks about 0.98 us on Apple M4

No remaining code-review findings in the complete PR diff. GitHub does not permit approving my own PR, so this is recorded as a COMMENT review. CI was not awaited.