Support integer, numeric, double and date CDX index keys#298
Merged
Conversation
Closes #292. Visual FoxPro stores non-character index keys as binary transforms whose unsigned byte order matches value order: integer keys are big-endian with the sign bit flipped, and numeric, double and date keys are big-endian IEEE 754 doubles with the sign bit flipped for non-negatives and every bit flipped for negatives (dates as Julian day numbers, sharing the epoch constant DbfValueDateTime uses). The integer transform is verified directly against the fixtures: a test decodes all 32 keys in the calls.CDX tags and matches them to the column values of the records they point at. CdxKeyEncoder implements the transforms; the query planner now accepts SignedLong, Number, Float, Double and Date tags (with key width checks) and encodes search values per column type. Integer bounds are adjusted to the integer domain (ceil/floor, with provably-empty and beyond-range handling); double-based strict bounds stay inclusive at the key level, since converting decimals to doubles can collapse a strict boundary, and the residual filter restores exactness. Binary keys compare with a zero pad byte rather than the character space pad, and duplicate-run stabilization pads accordingly. DateTime ('T') and Currency ('Y') keys remain unsupported pending verified fixtures for their encodings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ier characters Bogus can generate file names containing apostrophes (or other characters outside the bare-name alphabet), which fail tokenization in the tests that exercise unquoted table names. Replace anything outside word characters and dots, generalizing the previous ampersand-only workaround. Pre-existing flake, surfaced by a CI run on this branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Closes #292.
Summary
Non-character index tags now work as automatic access paths. In the
foxprodbfixtures this unlocks the integer-keyed primary/foreign-key tags —where CONTACT_ID = 1is now an index seek:The key transforms (
CdxKeyEncoder)I): 4-byte big-endian with the sign bit flipped. Verified against the fixtures byte-for-byte — before writing any code, a diagnostic dump confirmed1 → 80-00-00-01on real VFP data, and a permanent test now decodes all 32 keys in bothcalls.CDXtags and matches them to the column values of the records they point at.N/F/B) and Date (D): 8-byte big-endian IEEE 754 double — sign bit flipped for non-negatives, all bits flipped for negatives (the standard total-order transform, per the same xBase format references the DBF reader already cites); dates as Julian day numbers using the identical epoch constant asDbfValueDateTime(anchors tested: 1970-01-01 → 2440588, 2000-01-01 → 2451545). No fixture has anN/D-keyed tag, so these are covered by exact-bytes and order-preservation unit tests rather than differential ones — contributions of VFP files with such tags welcome.T) and Currency (Y) keys remain unsupported (fractional-day and int64-vs-double encodings unverified); tags on those columns fall back to scans.Planner semantics worth reviewing
CALL_ID >= 9.5seeks from 10,= 1.5andbetween 8.5 and 8.9are provably empty without touching the table, and bounds beyondintrange either return nothing or defer to a scan.>/<include the boundary in the index range and let the always-applied residual filter drop the boundary rows — the superset invariant does the safety work.0x00, not the character space pad — used both in comparators and duplicate-run stabilization (integer FK runs likeCONTACT_ID = 1now exercise that path with real duplicates).I, 8 for the double family) guard against mismatched tags.Test plan
33 new tests: encoder exact-bytes/round-trip/monotonicity (integers, doubles, dates, Julian anchors), the 32-key fixture round-trip proof, and a differential suite over
calls.dbf(equality incl. duplicate runs/no-match/non-integral/out-of-range, all four range operators incl. flipped operands and fractional bounds,BETWEENincl. the empty fractional window, integer parameters, residual predicates, ORDER BY satisfied by seeks and by a duplicated-key index scan — exercising stable duplicate ordering — and the typed-builder path). One phase-6 test that asserted integer tags fall back was removed, superseded by the positive coverage. Full suite: 415 passed, 1 pre-existing skip, zero warnings on both TFMs.🤖 Generated with Claude Code