Skip to content

Support integer, numeric, double and date CDX index keys#298

Merged
chrisrichards merged 3 commits into
mainfrom
feature/numeric-index-keys
Jul 6, 2026
Merged

Support integer, numeric, double and date CDX index keys#298
chrisrichards merged 3 commits into
mainfrom
feature/numeric-index-keys

Conversation

@chrisrichards

Copy link
Copy Markdown
Member

Closes #292.

Summary

Non-character index tags now work as automatic access paths. In the foxprodb fixtures this unlocks the integer-keyed primary/foreign-key tags — where CONTACT_ID = 1 is now an index seek:

select CALL_ID from calls.dbf where CONTACT_ID = 1      → index seek (=) on tag 'CONTACT_ID'
select CALL_ID from calls.dbf where CALL_ID between 4 and 9 → index range scan (between) on tag 'CALL_ID'
select * from calls.dbf order by CONTACT_ID             → index order scan on tag 'CONTACT_ID'

The key transforms (CdxKeyEncoder)

  • Integer (I): 4-byte big-endian with the sign bit flipped. Verified against the fixtures byte-for-byte — before writing any code, a diagnostic dump confirmed 1 → 80-00-00-01 on real VFP data, and a permanent test now decodes all 32 keys in both calls.CDX tags and matches them to the column values of the records they point at.
  • Numeric/Float/Double (N/F/B) and Date (D): 8-byte big-endian IEEE 754 double — sign bit flipped for non-negatives, all bits flipped for negatives (the standard total-order transform, per the same xBase format references the DBF reader already cites); dates as Julian day numbers using the identical epoch constant as DbfValueDateTime (anchors tested: 1970-01-01 → 2440588, 2000-01-01 → 2451545). No fixture has an N/D-keyed tag, so these are covered by exact-bytes and order-preservation unit tests rather than differential ones — contributions of VFP files with such tags welcome.
  • DateTime (T) and Currency (Y) keys remain unsupported (fractional-day and int64-vs-double encodings unverified); tags on those columns fall back to scans.

Planner semantics worth reviewing

  • Integer bounds are adjusted to the integer domain: CALL_ID >= 9.5 seeks from 10, = 1.5 and between 8.5 and 8.9 are provably empty without touching the table, and bounds beyond int range either return nothing or defer to a scan.
  • Double/date strict bounds stay inclusive at the key level: converting a decimal literal to a double can collapse a strict boundary onto the bound itself, so >/< include the boundary in the index range and let the always-applied residual filter drop the boundary rows — the superset invariant does the safety work.
  • Binary keys pad with 0x00, not the character space pad — used both in comparators and duplicate-run stabilization (integer FK runs like CONTACT_ID = 1 now exercise that path with real duplicates).
  • Key-width sanity checks per type (4 for I, 8 for the double family) guard against mismatched tags.

Test plan

33 new tests: encoder exact-bytes/round-trip/monotonicity (integers, doubles, dates, Julian anchors), the 32-key fixture round-trip proof, and a differential suite over calls.dbf (equality incl. duplicate runs/no-match/non-integral/out-of-range, all four range operators incl. flipped operands and fractional bounds, BETWEEN incl. the empty fractional window, integer parameters, residual predicates, ORDER BY satisfied by seeks and by a duplicated-key index scan — exercising stable duplicate ordering — and the typed-builder path). One phase-6 test that asserted integer tags fall back was removed, superseded by the positive coverage. Full suite: 415 passed, 1 pre-existing skip, zero warnings on both TFMs.

🤖 Generated with Claude Code

chrisrichards and others added 3 commits July 6, 2026 15:06
Closes #292.

Visual FoxPro stores non-character index keys as binary transforms
whose unsigned byte order matches value order: integer keys are
big-endian with the sign bit flipped, and numeric, double and date
keys are big-endian IEEE 754 doubles with the sign bit flipped for
non-negatives and every bit flipped for negatives (dates as Julian
day numbers, sharing the epoch constant DbfValueDateTime uses). The
integer transform is verified directly against the fixtures: a test
decodes all 32 keys in the calls.CDX tags and matches them to the
column values of the records they point at.

CdxKeyEncoder implements the transforms; the query planner now
accepts SignedLong, Number, Float, Double and Date tags (with key
width checks) and encodes search values per column type. Integer
bounds are adjusted to the integer domain (ceil/floor, with
provably-empty and beyond-range handling); double-based strict
bounds stay inclusive at the key level, since converting decimals to
doubles can collapse a strict boundary, and the residual filter
restores exactness. Binary keys compare with a zero pad byte rather
than the character space pad, and duplicate-run stabilization pads
accordingly. DateTime ('T') and Currency ('Y') keys remain
unsupported pending verified fixtures for their encodings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ier characters

Bogus can generate file names containing apostrophes (or other
characters outside the bare-name alphabet), which fail tokenization
in the tests that exercise unquoted table names. Replace anything
outside word characters and dots, generalizing the previous
ampersand-only workaround. Pre-existing flake, surfaced by a CI run
on this branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Jul 6, 2026

Copy link
Copy Markdown

@chrisrichards chrisrichards merged commit 438a8c8 into main Jul 6, 2026
3 checks passed
@chrisrichards chrisrichards deleted the feature/numeric-index-keys branch July 6, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support numeric and date CDX index keys

1 participant