Skip to content

Exact decimal precision and Arrow ARRAY support

Latest

Choose a tag to compare

@HenryNebula HenryNebula released this 24 Oct 17:59
· 2 commits to master since this release

Highlights

Decimal precision overhaul (#119)

  • NUMERIC/DECIMAL with precision > 38 now maps to Arrow decimal256 (up to 76 digits) instead of crashing with UnsupportedOperationException: Decimal size greater than 16 bytes
  • Values beyond decimal256 — including NUMERIC(1000, 64) — and unbounded numerics (Postgres untyped NUMERIC, Oracle NUMBER, SQLite) transfer as UTF-8 strings and rebuild to exact Python Decimals: no silent rounding of fractional digits past scale 17
  • Documented tiering: decimal128 (≤ 38 digits) → decimal256 (≤ 76) → exact string fallback

DECIMAL ARRAY columns (#121)

  • DECIMAL/NUMERIC array elements ride the same unbounded-decimal string fallback (element precision/scale are unknowable from ResultSetMetaData) and restore as exact Decimals on every fetch path — fetchone/fetchmany/fetchall, fetch_df (including pandas ndarray cells, #129), preserving NULL arrays and NULL elements

Arrow-native fetch paths (#125)

  • fetch_arrow_batches now imports the JVM VectorSchemaRoot via the zero-copy C Data Interface — every Arrow type the JVM produces is supported, not just primitive vectors; fetch_arrow_table casts decimal-array fallback columns back to list<decimal256> when the data fits

Performance

  • String-decimal fallback restored row-wise (not via zip transpose); Arrow-decimal batches skip the restore entirely
  • Relative performance regression tests vs jaydebeapi guard the speedups (~3–7× on the measured paths)

Fixes & maintenance

  • fetch_df works on pandas < 1.5 instead of requiring it; exponent-padded digits counted correctly in decimal width inference
  • DBAPITypeObject hashable for SQLAlchemy 2.x
  • Docs: production/preview publishing pipeline for GitHub Pages

Full changelog: v2.1.6...v2.1.7