Skip to content

pandas2df: unify integer classification, add bigint option - #1

Merged
jefferis merged 1 commit into
mainfrom
int-classify
Aug 17, 2026
Merged

pandas2df: unify integer classification, add bigint option#1
jefferis merged 1 commit into
mainfrom
int-classify

Conversation

@jefferis

Copy link
Copy Markdown
Member

What

Replaces the two separate integer-conversion paths in pandas2df (the typed int64/uint64 path and the object-column path) with a single classify_integer_strings() decision point, so a column classifies identically regardless of how pandas typed it.

Tiering (mirrors reticulate::py_to_r() where it is faithful)

Each column maps to the narrowest base R type that holds it exactly, reserving bit64::integer64 only for values base R cannot:

magnitude type
≤ 2³¹−1 integer
< 2⁵³ double (exact)
≥ 2⁵³ integer64

This drops two warts:

  • the int64/uint64 asymmetry — small uint64 count columns were being force-promoted to bit64; they are now plain integer like int64.
  • the object-vs-typed-path divergence — object-int columns used to become double and silently left overflow alone, while typed columns stop()-ped on overflow.

New bigint argument

pandas2df(x, bigint = c("auto", "integer64", "character")):

  • "auto" (default) — the tiering above.
  • "integer64" — every integer column → integer64, for a stable schema regardless of magnitude.
  • "character" — the large (≥ 2⁵³) tier → character, as data.table::fread() offers.

uint64 values beyond the signed 64-bit range cannot be held by integer64 (which is signed); they are returned as character — the only faithful option — with a warning unless bigint = "character".

Testing

  • New tests cover every tier boundary and all three modes plus the overflow warning.
  • Verified end-to-end against a real pandas frame spanning small/mid/large int64, small uint64 counts, and overflowing uint64.
  • R CMD check clean (0/0/0); full matrix (Python 3.9 / reticulate-default / 3.13) run on the branch.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

🤖 Generated with Claude Code

Replace the separate int64/uint64 and object-column integer paths with a
single classify_integer_strings() that tiers each column by magnitude,
mirroring reticulate's py_to_r where it is faithful and reserving bit64 only
where base R cannot represent a value exactly:

  * <= 2^31-1  -> integer
  * < 2^53     -> double (exact)
  * >= 2^53    -> integer64

This drops the uint64 special case (small uint64 count columns no longer get
forced to bit64) and the object-vs-typed-path divergence. New bigint argument:
'integer64' forces every integer column to integer64 for a stable schema;
'character' returns the large tier as character (fread style). uint64 values
beyond the signed 64-bit range cannot be held by integer64 and are returned as
character, with a warning outside 'character' mode.
@jefferis
jefferis merged commit 77ccbab into main Aug 17, 2026
6 checks passed
@jefferis
jefferis deleted the int-classify branch August 17, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant