pandas2df: unify integer classification, add bigint option - #1
Merged
Conversation
Replace the separate int64/uint64 and object-column integer paths with a single classify_integer_strings() that tiers each column by magnitude, mirroring reticulate's py_to_r where it is faithful and reserving bit64 only where base R cannot represent a value exactly: * <= 2^31-1 -> integer * < 2^53 -> double (exact) * >= 2^53 -> integer64 This drops the uint64 special case (small uint64 count columns no longer get forced to bit64) and the object-vs-typed-path divergence. New bigint argument: 'integer64' forces every integer column to integer64 for a stable schema; 'character' returns the large tier as character (fread style). uint64 values beyond the signed 64-bit range cannot be held by integer64 and are returned as character, with a warning outside 'character' mode.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the two separate integer-conversion paths in
pandas2df(the typedint64/uint64path and the object-column path) with a singleclassify_integer_strings()decision point, so a column classifies identically regardless of how pandas typed it.Tiering (mirrors
reticulate::py_to_r()where it is faithful)Each column maps to the narrowest base R type that holds it exactly, reserving
bit64::integer64only for values base R cannot:≤ 2³¹−1integer< 2⁵³double(exact)≥ 2⁵³integer64This drops two warts:
int64/uint64asymmetry — smalluint64count columns were being force-promoted tobit64; they are now plainintegerlikeint64.doubleand silently left overflow alone, while typed columnsstop()-ped on overflow.New
bigintargumentpandas2df(x, bigint = c("auto", "integer64", "character")):"auto"(default) — the tiering above."integer64"— every integer column →integer64, for a stable schema regardless of magnitude."character"— the large (≥ 2⁵³) tier → character, asdata.table::fread()offers.uint64values beyond the signed 64-bit range cannot be held byinteger64(which is signed); they are returned as character — the only faithful option — with a warning unlessbigint = "character".Testing
int64, smalluint64counts, and overflowinguint64.R CMD checkclean (0/0/0); full matrix (Python 3.9 / reticulate-default / 3.13) run on the branch.Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
🤖 Generated with Claude Code