fix: exact int64 boundary handling in numeric finalization (#34)

[JohnnyWilson16]

Summary

Fixes #34.

_finalize_numeric used float(nonnull.abs().max()) < float(2**63 - 1024) to decide whether an integral column fits int64. Two real defects:

• Magnitude-only check: -2**63 is a valid int64, but its magnitude (2**63) fails the check → needless float64 demotion of legitimate data.
• Boundary exclusion: 2**63 - 1024 — the largest float64 below 2**63 — converts to int64 exactly, but strict < rejected it → float64 demotion.

Why the issue's proposed fix is unsafe

The issue suggests _INT64_SAFE = float(2**63 - 1) with <=. That constant rounds up to 2**63 in float64 (float(2**63 - 1) == float(2**63) is True), so the proposed guard would admit float(2**63) and let .astype("int64") wrap around — it would introduce the very overflow the issue worries about. Notably, the current code does not actually overflow (float rounding aligns at the 1024-ulp boundary); its failure mode is wrong dtype decisions, not corruption. The PR description of record for #34 should reflect that.

Fix

Compare min()/max() in Python-int space against exact int64 bounds (_fits_int64). Python ints are arbitrary-precision → no rounding at the boundary; int() is exact because integrality (% 1 == 0) is checked first; inf raises OverflowError → treated as not fitting.

Behavior/compat

Values in {-2**63} ∪ [2**63 - 1024] (and int64-extreme integers arriving via object/int paths) now finalize as int64/Int64 instead of float64 — numerically identical, dtype now correct. Everything else unchanged.

Tests

• test_finalize_numeric_int64_boundaries_no_overflow_no_demotion and test_finalize_numeric_int64_min_not_rejected_by_abs_asymmetry — both fail on main (float64 demotion), pass with the fix; also pin that float(2**63) stays float64 and the nullable Int64 path is exact.
• test_huge_integer_strings_still_stay_float_end_to_end — pipeline-level control (2**64 stays float64).

Verification

pytest tests/test_dtypes.py -q             # 25 passed
pytest -m "not online and not large"       # green, coverage 93.49% (gate 93)
ruff check src tests && mypy src/freshdata # clean

Summary by CodeRabbit

• Bug Fixes
  • Improved numeric type handling near the int64 boundaries.
  • Values within the exact int64 range now convert correctly without unnecessary conversion to float64.
  • Values outside the representable range continue to be safely handled as float64.
  • Corrected handling of the minimum int64 value and very large integer-like inputs.

[strix-security]

Strix Security Review

No security issues found.

_{Updated for 09b6300.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6a1bd919-b0b3-4048-ac9b-48c4e6d8fcf7

📥 Commits

Reviewing files that changed from the base of the PR and between a136c05 and 10685d6.

📒 Files selected for processing (3)

• CHANGELOG.md
• src/freshdata/steps/dtypes.py
• tests/test_dtypes.py

---

📝 Walkthrough

Walkthrough

Changes

Int64 finalization

Layer / File(s)	Summary
Replace float overflow guard with exact bounds src/freshdata/steps/dtypes.py	Adds explicit int64 limits and checks integer-space minimum and maximum values before casting integral data.
Validate int64 boundary behavior tests/test_dtypes.py, CHANGELOG.md	Tests upper and lower int64 boundaries, oversized integer-like strings, nullable integers, and documents the corrected behavior.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately describes the main change: exact int64 boundary handling in numeric finalization.
Description check	✅ Passed	The description is detailed and covers the fix, rationale, tests, and verification, with only minor template sections missing.
Linked Issues check	✅ Passed	The changes address #34 by replacing the float threshold with exact int64 range checks and preserve the intended boundary behavior.
Out of Scope Changes check	✅ Passed	The PR changes stay focused on numeric finalization, tests, and changelog notes; no unrelated code changes are evident.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/int64-boundary-finalize-numeric-jwd

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — 2026-07-11T04:06:10Z

• freshdata: 1.1.1
• python: 3.12.13
• platform: Linux-6.17.0-1018-azure-x86_64-with-glibc2.39

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %
crm	10,200	40	1.298	1.317	14.0	100.0	0.0	100.0	93.84	✅	100.0
event_log	10,000	25	0.475	0.487	5.1	100.0	0.0	100.0	99.677	✅	100.0
finance	10,200	60	1.650	1.657	18.3	100.0	0.0	100.0	99.499	✅	100.0
gold	10,200	7	0.195	0.199	3.7	100.0	0.0	100.0	98.3	✅	100.0
provenance	10,000	18	0.406	0.407	7.3	100.0	0.0	100.0	99.765	✅	100.0
wide_schema	10,000	100	2.998	3.033	13.4	100.0	0.0	100.0	96.042	✅	100.0

Authored-code reduction (Metric 6)

• FreshData: 3 lines
• pandas baseline: 26 lines (88.46% reduction)