perf(python): back int64 arrays with typed storage - #4864
Merged
Conversation
`TypedArrayCompatible` excluded Int64 and UInt64 with a comment inherited from
the JavaScript target — "don't use typed array for int64 until we remove our
int64 polyfill and use JS BigInt". That constraint has never applied to Python.
The Rust core has backed these with `Vec<i64>`/`Vec<u64>` all along and
`Int64ArrayCons`/`UInt64ArrayCons` are exported from `array_.py`; nothing had
ever emitted them.
So `Array.map` over an `int64[]` passed `None` as the cons and fell back to
boxed `Generic` storage, keeping one wrapper object per element. Only array
literals and `Seq.toArray` specialized, because those go through the typed
constructor rather than a library call.
For int64 this is a win on every axis, since the values are still pyo3 wrappers
and the boxing that specialization avoids is real (1000 elements):
storage a == b sum(a)
Int64 0.09 us 17.6 us
Generic 8.53 us 33.6 us
Equality is 95x because `Vec<i64> == Vec<i64>` compares entirely in Rust where
`Generic` walks element-by-element through Python.
`TestArray.fs` had no int64 coverage at all, which is why this went unnoticed.
The new tests assert value round-tripping through the storage, the boundary
values that would expose a botched i64/u64 extraction, and structural equality.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Python Type Checking Results (Pyright)
Excluded files with errors (4 files)These files have known type errors and are excluded from CI. Remove from
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TypedArrayCompatiblein the PythonReplacements.fsexcludedInt64/UInt64with a comment inherited from the JavaScript target:
That constraint has never applied to Python. The Rust core has backed these with
NativeArray::Int64(Vec<i64>)/UInt64(Vec<u64>)all along, andInt64ArrayCons/UInt64ArrayConsare exported fromarray_.py— nothing hadever emitted them.
The visible effect:
Array.mapover anint64[]emittedmap(f, xs, None)andfell back to boxed
Genericstorage, keeping one wrapper object per element.Only array literals and
Seq.toArrayspecialized, because those go through thetyped constructor rather than a library call — which is also why it looked fine
in passing.
Effect
For int64 this is a win on both axes rather than a tradeoff, because int64
values are still pyo3 wrappers, so the boxing that specialization avoids is
real. 1000 elements, release core:
a == bsum(a)a[500]Equality is 95× because
Vec<i64> == Vec<i64>compares entirely in Rust, whereGenericwalks element-by-element through Python. Memory drops from an 8-bytepointer plus a wrapper object per element to a flat 8 bytes.
Storage kind before → after, via quicktest:
Note on scope
Typed storage extracts to
i64/u64and raises if the value does not fit,where
Genericaccepts anything. For a statically-typedint64[]that isunreachable, and the boundary values are covered by the new tests — but it is a
tightening, not a pure no-op, so it is worth knowing about.
This does not close the remaining specialization holes:
List.toArray,Array.ofSeqandSet.toArraystill produceGeneric, because those libraryfunctions erase
'Tbefore they allocate and there is no cons for the call siteto pass. Fixing that means threading a parameter through F# sources shared with
other targets, so it is deliberately left out of here.
Test plan
--force-fable-library: 2445 passed (was 2442)TestArray.fscoverage — the file had no int64 tests at all, whichis why this went unnoticed. Asserted against .NET first (228 array tests
pass on .NET): value round-tripping through the storage, boundary values
(
Int64.MinValue/MaxValue,UInt64.MaxValue) that would expose a botchedextraction, and structural equality — the operation the specialization is for
including
Int64.MaxValueround-tripping as 9223372036854775807🤖 Generated with Claude Code