-
-
Notifications
You must be signed in to change notification settings - Fork 67
Closed
Description
I tried to make use of an ArrayPartition
to prevent Arrow.jl arrays from being loaded into ram when concatenating tables.
The problem is that as soon as you want to actually materialise an array to do some operation the type constraints of copy
are violated since a copied arrow array becomes a normal array (which I guess is due to the original array being a special type which is mmaped to disk).
I understand this is not the intended use case of this package, but I wonder if something like
function Base.copy(A::ArrayPartition{T}) where T
newx = copy.(A.x)
ArrayPartition{T, typeof(newx)}(newx)
end
would work instead of current implementation? This is still type stable (as long as copy.(A.x)
is type stable) so it should not have any performance disadvantages.
Metadata
Metadata
Assignees
Labels
No labels