Custom storage for field arrays

[darsnack]

I am trying to figure out how to change what array backs a certain field of a StructArray. Here's a MWE to describe what I want:

julia> struct Foo{T}
         x::T
       end

julia> foos = StructArray(Foo(rand(100)) for _ in 1:10, _ in 1:20);

julia> typeof(foos.x)
Matrix{Vector{Float64}} (alias for Array{Array{Float64, 1}, 2})

Ideally, I would want to be able to specify that for any StructArray{Foo}, foos.x is a dense array. So, in the example, I would want a Array{Float64, 3} with the last dimension being the vector dimension. Assuming I have a special structure that makes a Array{T, 3} look like a Matrix{Vector{T}}, is it possible to tell StructArrays how to use this structure for foos.x?

I tried looking through the custom scheme docs, but it seemed like that was designed for a different purpose (rerouting getproperty?).

[piever]

Normally, storage is changed with https://juliaarrays.github.io/StructArrays.jl/dev/#StructArrays.replace_storage. Once you have that, things will just work, as long as the eltype is respected. (ArraysOfArrays would work well if it weren't for JuliaArrays/ArraysOfArrays.jl#2.)

For example:

julia> struct VectorOfArrays{T, M, N} <: AbstractVector{Array{T, N}}
           content::Array{T, M}
       end

julia> VectorOfArrays(content::Array{T, M}) where {T, M} = VectorOfArrays{T, M, M - 1}(content)
VectorOfArrays

julia> Base.size(v::VectorOfArrays) = (last(size(v.content)),)

julia> Base.axes(v::VectorOfArrays) = (last(axes(v.content)),)

julia> function Base.getindex(v::VectorOfArrays, i::Int)
           sz = Base.front(size(v.content))
           p = prod(sz)
           ptr = pointer(v.content, (i - 1) * p + 1)
           return Base.unsafe_wrap(Array, ptr, sz)
       end

julia> using StructArrays

julia> struct Foo{T}
           x::T
       end

julia> foos = StructArray(Foo(rand(100)) for _ in 1:10);

julia> foos_flat = StructArrays.replace_storage(foos) do v
           VectorOfArrays(reduce(hcat, v))
       end;

julia> foos_flat.x.content
100×10 Matrix{Float64}:

(I think my VectorOfArrays is not quite what you want, you wanted ArrayOfVectors, but I imagine it should be easy to adjust.)

[darsnack]

Ah, I never thought to use replace_storage! Amazing this will work perfectly for me. Thanks!

[darsnack]

I see here that you used unsafe_wrap to return a "view" into the dense array without actually creating a SubArray (which fulfills the requirement that the eltype is respected). In my case, plain Arrays were just the MWE. I am actually using an array wrapper that performs circular indexing of the last dimension. Is there something like unsafe_wrap but for more than Array?

In my actual code, I want to translate a Matrix{<:CircularArray{T, 1}} into a CircularArray{T, 3}, but right now, my wrapper that does this translation returns a view into the CircularArray{T, 3}. It's kind of hacked together. I have ArrayOfCircularVectors{...} <: AbtractArray{CircularVector{T, 1}, M}, but getindex returns a view of the eltype instead of the promised eltype.

Aside for just being confusing to not return the correct eltype, this ends up causing problems further down the line for things like displaying the struct array containing these components.