pointer(A::Array{T}) has misleading type for non-bitstype A

[chethega]

julia> function foo()
       A=[([],), ([],)];
       x=unsafe_load(pointer(A));
       A[2]=x;
       p1 = unsafe_load(reinterpret(Ptr{UInt64},pointer(A)), 1)
       p2 = unsafe_load(reinterpret(Ptr{UInt64},pointer(A)), 2)
       p3 = unsafe_load(reinterpret(Ptr{UInt64},p2))
       @show p1,p2,p3
       return A
       end

julia> r=foo();
(p1, p2, p3) = (0x00007f2f0543f8d0, 0x00007f2f05638110, 0x00007f2f0543f8d0)

julia> r
2-element Array{Tuple{Array{Any,1}},1}:

signal (11): Segmentation fault

I am not entirely sure whether this is supposed to be working. If not, then maybe unsafe_load could error out earlier.

If this is supposed to be working, we see that someone got confused about types / numbers of indirection: A[2] points to something that points to A[1], instead of pointing to something that points to A[1][1] (like e.g. A[1]). Also reproduces in the REPL (ie without compilation). Tested on 1.0.2.

[KristofferC]

DId this come up in any real code for which this is a MWE of?

[chethega]

No; when working with constructions that are new to me (like above) I tend to run some tests before deciding whether to structure my code around them. I think that the correct way is to ccall something to get an object from a pointer. But since the above doesn't throw an error (honestly what I expected) and is the obvious syntax, well, the result cannot be intended.

Edit: To be clear, I think the issue is that julia mistakenly believes that unsafe_load(p::Ptr{T})::T. So either unsafe_load should error for types that cannot be directly loaded from pointers (I think 0.6 did that), or it should return something of type T, or the compiler should be aware that the return value is of a different type (boxed T?).

[yuyichao]

Yes unsafe_load is supposed to work, but not supposed to work with your input. The correct input is a pointer to the type, not a pointer to a slot for the pointer to that type.

julia> A = [([],), ([],)]
2-element Array{Tuple{Array{Any,1}},1}:
 ([],)
 ([],)

julia> B = Base.cconvert(Ref{Tuple{Vector{Any}}}, A[1])
Base.RefValue{Tuple{Array{Any,1}}}((Any[],))

julia> Base.unsafe_convert(Ref{Tuple{Vector{Any}}}, B)
Ptr{Tuple{Array{Any,1}}} @0x00007fd11de85520

julia> unsafe_load(Base.unsafe_convert(Ref{Tuple{Vector{Any}}}, B))
(Any[],)

The fundamental issue is that Ptr of non bitstype is ambiguious. In the current state, it makes sense for both pointer(A) and Base.unsafe_convert(Ref{Tuple{Vector{Any}}}, B) to return such a type and this is not fixable without a breaking change that changes this.

[yuyichao]

In another word, the correct fix is to make pointer(A) returns Ptr{Ptr{Tuple{Vector{Any}}}} instead. That will be a breaking change.

[vtjnash]

So either unsafe_load should error for types that cannot be directly loaded from pointers (I think 0.6 did that), or it should return something of type T, or the compiler should be aware that the return value is of a different type (boxed T?).

All of those statements are true. What is false is the assumption that unsafe_load will work with Array-derived pointers. That is not fixable, although various breaking changes (such as suggested by yuyichao) could achieve partial compatibility for some cases.

[chethega]

Changing the type of pointer(A) would be a nice fix.

How breaking would that change be? That is, there are not a lot of useful things people can do with the current pointer type. In order to avoid future breakage if any kind of inline storage comes, Ptr{Nothing} could also be a sane return type (effectively void*).

[mbauman]

You definitely need to know quite a bit about our array layout internals in order to work with unsafe_load. Another place that this will fail is with discriminated unions:

julia> A = [1, missing, 3]
3-element Array{Union{Missing, Int64},1}:
 1
  missing
 2

julia> ptr = pointer(A)
Ptr{Union{Missing, Int64}} @0x0000000116dda6d0

julia> unsafe_load(ptr, 1)

signal (11): Segmentation fault: 11

In fact, there are cases where unsafe_load couldn't possibly do the right thing there as the pointer may get disconnected from the array header.

My read on this is that the layout of bitstype arrays is deterministic, documented, and can be relied upon, but venture outside of that and you're in undocumented/unstable waters.

[vtjnash]

Ptr{Nothing} is not quite the same as void* and is a useful operation in some cases (#29929). It could be Ptr{Union{}} though to express that it's illegal to use the result of pointer.

[chethega]

Ptr{Nothing}===Ptr{Cvoid} is the return-type of pointer_from_objref. But Ptr{Ptr{Nothing}} or Ptr{Ptr{Any}} or the correct type (Ptr{Ptr{T}}) might be better for non-inline stored arrays. For bitsunions, I agree with @mbauman that there is no sensible type, so it wants to be untyped (whether that means Ptr{Nothing}, Ptr{Cvoid} or Ptr{Union{}}).