ArrayInterface 3.1.7 causes `sum` to freeze on array with missings

[cstjean]

On Julia 1.5.4 (and 1.5.2)

(@v1.5) pkg> add Feather, ArrayInterface@3.1.7

julia> using ArrayInterface, Feather
[ Info: Precompiling Feather [becb17da-46f6-5d3c-ad1b-1c5fe96bc73c]

julia> U2 = Matrix{Union{Missing, Float32}}(undef, 10, 10);

julia> my_sum(values::AbstractMatrix) = dropdims(sum(values, dims=2), dims=2)
my_sum (generic function with 1 method)

julia> my_sum(U2)
^C^C^C^C^C^C^C^CWARNING: Force throwing a SIGINT
^C^C^C^C^C^C^C^CWARNING: Force throwing a SIGINT
^C^C^C^C^CSegmentation fault (core dumped)

freezes forever. @etpinard (who is coauthor of this MWE, 👍) says that #137 is to blame. Not sure why Feather is necessary. @ExpandingMan, does it ring a bell? Importing Feather's dependencies doesn't cause the issue, so it should be an interaction with Feather's code...

[ExpandingMan]

That is quite bizarre....

Just to be sure, you are aware that Feather.jl is basically deprecated in favor of the new Arrow.jl right? Granted you still need Feather.jl to read the old feather format.

[etpinard]

To add to the confusion: using Julia 1.6.0, the above MWE works fine.

[Tokazama]

Does this happen with other packages that depend on DataFrames.jl

[etpinard]

Does this happen with other packages that depend on DataFrames.jl

Yep, it does look like it:

(env) pkg> add DataFrames, ArrayInterface@3.1.7

julia> using DataFrames, ArrayInterface

julia> U2 = Matrix{Union{Missing, Float32}}(undef, 10, 10);

julia> my_sum(values::AbstractMatrix) = dropdims(sum(values, dims=2), dims=2)

julia> my_sum(U2)

also hangs forever.

[etpinard]

Digging deeper, off the latest master, commenting out

ArrayInterface.jl/src/axes.jl

Line 126 in cd6e5d2

axes(a, dims::Tuple) = (axes(a, first(dims)), axes(a, tail(dims))...)

and/or

ArrayInterface.jl/src/axes.jl

Line 127 in cd6e5d2

axes(a, ::Tuple{}) = ()

makes the MWE work.

[Tokazama]

That's bizarre b/c axes is not overloaded or exported

[Tokazama]

I'm testing this out right now and I think it's a compiler problem. I'm seeing "subtype_tuple_varargs" and "subtype_tuple_tail" being called from a "src/subtype.c" and the julia file calling it is in ".compiler/abstractinterpretation.jl".

DataFrames.jl makes pretty extensive use of precompilation and I wouldn't be surprised if something has changed in "abstractinterpretation.jl" that fixed this issue.

[ChrisRackauckas]

SciML/DifferentialEquations.jl#740 seems like another casualty of this? It might be best to revert for now?

[Tokazama]

I'm fine with changing that line, but it would be nice to know what is going on here. This seems completely random to me, so I'm not sure how to avoid this issue in the future.

[ChrisRackauckas]

@JeffBezanson ?

[glwagner]

This issue also has caused problems with Oceananigans.jl (one of our functions hangs on compilation with ArrayInterface@3.1.7. It seems that restricting ArrayInterface to 3.1.6 solves this problem.

[glwagner]

Possible related issue: JuliaLang/julia#40457

[chriselrod]

julia> @time using DataFrames, ArrayInterface
  1.164565 seconds (4.06 M allocations: 248.914 MiB, 7.56% gc time)

julia> using DataFrames, ArrayInterface

julia> U2 = Matrix{Union{Missing, Float32}}(undef, 10, 10);

julia> my_sum(values::AbstractMatrix) = dropdims(sum(values, dims=2), dims=2)
my_sum (generic function with 1 method)

julia> @time my_sum(U2)
  0.203117 seconds (905.60 k allocations: 51.556 MiB, 18.88% gc time, 99.99% compilation time)
10-element reshape(::Matrix{Union{Missing, Float32}}, 10) with eltype Union{Missing, Float32}:
 missing
 missing
 missing
 missing
 missing
 missing
 missing
 missing
 missing
 missing

julia> @time using ArrayInterface, Feather
  0.776953 seconds (2.19 M allocations: 143.416 MiB, 2.19% gc time)

julia> U2 = Matrix{Union{Missing, Float32}}(undef, 10, 10);

julia> my_sum(values::AbstractMatrix) = dropdims(sum(values, dims=2), dims=2)
my_sum (generic function with 1 method)

julia> @time my_sum(U2)
  0.183003 seconds (905.61 k allocations: 51.626 MiB, 9.89% gc time, 99.98% compilation time)
10-element reshape(::Matrix{Union{Missing, Float32}}, 10) with eltype Union{Missing, Float32}:
 missing
 missing
 missing
 missing
 missing
 missing
 missing
 missing
 missing
 missing

(@v1.7) pkg> st ArrayInterface Feather
      Status `~/.julia/environments/v1.7/Project.toml`
  [4fba245c] ArrayInterface v3.1.7
  [becb17da] Feather v0.5.9

So I'm not able to reproduce this.

Although I only tested on 1.6.1 and master.
If it was a compiler problem, maybe it's been fixed?