ComponentArrays.jl crashes Julia in VSCode but not in REPL

[amrods]

I'm crossposting from Discourse

The following code crashes Julia in VSCode, but the REPL can run it with no problem.

using ComponentArrays

szE = 2
szG = 2
szA = 9
szK = 5
szT = 44
szB = 2
szZ = 2

costdata = Array{Float64}(undef, szT)
wwdata = Array{Float64}(undef, szG, szE, szA, szZ, szK, szT)
Xdata = Array{Int}(undef, szG, szE, szA, szT)
N19data = Array{Int}(undef, szG, szB, szE, szT-6)
Adata = Array{Float64}(undef, szT)
ω = Array{Float64}(undef, szG, szA, szT)
Mdata = Array{Int}(undef, szA, szA, szE, szE, szT)
Smdata = Array{Int}(undef, szA, szE, szT)
Sfdata = Array{Int}(undef, szA, szE, szT)
LLdata = Array{Float64}(undef, szG, szE, szA, szZ, szK, szT)
P_1data = Array{Float64}(undef, szK, szT-1)
edata = Array{Float64}(undef, szK, szT)

## block

data = ComponentArray(
    ww = wwdata,
    e = edata,
    P_1 = P_1data,
    X = Xdata,
    LL = LLdata,
    N19 = N19data,
    A = Adata,
    ω = ω,
    cost = costdata,
    M = Mdata,
    Sm = Smdata,
    Sf = Sfdata
)

println("success")

I suspect it has to do with memory use because if I comment out the line with ww in the definition of the ComponentArray the code runs without crashing. What could be happening?

I'm using Julia 1.5.3, VSCode 1.53.0 and ComponentArrays.jl 0.8.11.

[jonniedie]

Oh, that's a weird one. I'm not sure what would cause that and I am not able to reproduce the error on my side (same Julia and ComponentArrays version, but VSCode 1.52.1 instead).

One thing you can try is changing the ComponentArray( part to ComponentArray{Union{Float64, Int}}(. ComponentArrays, like regular arrays, will try to convert every element to the same type. So in this case it will try to make things all Float64s. Giving an explicit eltype of Union{Float64, Int} will allow elements to keep their type. If this works, than there must have been some issue when converting from Int to Float64. But I can't imagine what that would be (or why it only happens in VSCode).

[amrods]

That seemed to do it. Thank you. One thing though is that the same problem happens with Atom/Juno and VSCodium.

[jonniedie]

Hmmm.. that is weird. I'll have to try it on my other computer that has Juno on it tomorrow. Also, what OS are you using?

[amrods]

MacOS 11.2.

[jonniedie]

Hm. I'm on 11.1. I'll see if this happens when I upgrade to 11.2. But also I'm running it through Rosetta so that might change things.

But that would be weird if the OS was the issue so.... I don't know. I guess I'll just play around until I can replicate it.

[jonniedie]

Okay, I was able to reproduce it on a different computer. I had a line that did vcat(array...) on construction that I used to promote everything to a common type. It was being dispatched in this case to a vcat(::Float64, ::Float64, ::Float64, ::Vararg{Number,N} where N) method and for some reason failing there. I just changed it to reduce(vcat, array) and it seems to work. I know there's gotta be a better way to do this by directly promoting the array eltype myself, but everything I try seems to be much slower or doesn't work like I want it to.

[amrods]

Hmm, but why didn't that crash the REPL?

One thing that is not clear to me is that the ComponentArrays promotes everything to a common type, so is it futile to attempt to keep arrays as Ints or Floats? am I better off simply converting everything to Float64 before feeding that to a CA?

[jonniedie]

Yeah, I'm still not sure why it didn't crash in the REPL. I'm not totally sure why it crashed in the first place, I just noticed that that was the line that seemed to do it for me.

You'll want to use the Union trick if you want to have mixed eltypes, just like you would with normal Arrays. Promotion with ComponentArrays works the same as it does for normal Arrays because CA's store everything as Arrays under the hood. If you'd like the elements to keep their type, you have to explicitly parameterize with a wider element type. Here's an example:

julia> ComponentArray(a=1, b=2.0) # default conversion to Float64
ComponentVector{Float64}(a = 1.0, b = 2.0)

julia> ComponentArray{Int}(a=1, b=2.0) # explicit conversion to Int
ComponentVector{Int64}(a = 1, b = 2)

julia> ComponentArray{Real}(a=1, b=2.0) # widen the eltypes to Real
ComponentVector{Real}(a = 1, b = 2.0)

julia> ComponentArray{Union{Float64, Int}}(a=1, b=2.0) # widen the eltypes to Union{Float64, Int}
ComponentVector{Union{Float64, Int64}}(a = 1, b = 2.0)

This is analogous to what you would do with normal Arrays:

julia> [1, 2.0] # default conversion to Float64
2-element Array{Float64,1}:
 1.0
 2.0

julia> Int[1, 2.0] # explicit conversion to Int
2-element Array{Int64,1}:
 1
 2

julia> Real[1, 2.0] # widen the eltypes to Real
2-element Array{Real,1}:
 1
 2.0

julia> Union{Float64 ,Int}[1, 2.0] # widen the eltypes to Union{Float64, Int}
2-element Array{Union{Float64, Int64},1}:
 1
 2.0

For your situation, it depends. If you don't need the integer elements to actually be integers (i.e. you aren't using them to index or anything), converting everything to Float64 will be fastest because homogeneous arrays are always faster than heterogeneous.. If you do need the integers to remain integers, Union{Float64, Int} is the best way to do it (it is faster than Real because the compiler can specialize for unions of fewer than, I think, 4 types).

[amrods]

I see, thank you for the detailed explanation.