Error when combining RasterSeries along a new dim

[tcarion]

Hi!

I get an error when I try to combine multiple netCDF files along a new custom dimension (see this thread for more insight).
Here's what I'm trying to do:

using Rasters
files = ["member0.nc", "member1.nc", "member2.nc"]
mydim = Dim{:member}(1:length(files))
series = RasterSeries(files, mydim; child = Raster, name = :spec001_mr)
combined_raster = Rasters.combine(series, Dim{:member})

You can see the error stack more below.

Please note that it works when combining from only two files (e.g. files = ["member0.nc", "member1.nc"]).

I attached the MWE script file and the .nc files that I'm using:
mwe.zip

Error stack:

ERROR: LoadError: BoundsError: attempt to access NTuple{7, Int64} at index [8]
Stacktrace:
  [1] getindex
    @ ./tuple.jl:29 [inlined]
  [2] #75
    @ ~/.julia/packages/DiskArrays/tD6J0/src/permute_reshape.jl:49 [inlined]
  [3] map
    @ ./tuple.jl:223 [inlined]
  [4] map (repeats 3 times)
    @ ./tuple.jl:224 [inlined]
  [5] reshape_disk(parent::Rasters.FileArray{Rasters.NCDfile, Float32, 6, Symbol, DiskArrays.GridChunks{6}, DiskArrays.Unchunked}, dims::NTuple{7, Int64})
    @ DiskArrays ~/.julia/packages/DiskArrays/tD6J0/src/permute_reshape.jl:46
  [6] _reshape
    @ ~/.julia/packages/DiskArrays/tD6J0/src/permute_reshape.jl:94 [inlined]
  [7] reshape
    @ ./reshapedarray.jl:112 [inlined]
  [8] reshape
    @ ./reshapedarray.jl:111 [inlined]
  [9] reshape
    @ ./reshapedarray.jl:141 [inlined]
 [10] _maybereshape
    @ ~/.julia/packages/Rasters/CrXfm/src/methods.jl:1657 [inlined]
 [11] #380
    @ ~/.julia/packages/Rasters/CrXfm/src/methods.jl:1650 [inlined]
 [12] (::Base.BottomRF{Rasters.var"#380#381"{Dim{:member, DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, UnionAll}})(acc::Raster{Float32, 7, Tuple{X{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, Y{Colon}}}, Dim{:height, DimensionalData.Dimensions.LookupArrays.Sampled{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Ti{DimensionalData.Dimensions.LookupArrays.Sampled{Dates.DateTime, Vector{Dates.DateTime}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Irregular{Tuple{Nothing, Nothing}}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Dim{:pointspec, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:nageclass, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:member, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}}, Tuple{}, Array{Float32, 7}, Symbol, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, Missing}, x::Raster{Float32, 6, Tuple{X{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, Y{Colon}}}, Dim{:height, DimensionalData.Dimensions.LookupArrays.Sampled{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Ti{DimensionalData.Dimensions.LookupArrays.Sampled{Dates.DateTime, Vector{Dates.DateTime}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Irregular{Tuple{Nothing, Nothing}}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Dim{:pointspec, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:nageclass, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}}, Tuple{}, Rasters.FileArray{Rasters.NCDfile, Float32, 6, Symbol, DiskArrays.GridChunks{6}, DiskArrays.Unchunked}, Symbol, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, Missing})
    @ Base ./reduce.jl:81
 [13] _foldl_impl(op::Base.BottomRF{Rasters.var"#380#381"{Dim{:member, DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, UnionAll}}, init::Base._InitialValue, itr::RasterSeries{Raster{Float32, 6, Tuple{X{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, Y{Colon}}}, Dim{:height, DimensionalData.Dimensions.LookupArrays.Sampled{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Ti{DimensionalData.Dimensions.LookupArrays.Sampled{Dates.DateTime, Vector{Dates.DateTime}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Irregular{Tuple{Nothing, Nothing}}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Dim{:pointspec, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:nageclass, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}}, Tuple{}, Rasters.FileArray{Rasters.NCDfile, Float32, 6, Symbol, DiskArrays.GridChunks{6}, DiskArrays.Unchunked}, Symbol, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, Missing}, 1, Tuple{Dim{:member, DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Vector{Raster{Float32, 6, Tuple{X{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, Y{Colon}}}, Dim{:height, DimensionalData.Dimensions.LookupArrays.Sampled{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Ti{DimensionalData.Dimensions.LookupArrays.Sampled{Dates.DateTime, Vector{Dates.DateTime}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Irregular{Tuple{Nothing, Nothing}}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Dim{:pointspec, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:nageclass, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}}, Tuple{}, Rasters.FileArray{Rasters.NCDfile, Float32, 6, Symbol, DiskArrays.GridChunks{6}, DiskArrays.Unchunked}, Symbol, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, Missing}}})
    @ Base ./reduce.jl:62
 [14] foldl_impl
    @ ./reduce.jl:48 [inlined]
 [15] mapfoldl_impl
    @ ./reduce.jl:44 [inlined]
 [16] #mapfoldl#244
    @ ./reduce.jl:162 [inlined]
 [17] mapfoldl
    @ ./reduce.jl:162 [inlined]
 [18] #foldl#245
    @ ./reduce.jl:180 [inlined]
 [19] foldl
    @ ./reduce.jl:180 [inlined]
 [20] combine(ser::RasterSeries{Raster{Float32, 6, Tuple{X{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, Y{Colon}}}, Dim{:height, DimensionalData.Dimensions.LookupArrays.Sampled{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Ti{DimensionalData.Dimensions.LookupArrays.Sampled{Dates.DateTime, Vector{Dates.DateTime}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Irregular{Tuple{Nothing, Nothing}}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Dim{:pointspec, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:nageclass, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}}, Tuple{}, Rasters.FileArray{Rasters.NCDfile, Float32, 6, Symbol, DiskArrays.GridChunks{6}, DiskArrays.Unchunked}, Symbol, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, Missing}, 1, Tuple{Dim{:member, DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Vector{Raster{Float32, 6, Tuple{X{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, EPSG, EPSG, Y{Colon}}}, Dim{:height, DimensionalData.Dimensions.LookupArrays.Sampled{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Ti{DimensionalData.Dimensions.LookupArrays.Sampled{Dates.DateTime, Vector{Dates.DateTime}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Irregular{Tuple{Nothing, Nothing}}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}}}, Dim{:pointspec, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}, Dim{:nageclass, DimensionalData.Dimensions.LookupArrays.NoLookup{Base.OneTo{Int64}}}}, Tuple{}, Rasters.FileArray{Rasters.NCDfile, Float32, 6, Symbol, DiskArrays.GridChunks{6}, DiskArrays.Unchunked}, Symbol, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDfile, Dict{Symbol, Any}}, Missing}}}, dim::Type{Dim{:member}})
    @ Rasters ~/.julia/packages/Rasters/CrXfm/src/methods.jl:1648
 [21] top-level scope
    @ ~/mwe.jl:9

[rafaqz]

Thanks for the thorough issue report. This should be fixed for the next minor release - there are a few changes currently in progress that make it hard to patch.

[rafaqz]

Ok this is actually a DiskArrays.jl bug in reshape - which is required to cat the arrays together.

If you read the series first to move the disk based data to memory, it works fine:

combined_raster = Rasters.combine(read(series), Dim{:member})

This doesn't happen for 2 file because we never have to cat arrays with different dimensionality together when we only have 2 arrays.

[tcarion]

It works fine indeed, thanks!

[rafaqz]

Should work without that soon too, in the PR this is fixed in DiskArrays.jl