BoundsError in MemPool.sra_migrate! when calling DTable(::DataFrame)

[StevenWhitaker]

I used the MWE in #60 (comment), but with enable_disk_caching!(50, 2^10 * 20) (my computer has 32 GB RAM) inserted after loading packages. On the 38th call to main I get a BoundsError.

julia> include("mwe.jl")

julia> for i = 1:100 (i % 10 == 0 && @show(i)); main() end
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
i = 10
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
i = 20
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
i = 30
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
      From worker 2:    ┌ Info:
      From worker 2:    └   length(dt3) = 233930
ERROR: On worker 2:
BoundsError: attempt to access 185-element Vector{Int64} at index [187]
Stacktrace:
  [1] getindex
    @ ./essentials.jl:13
  [2] #107
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:920
  [3] iterate
    @ ./generator.jl:47 [inlined]
  [4] #105
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:925
  [5] with_lock
    @ ~/.julia/packages/MemPool/l9nLj/src/lock.jl:80
  [6] #sra_migrate!#103
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:849
  [7] sra_migrate!
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:826 [inlined]
  [8] write_to_device!
    @ ~/.julia/packages/MemPool/l9nLj/src/storage.jl:817
  [9] #poolset#160
    @ ~/.julia/packages/MemPool/l9nLj/src/datastore.jl:386
 [10] #tochunk#139
    @ ~/.julia/packages/Dagger/M13n0/src/chunks.jl:267
 [11] tochunk (repeats 2 times)
    @ ~/.julia/packages/Dagger/M13n0/src/chunks.jl:259 [inlined]
 [12] #DTable#1
    @ ~/.julia/packages/DTables/BjdY2/src/table/dtable.jl:38
 [13] DTable
    @ ~/.julia/packages/DTables/BjdY2/src/table/dtable.jl:28
 [14] #create_dt_from_cols#9
    @ ~/tmp/mwe.jl:76
 [15] create_dt_from_cols
    @ ~/tmp/mwe.jl:68 [inlined]
 [16] add_value_col!
    @ ~/tmp/mwe.jl:53
 [17] query
    @ ~/tmp/mwe.jl:26
 [18] #invokelatest#2
    @ ./essentials.jl:819 [inlined]
 [19] invokelatest
    @ ./essentials.jl:816
 [20] #110
    @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
 [21] run_work_thunk
    @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
 [22] macro expansion
    @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
 [23] #109
    @ ./task.jl:514
Stacktrace:
 [1] remotecall_fetch(::Function, ::Distributed.Worker; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Distributed ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:465
 [2] remotecall_fetch(::Function, ::Distributed.Worker)
   @ Distributed ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:454
 [3] #remotecall_fetch#162
   @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492 [inlined]
 [4] remotecall_fetch
   @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492 [inlined]
 [5] main
   @ ~/tmp/mwe.jl:19 [inlined]
 [6] top-level scope
   @ ./REPL[2]:1

[StevenWhitaker]

I verified I get this same error (not always on the 38th call) in Windows. (The OP was in WSL 2.)

[jpsamaroo]

Whoops, deleteat! in a loop iterating over the same collection is always a bad idea - surprisingly, I didn't catch that when I wrote this! Please give JuliaData/MemPool.jl#73 a try, it should fix it.

[krynju]

I did reproduce it on a different piece of code, but with the fix it wouldn't appear anymore, so looks like this is fixed properly

julia> map(x -> (r=x.a + 1,), d) |> fetch
ERROR: On worker 3:
BoundsError: attempt to access 12-element Vector{Int64} at index [13]
Stacktrace:
  [1] getindex
    @ .\essentials.jl:13
  [2] #107
    @ C:\Users\krynjupc\.julia\packages\MemPool\l9nLj\src\storage.jl:920
  [3] iterate
    @ .\generator.jl:47 [inlined]

[StevenWhitaker]

Looks like it's fixed! Thanks!