Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to cleanup datastore after 0 seconds Forcibly evicting all entries #445

Closed
StevenWhitaker opened this issue Oct 26, 2023 · 1 comment · Fixed by JuliaData/MemPool.jl#69
Closed

Failed to cleanup datastore after 0 seconds Forcibly evicting all entries #445

StevenWhitaker opened this issue Oct 26, 2023 · 1 comment · Fixed by JuliaData/MemPool.jl#69
Labels
bug storage upstream

Comments

@StevenWhitaker
Copy link

I'm not sure if the following behavior is caused by Dagger.jl or not, but I'm wondering if it's related to the issues I've been seeing with my code and using Dagger.jl with Distributed.jl (see, e.g., #438).

I have code that I run that uses DTables.jl, but it frequently (but not always) does not run to completion. I tried to distill the essence of the code to create the MWE in #438, but that MWE doesn't capture everything I'm seeing with my actual code. (In fact, that MWE might be working perfectly now.) Currently, my code hangs frequently, and I'm not sure why. The CPU utilization drops to 0 when it hangs, so it's almost as if tasks aren't getting scheduled to run (but I really have no idea if that's the case).

Anyway, I also noticed that every time I exit Julia after running my code (as long as I don't have to kill the Julia processes from the terminal, which sometimes I have to do when it hangs because ctrl-c won't work), I get quite a large dump of debugging messages about evicting datastore entries. See an example after the questions below. (Also note that Dagger.jl is my only dependency that uses MemPool.jl.)

Questions:

  1. @jpsamaroo do you have any insight as to why the datastore fails to clean? Does it mean my code is doing something wrong, like somehow holding onto objects longer than it should? Or could it be another Dagger.jl issue?
  2. Could it be that, when running a task, if the task gets interrupted (e.g., to do garbage collection) there is an issue with rescheduling it? (Do tasks get interrupted for garbage collection or other reasons when in a single-threaded, multi-process environment?)
  3. (Unrelated) There are a couple of places in my code with @sync and @async. Would these cause any issues with Dagger.jl? I've tried removing them, but I haven't noticed any difference in behavior.
┌ 2023-10-26T11:14:16.892 ] pid: 8730 proc: 1 Debug:  Failed to cleanup datastore after 0 seconds
└ Forcibly evicting all entries
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 56 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 60 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 220 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 308 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 67 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 73 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 115 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 112 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 185 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 86 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.899 ] pid: 8730 proc: 1 Debug:  Evicting ref 168 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 263 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 183 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 177 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 75 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 111 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 264 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 68 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 82 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 130 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 125 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 77 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 172 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 71 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 66 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 280 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 59 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.900 ] pid: 8730 proc: 1 Debug:  Evicting ref 127 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 116 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 79 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 279 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 195 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 141 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 278 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 135 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 138 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 107 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 46 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 276 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 295 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 57 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 152 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 170 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 129 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 250 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.901 ] pid: 8730 proc: 1 Debug:  Evicting ref 78 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 133 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 72 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 258 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 184 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 1 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 137 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 154 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 288 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 270 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 299 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 40 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 283 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 165 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 142 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 55 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 265 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 136 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.902 ] pid: 8730 proc: 1 Debug:  Evicting ref 117 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 45 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 145 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 282 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 275 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 158 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 176 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 148 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 92 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 118 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 162 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 84 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 95 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 292 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 93 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.903 ] pid: 8730 proc: 1 Debug:  Evicting ref 296 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 304 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 147 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 157 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 287 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 266 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 146 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 74 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 61 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 303 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 159 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 193 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 105 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 285 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 271 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 166 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 89 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 51 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 274 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 143 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 48 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.904 ] pid: 8730 proc: 1 Debug:  Evicting ref 97 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 284 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 134 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 272 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 182 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 164 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 153 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 186 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 253 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 64 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 267 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 90 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 139 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 52 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 179 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 300 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 43 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 69 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.905 ] pid: 8730 proc: 1 Debug:  Evicting ref 171 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 302 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 85 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 119 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 126 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 156 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 124 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 307 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 261 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 144 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 273 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 257 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 290 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 81 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 187 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 0 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 189 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.906 ] pid: 8730 proc: 1 Debug:  Evicting ref 294 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 161 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 88 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 120 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 260 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 297 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 37 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 83 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 190 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 121 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 281 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 174 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 269 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 123 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 305 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 268 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.907 ] pid: 8730 proc: 1 Debug:  Evicting ref 196 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 262 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 151 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 54 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 63 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 191 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 91 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 62 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 150 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 122 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 58 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 173 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 256 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 188 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 277 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 76 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 50 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 194 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.908 ] pid: 8730 proc: 1 Debug:  Evicting ref 167 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 42 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 87 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 132 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 140 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 301 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 169 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 180 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 255 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 160 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 289 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 49 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 291 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 94 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 128 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 259 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 70 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 38 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.909 ] pid: 8730 proc: 1 Debug:  Evicting ref 163 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.910 ] pid: 8730 proc: 1 Debug:  Evicting ref 131 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.910 ] pid: 8730 proc: 1 Debug:  Evicting ref 192 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.910 ] pid: 8730 proc: 1 Debug:  Evicting ref 53 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.910 ] pid: 8730 proc: 1 Debug:  Evicting ref 47 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.910 ] pid: 8730 proc: 1 Debug:  Evicting ref 175 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.910 ] pid: 8730 proc: 1 Debug:  Evicting ref 286 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.911 ] pid: 8730 proc: 1 Debug:  Evicting ref 178 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.911 ] pid: 8730 proc: 1 Debug:  Evicting ref 96 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.912 ] pid: 8730 proc: 1 Debug:  Evicting ref 306 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.912 ] pid: 8730 proc: 1 Debug:  Evicting ref 149 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.912 ] pid: 8730 proc: 1 Debug:  Evicting ref 155 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.912 ] pid: 8730 proc: 1 Debug:  Evicting ref 181 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.912 ] pid: 8730 proc: 1 Debug:  Evicting ref 65 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.913 ] pid: 8730 proc: 1 Debug:  Evicting ref 293 with device MemPool.CPURAMDevice()
[ 2023-10-26T11:14:16.913 ] pid: 8730 proc: 1 Debug:  Evicting ref 298 with device MemPool.CPURAMDevice()
      From worker 4:    ┌ 2023-10-26T11:14:17.563 ] pid: 8752 proc: 4 Debug:  Failed to cleanup datastore after 0 seconds
      From worker 4:    └ Forcibly evicting all entries
      From worker 3:    ┌ 2023-10-26T11:14:17.565 ] pid: 8751 proc: 3 Debug:  Failed to cleanup datastore after 0 seconds
      From worker 3:    └ Forcibly evicting all entries
      From worker 3:    [ 2023-10-26T11:14:17.585 ] pid: 8751 proc: 3 Debug:  Evicting ref 5 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.585 ] pid: 8751 proc: 3 Debug:  Evicting ref 16 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.585 ] pid: 8752 proc: 4 Debug:  Evicting ref 5 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.585 ] pid: 8751 proc: 3 Debug:  Evicting ref 20 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 12 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 24 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.586 ] pid: 8752 proc: 4 Debug:  Evicting ref 35 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 8 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 17 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.586 ] pid: 8752 proc: 4 Debug:  Evicting ref 30 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 1 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 19 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 0 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 22 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 6 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 23 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.586 ] pid: 8751 proc: 3 Debug:  Evicting ref 11 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 9 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 14 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 32 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 3 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 7 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 4 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 13 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 15 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 2 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 10 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 18 with device MemPool.CPURAMDevice()
      From worker 3:    [ 2023-10-26T11:14:17.587 ] pid: 8751 proc: 3 Debug:  Evicting ref 21 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 6 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 45 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 4 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 13 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 52 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 12 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 28 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 23 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 41 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 43 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 11 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 36 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 39 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 7 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 25 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 34 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 50 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 2 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 10 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.587 ] pid: 8752 proc: 4 Debug:  Evicting ref 18 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 26 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 27 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 42 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 16 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 20 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 19 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 0 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 49 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 44 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 9 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 31 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 29 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 46 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 21 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 38 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 24 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 8 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 17 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 37 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 1 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 53 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 22 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 47 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 14 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 3 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 51 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 33 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.588 ] pid: 8752 proc: 4 Debug:  Evicting ref 40 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.589 ] pid: 8752 proc: 4 Debug:  Evicting ref 48 with device MemPool.CPURAMDevice()
      From worker 4:    [ 2023-10-26T11:14:17.589 ] pid: 8752 proc: 4 Debug:  Evicting ref 15 with device MemPool.CPURAMDevice()
      From worker 2:    ┌ 2023-10-26T11:14:17.677 ] pid: 8748 proc: 2 Debug:  Failed to cleanup datastore after 0 seconds
      From worker 2:    └ Forcibly evicting all entries
[ 2023-10-26T11:14:18.045 ] pid: 8730 proc: 1 Warning:  Worker 3 died, rescheduling work
      From worker 5:    ┌ 2023-10-26T11:14:17.835 ] pid: 8754 proc: 5 Debug:  Failed to cleanup datastore after 0 seconds
[ 2023-10-26T11:14:18.122 ] pid: 8730 proc: 1 Warning:  Worker 4 died, rescheduling work
      From worker 5:    └ Forcibly evicting all entries
      From worker 5:    [ 2023-10-26T11:14:17.878 ] pid: 8754 proc: 5 Debug:  Evicting ref 5 with device MemPool.CPURAMDevice()[ 2023-10-26T11:14:18.357 ] pid: 8730 proc: 1 Warning:  Worker 5 died, rescheduling work

      From worker 5:    ┌ 2023-10-26T11:14:18.483 ] pid: 8754 proc: 5 Error:  Fatal error on process 5
      From worker 5:    │   exception =
      From worker 5:    │    attempt to send to unknown socket
      From worker 5:    │    Stacktrace:
      From worker 5:    │     [1] error(s::String)
      From worker 5:    │       @ Base ./error.jl:35
      From worker 5:    │     [2] send_msg_unknown(s::Sockets.TCPSocket, header::Distributed.MsgHeader, msg::Distributed.ResultMsg)
      From worker 5:    │       @ Distributed ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/messages.jl:99
      From worker 5:    │     [3] send_msg_now(s::Sockets.TCPSocket, header::Distributed.MsgHeader, msg::Distributed.ResultMsg)
      From worker 5:    │       @ Distributed ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/messages.jl:115
      From worker 5:    │     [4] deliver_result(sock::Sockets.TCPSocket, msg::Symbol, oid::Distributed.RRID, value::Nothing)
      From worker 5:    │       @ Distributed ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:102
      From worker 5:    │     [5] macro expansion
      From worker 5:    │       @ ~/programs/julia/julia-1.9.3/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:302 [inlined]
      From worker 5:    │     [6] (::Distributed.var"#113#115"{Distributed.CallWaitMsg, Distributed.MsgHeader, Sockets.TCPSocket})()
      From worker 5:    └       @ Distributed ./task.jl:514
      From worker 5:    [ 2023-10-26T11:14:18.487 ] pid: 8754 proc: 5 Debug:  Evicting ref 35 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 55 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 110 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 60 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 30 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 32 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 6 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 67 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 45 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 73 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 112 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 64 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 90 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 4 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 13 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 54 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 63 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 86 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 104 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 91 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 62 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 58 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 52 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 12 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 28 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 75 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 23 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 111 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 92 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 41 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 43 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 11 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 36 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 68 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 69 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 98 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 82 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 85 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 39 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 84 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 77 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 7 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 25 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.497 ] pid: 8754 proc: 5 Debug:  Evicting ref 95 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 108 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 71 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 66 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 76 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 103 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 34 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 50 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 59 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 93 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 2 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 10 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 27 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 26 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 18 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 42 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 87 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 100 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 79 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 16 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 20 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 81 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 19 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 0 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 49 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 44 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 9 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 31 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 74 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 106 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 109 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 61 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 29 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 94 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 107 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 46 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 102 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 70 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 21 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 38 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 88 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 101 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.498 ] pid: 8754 proc: 5 Debug:  Evicting ref 78 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 105 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 72 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 24 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 8 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 17 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 37 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 1 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 53 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 22 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 47 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 83 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 99 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 89 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 14 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 3 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 80 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 96 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 51 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 33 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 40 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 48 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 15 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 65 with device MemPool.CPURAMDevice()
      From worker 5:    [ 2023-10-26T11:14:18.499 ] pid: 8754 proc: 5 Debug:  Evicting ref 97 with device MemPool.CPURAMDevice()
┌ 2023-10-26T11:14:22.138 ] pid: 8730 proc: 1 Warning:  Forcibly interrupting busy workers
└   exception = rmprocs: pids [2] not terminated after 5.0 seconds.
[ 2023-10-26T11:14:22.146 ] pid: 8730 proc: 1 Warning:  rmprocs: process 1 not removed
@jpsamaroo
Copy link
Member

@jpsamaroo do you have any insight as to why the datastore fails to clean? Does it mean my code is doing something wrong, like somehow holding onto objects longer than it should? Or could it be another Dagger.jl issue?

It could be either your code, DTables, or Dagger causing this, but it may be that it's not actually a problem but just looks like one from the intimidating warnings. Since these are all pure in-memory objects (that's what MemPool.CPURAMDevice represents), I can workaround this in MemPool by just ignoring such objects in this eviction code, because they'll be destroyed once Julia exits anyway. I'll put this together momentarily.

Could it be that, when running a task, if the task gets interrupted (e.g., to do garbage collection) there is an issue with rescheduling it? (Do tasks get interrupted for garbage collection or other reasons when in a single-threaded, multi-process environment?)

The only time I'd expect there to be an issue with handling an interruption is if you send a Ctrl-C/SIGINT to a worker - Julia doesn't handle this well at all right now, and it will cause random things to break or hang. A solution is in the works, but it will take some time (and will only show up in a future Julia version).

(Unrelated) There are a couple of places in my code with @sync and @async. Would these cause any issues with Dagger.jl? I've tried removing them, but I haven't noticed any difference in behavior.

They shouldn't cause any problems, I use @sync and Threads.@spawn together frequently when working with Dagger.

@jpsamaroo jpsamaroo mentioned this issue Oct 27, 2023
Merged
@StevenWhitaker StevenWhitaker mentioned this issue Nov 12, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug storage upstream
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

exit_hook: Ignore CPURAMDevice JuliaData/MemPool.jl
2 participants
@jpsamaroo @StevenWhitaker