pipeline failing with cryptic error

[lucas-johnson]

I'm running a large pipeline with targets and crew handling parallelization on my local machine. It's failing with a cryptic error message (see below). I was expecting the pipeline to finish successfully or at least provide more information with this error. I'm suspecting this error might OOM related, as I've had OOM issues with this pipeline previously when using future instead of crew, but those issues manifested with different error messages. Can somebody diagnose this error message?

▶ end pipeline [29.79 minutes]
Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: invalid daemons: structure(5L, class = "errorValue")
 dispatcher running at pid 881616 with status "disk_sleep". Connection issue?

I have posted a similar question on SO

[wlandau]

Seems related to ropensci/targets#1101. It might fix itself if you install the latest CRAN versions of nanonext, mirai, and crew. @shikokuchuo has also mentioned that this sort of error might happen if there are enough busy threads or processes to max out the CPUs. (I am having trouble finding that thread.) The improvements mentioned in shikokuchuo/mirai#63 (reply in thread) (development versions only right now) might also help. Other than that, it's hard to say what is happening without a minimal reprex.

[lucas-johnson]

Back to getting hit with 'errorValue' int 5 even with transient memory. No OOM killer invocation in the logs.

[shikokuchuo]

Hi @lucas-johnson thanks for reporting. Can I suggest a 2-point plan to get to the bottom of this:

1. Use the latest dev versions of all packages - including mirai 0.11.0.9002. I've added some measures that shouldn't really be necessary, but might help in this instance.

2. If you continue getting the same error, move back to CRAN versions of all packages - nanonext, mirai and crew. This will help rule out the possibility that a recent change is causing this.

If you end up still getting the error, it may be that it really is a timeout and dispatcher is stuck elsewhere, thus unable to respond in time. The send/receives are all asynchronous, but moving the data across buffers may be the issue. In this case, we'll have to figure out a way of handling this in our packages.

[lucas-johnson]

Sorry for the long term radio silence – the error persists for 1 and 2 above ^.

I also was able to reproduce the behavior from the mirai-only example in @baslat's thread below. Perhaps I should also try the crew-only example? Would that be helpful?

[shikokuchuo]

Hi @lucas-johnson, as these packages are in active development trying them at random points in time isn’t guaranteed to work. You are welcome with nanonext build 9005, mirai build 9010 and crew 0.6.0 to see if they solve the issue. If not, the actual error output would be helpful. Otherwise you will probably have to check back in once new CRAN releases have been made, unless you can pinpoint it down to a reproducible example.

[lucas-johnson]

I understand these packages are in active development, and I appreciate all the awesome work you and @wlandau have done. I'm sorry I haven't been more helpful given that I haven't produced a reprex.

I've rolled back to using future instead of crew for now, as I have some deadlines approaching.

I'm going to spend some more time in the interim to see if I can produce a reprex and will report back if/when I do. Thank you!

[baslat]

I'm getting a similar error, with {crew}. I'll try changing the memory setting.

  Last error: invalid daemons: structure(5L, class = "errorValue")
 dispatcher running at pid 37098 with status "running". Connection issue?

[shikokuchuo]

@wlandau if you're looking to eliminate this error, I can offer the following analysis. Please disregard my previous comments as much has changed in the packages since.

This in my view is a real timeout. From the time the status() request is sent, over 10s (yes, lengthened) has elapsed. I have come to the conclusion that this is perfectly possible when returning a large enough object - could be just the serialization alone - in which case dispatcher may be blocking on that task and not be able to respond in time. To fix this, you can either bring back re-tries (can condition on errorValue being 5), or else a way for the user to resume such an interruption. The actual tasks should still be ongoing - in mirai, this is only a return 'errorValue' and not an actual error (does not halt execution). Hope this is clear.

[wlandau]

That makes a lot of sense. As of 1f4db33, brought back retries, and the seconds_timeout argument of crew_controller_local() now controls the overall timeout at the crew level. @baslat, you could try this and increase seconds_timeout to something large enough to cover serialization.

For large data workflows in targets, I am recommending that S3 buckets and/or local storage to manage data, which means crew and mirai only have to send instructions to parallel workers, not large data objects. You can make this happen with storage = "worker" and retrieval = "worker" in targets. Please have a look at https://books.ropensci.org/targets/performance.html#worker-storage.

[baslat]

I updated crew and mirai, and tried tar_make() with seconds_timeout = 1200, and I get a segfault error. I then updated nanonext too, and got this error:

Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: 'errorValue' int 11 | Incorrect state
 The mirai dispatcher is not running. Please call the start() method of the controller (e.g. your_controller$start() before using methods like push(), collect(), and scale(). If you already did, then the dispatcher may have exited too early.

I will check out the link you sent and experiment with those settings, thanks.

[shikokuchuo]

Just to note that 'errorValue' int 11 | Incorrect state is the 'correct' error when dispatcher in no longer running due to OOM etc. This is making sense now - think the potential issues were all fixed back in mirai 0.11.3.

[wlandau]

Yes, this seems like a likely out-of-memory error on the dispatcher. @baslat, for your purposes, storage = "worker" and retrieval = "worker" in targets (in tar_target() or tar_option_set()) will avoid sending all that data to the dispatcher and should solve the issue for you. Please also consider transient memory and garbage collection: https://books.ropensci.org/targets/performance.html.