[help] Help troubleshooting error: _store_ basic_ios::clear: iostream error

[kendonB]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I tried running a 600-worker crew on a slurm cluster and got this at the end:

✖ error branch simulations_d4efd5b6
▶ end pipeline [36.55 minutes]
Error:
! Error running targets::tar_make()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: _store_ basic_ios::clear: iostream error

I am using:

  format = "qs",
  deployment = "worker", 
  storage = "worker", 
  retrieval = "worker"

and seconds_reporter=15, seconds_meta_append=15.

Any ideas for troubleshooting? I got the error quite quickly (~10 mins in) first try running, then added the seconds_reporter=15, seconds_meta_append=15, it ran fine for about 35 minutes and I interrupted, then it happened again the third time about 35 minutes in.

[kendonB]

Have enlarged the targets to make them fewer and slower. Will see if that works.

[wlandau]

Sounds like a qs issue: https://github.com/search?q=repo%3Atraversc%2Fqs+basic_ios%3A%3Aclear%3A+iostream&type=issues. If you're running out of memory, try memory = "transient" and garbage_collection = TRUE. If not, you can try reducing the problem to a qs-only reprex.

[kendonB]

wasn't reproducible so was likely hitting an IO resource limit. The only thing targets could consider doing is a few retries upon encountering saving errors?