Large jump in disk space usage at the start of each iteration within a loop

[pmac0451]

Hello, I use the foreach, future, doFuture and future.callr packages to run some large calculations in parallel as part of a scheduled job which runs every night. Recently my job has routinely failed with an error saying the agent has run out of disk space (NB hard drive space, not RAM!). I've added more logging to the process and I can see from the logging that at the start or end (it's not 100% clear) of each iteration within a loop there is indeed a big jump in disk space usage. If it makes it to the end of all the iterations without being killed then the space is released and it can continue, but if there are too many then it runs out and the process is killed. I haven't made any changes to my code in a long time and I don't think the size of the data involved in each parallel loop has increased much recently either. There is nothing in the code that I'm knowingly running within each iteration that should leave behind any large files on disk.

Could any of the recent changes to any of the above packages result in data being written to a temporary file between each iteration, and not deleted until the end of the entire loop? And if that's intentional, is there anything I can do to stop this?

From looking at my logs I've noticed that the jump in disk space usage between each iteration is roughly the same each time (c500mb), even though the size of each iteration can vary quite a lot. From this I wonder if what it's storing is the (common) inputs to each iteration, which is roughly the right size? Is this something to do with the ability to resume futures in the even of an error and is there a way to turn this off?

The relevant parts of my script are as follows - I don't think the rest is relevant but I can provide more detail if required.

library(foreach)
library(doParallel)
library(doFuture)
library(future.callr)

largeInputDataList <- # large list of input data

future::plan(callr, workers = 4) # 4 available cores
options(doFuture.rng.onMisuse = "ignore")

result <-
  foreach(
    id = ids, # vector of 55 elements
    .errorhandling = "pass",
    .options.future = list(chunk.size = 1) # For dynamic load balancing
  ) %dofuture% {
    # Code that ultimately returns a small data frame of a handful of rows,
    # but in the meantime does some large and complex calculations using largeInputDataList
  }

[HenrikBengtsson]

Hello, I use the foreach, future, doFuture and future.callr packages ...
Could any of the recent changes to any of the above packages result in data being written to a temporary file between each iteration, and not deleted until the end of the entire loop? And if that's intentional, is there anything I can do to stop this?

No, nothing in those package save anything to disk.

There's very little to go by in your mock-up code, so unfortunately I cannot say anything more useful.

[HenrikBengtsson]

Recently my job has routinely failed with an error saying the agent has run out of disk space (NB hard drive space, not RAM!).

To confirm or rule out your hypothesis that one or more packages you're using causes this, you could roll back packages to the versions available around the date you think this started to happen. The https://packagemanager.posit.co/client/#/repos/cran/setup?snapshot=2025-01-01 service can be helpful here.

[pmac0451]

Thanks for getting back to me. I'll do more investigation - I'll try running the same loop using a bog standard for loop and seeing if I can see the same issue, in which case clearly my code is doing something I'm unaware of. If the standard for loop doesn't have the same problem then is it possible that behind the scenes it's writing data to a swap file or something like that on the hard drive to save RAM? I'll let you know how I get on anyway. Thanks again.

[pmac0451]

Hello again, I've now done three otherwise identical test runs:

1. one using the above parallel structure with 4 workers
2. one with 1 worker
3. one using a simple for loop with no parallelisation.

As you can see from the chart below there is very clearly a large jump in disk space usage at the beginning of each iteration within the parallel loops, and no such jump when using a for loop. The initial jump in usage is exactly 4 times bigger in the 4-worker run vs the 1-worker run, and I can see in the 1-worker run (where things are sequential so it's easier to see) that the jump in usage at each iteration is almost exactly the same at about 510Mb. As I mentioned above, I think 510Mb is roughly the size of the common data being used by each loop, so it seems like it's storing that data at the start of each iteration and not clearing it out at the end before the next iteration.

I use the same parallel structure (albeit without the .options.future = list(chunk.size = 1) option) elsewhere in my pipeline and I've plotted the disk usage in the same way and it uses effectively none at all. This run also has common data but it's much much smaller than 510Mb (around 16Mb I believe).

I wonder if that means:

1. The chunk.size option is causing a different behaviour (this seems unlikely)
2. When the common data to each future is above a certain size it is being stored in temp files (or similar) rather than in memory? Could it be something outside of the futureverse packages that's deciding to do this? It wouldn't be a problem if it cleared up after itself after each iteration, but it's only clearing up at the end of the entire parallel loop (if it gets that far) which is what's ultimately causing my pipeline failures when all the hard disk space is gone.

Or of course it could be something else entirely!

I'll keep digging in the meantime.

[pmac0451]

Hello again, I tried another couple of runs where I removed all the code from each iteration so that all it does in each one is log the disk space. With this I get the same pattern as above i.e. I still see a steady increase in disk space usage at what looks like the start of each iteration. This time though, it obviously knows that I'm not using any data so the increase is very small (c450kB) so the run completes without any problems. This does suggest that it's not the size of the data being used in each iteration that's the problem though, it's still writing data to disk.

[pmac0451]

Last update today - I've tried listing all the files and folders in the /tmp directory of the machine running the pipeline and it looks like there are a whole load of files in there which correspond to each R instance created by callr. There are dozens in the format /tmp/callr-env-12fa062299bf2d, and fewer with the same format except instead of env they have fun, scr, sev, spr, uev or upr. Does that give us a clue to why/where these are being created and possibly how to clear them out? Looks like possibly here in the callr package that these files are being created? Is there a way to make sure they are deleted when an iteration is over?

Edit: the largest such files are actually the fun versions which are each around the 500Mb mark that is causing the problem. These look like they are created in the callr package here.

filepath                                        size      isdir  mode
/tmp/callr-fun-223be9646fab31                   521587671 FALSE  644
/tmp/callr-fun-223be960aa8e1                    521587668 FALSE  644
/tmp/callr-fun-184b5c123d6a9f                   509213921 FALSE  644
/tmp/callr-fun-184b5c56bfa7d6                   509213804 FALSE  644
/tmp/callr-fun-184b5c7df24028                   509213804 FALSE  644
/tmp/callr-fun-223be95958460b                   471453696 FALSE  644
/tmp/callr-fun-1c1adc46e5b31f                   451751936 FALSE  644
/tmp/callr-fun-204f6ce1cf624                    421748736 FALSE  644
/tmp/callr-fun-1f55201eb5dd1c                   418709504 FALSE  644
/tmp/callr-fun-2146bd792303e4                    90505216 FALSE  644

[pmac0451]

Groping towards a solution - I think it's to do with the temp directory in Ubuntu defaulting to /tmp instead of a per-session directory, and it not being permitted to delete files from /tmp afterwards (which is perhaps due to the settings of the machine I'm running on). From this post it might be possible to change this if we can set one of the temp dir env variables before each callr future is launched....?

I have managed to get it to use a per-session temp file (by simply using the -t option when I run the script via littler) but it's not made any difference to memory usage - I'm now wondering if it's a local setting at our end which limits the ability to delete files from /tmp and its subfolders...

EDIT: I think this is a red herring - I don't think the issue is related to permissions as I can delete files in other /tmp directories - see the crude solution I've ended up with below.

[pmac0451]

I've come up with a crude solution to this issue. At the beginning of each parallel iteration I manually delete all the callr-fun- files in the "outside" temp directory, i.e. the temp directory of the R session which calls the parallel loop. The code updates look like this:

1. Before parallel loop
   mainTempDir <- tempdir()

2. Within each parallel iteration (right at the start)

filesInMainTempToDelete <-
  grep(
    pattern = "callr-fun-",
    x = list.files(mainTempDir, full.names = TRUE),
    value = TRUE
  )

if (length(filesInMainTempToDelete) > 0L) {
  message(
    "Delete files:\n",
    paste(filesInMainTempToDelete, collapse = "\n"),
    "\n"
  )
  unlink(filesInMainTempToDelete)
}

This has solved my problem by avoiding the build up of the large callr-fun- files that are only cleared out at the end of the entire parallel loop. The chart of disk space usage now looks like this - it jumps up by about 500Mb around the start of each iteration but it never gets above about 2Gb as the offending callr-fun- files are almost immediately deleted:

Deleting the files seems to make no difference to the outcome of my runs.

I feel like this shouldn't be necessary and that maybe callr or future.callr should clear out these files at the end of each iteration? Or maybe there is a setting at my end preventing this, although I can't think what it would be!