Yet another "huge intermediate files" issue

[tamasgal]

I have read through couple of issues (like #452) but have not found any solution to my problem, although I know and see that many people are struggling with similar problems.
Either I am (or we are) doing something wrong or this is not fitting the scope of nextflow, so let me please show you a simplified example and maybe someone has an idea how to elegantly design a workflow for it.

Here is the extremely simplified pipeline with the following processes:

1. GrabDataFromTapeServers - input is a list of remote file locations on a Grid, output is a single 5-20GB file
2. ReconstructEvents - input is the output of GrabDataFromTapeServers (5-20GB file), output is a small 100MB file

I have the file grabbing functionality in a separate process (GrabDataFromTapeServers) because there are different configuration options depending on which batch farm is used and this process is also used in other workflows.

The problem is: we need to launch tens of thousands of jobs and we have at best working space of a few hundred TB, largely depending on the batch farm used. The "big" file is usually only used in one consecutive process and can be thrown away afterwards.

Deleting it does not work, since the following process (in this case ReconstructEvents) only has access to a symlink. I'd rather not hack around to follow the symlink and delete the file, but maybe that's the way to go?

Using scratch true is not a solution either since the output file will be copied to the work/ directory anyways. We use it though to minimise east-west-traffic.

A solution would be integrating the file-grabbing processinto ReconstructEvents and deleting the original file in the same process. The problem is that we need to change a lot of workflows which use GrabDataFromTapeServers and that would be code duplication and additional maintenance overhead.

Is there a recommended solution for such a workflow?

[bentsherman]

If you have read through the previous issues then I assume you came across the proposed solution by GEMmaker for processing RNAseq data at scale. I think this solution would work for you as well, and is probably the best option at this time. Of course, I worked on GEMmaker so I'm biased :)

I didn't come up with the solution, but I recently ported the code to DSL2, which made it much cleaner and easier to understand. Refer to the code here. The two key components of the solution are:

1. beginning of workflow -- samples are tracked via metadata files and moved through stage/process/done directories, which allows you to control how many samples are processed in parallel
2. end of workflow -- intermediate files are fed to a final "clean" process that "deletes" the files without invalidating the cache, basically it deletes the file contents but preserves the file timestamps. the file channel must be combined with "done" signals from downstream channels.

The end result is (1) you can control how much intermediate data is created and (2) retain a partial level of caching. Basically, samples that finish completely will be cached, otherwise they will be re-run from the beginning. In your case I think the boilerplate logic would not be very much compared to GEMmaker, which has multiple branches each with several processes in sequence.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.