Optional minimal file retention mode

[NGoetz]

Users should have the option to only preserve/generate the minimum amount of intermediate files. By default we keep a lot of files, in order to keep track of the process, but an experienced user should be able to use a more lustre friendly execution mode.

[AxelKrypton]

I am not sure it is a good idea or at least that it is worth the effort. 😅 How many files do we store that you would get rid of in this new mode? Could you make a folder tree as example and somehow mark what is kind of superfluous? 🤔

[NGoetz]

Everything except the final particle list from the Afterburner is eventually superflous.
The user cannot at the moment remove files once unneeded, as the workflow is one continous process.

[AxelKrypton]

Everything except the final particle list from the Afterburner is eventually superflous.

I believe this is way more intricate that it might sound. We offer full flexibility, so we cannot even assume the Afterburner is the last stage run by the user. We could define the "result" for each stage and only keep that for the last stage run.

The user cannot at the moment remove files once unneeded, as the workflow is one continous process.

This sentence makes me guess that you would like this feature to happen on the way internally to the sampler. Something like: Once I set-up and I am about running a given phase, I can delete everything about previous phases that is not input for what I am about to run. And once I did the last one, I remove everything but the result. Would this still be fine in case of failure? What about reproducibility? In that mode we would need to store the full hybrid handler config file as part of "the result". 🤔

[NGoetz]

Would this still be fine in case of failure? What about reproducibility? In that mode we would need to store the full hybrid handler config file as part of "the result".

I agree. In case of failure, we usually know that the failure happened in the phase currently running, as this setup is anyway rather for well-tested production. If a stage successfully finished, the user should have the functionality to remove its files.