Dry run

[bentsherman]

Close #844

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	93af8fe
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/64e778e021aea7000817e30c

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	a1caced
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/6626e2d9fe8f97000a4d0dbd

[bentsherman]

Added basic cache invalidation reporting to this PR. It requires the cache to also load the index file of the previous run, so that when a cache entry can't be found, it can search the previous run for a matching task (i.e. matching process + tag) and report any differences between the two tasks.

Currently it only reports the different hashes and the scripts (if they are different), but could be expanded to include things like:

• container / conda / spack / modules config
• stdin
• input environment vars
• input files (compare metadata or checksum based on cache mode)

The last point requires the input file metadata to be saved to the cache as discussed for #3802 and #3849 . I would be fine to address that in a future iteration, since for now, even just reporting the changed hashes is a great improvement.

Other notes:

• we can use java-diff-utils to print a nice diff of the script text
• comparing the previous run adds a lot of read pressure to the cache store, I'm concerned it might slow down large runs a lot. I would consider enabling this "deep" cache analysis only when the dry run is enabled

[ewels]

• I would consider enabling this "deep" cache analysis only when the dry run is enabled

And / or as a opt-in? I can imagine some people (me) wanting it routinely if not doing large runs. Might be annoying to have to go back and re-do a dry run every time.