Add cloud cache plugin

[bentsherman]

Close https://github.com/seqeralabs/nf-tower-cloud/issues/4822

Adds a new cache store backed by S3. The cache entry for each task is saved to the following S3 path:

s3://<bucket>/<path>/<session-id>/<task-hash>

It can be enabled by setting NXF_CACHE_STORE to s3 and NXF_CACHE_PATH to the desired S3 path.

Notes:

• Currently included in the nf-amazon plugin, but could be generalized to any object storage since it uses the Path API. Maybe better to make it part of the nextflow module and call it something like ObjectCacheStore.
• Consider exposing config options in addition to environment variables.
• Consider using buckets for the task hash (like the work directory) instead of storing all task entries under the same S3 prefix. I don't know if this will actually help though. Usually Nextflow just needs to try a particular key, not iterate through all keys.
• Consider setting the default cache path based on the work directory (e.g. ${workDir}/../cache

[sonatype-lift]

Sonatype Lift is retiring

Sonatype Lift will be retiring on Sep 12, 2023, with its analysis stopping on Aug 12, 2023. We understand that this news may come as a disappointment, and Sonatype is committed to helping you transition off it seamlessly. If you’d like to retain your data, please export your issues from the web console.
We are extremely grateful and thank you for your support over the years.

📖 Read about the impacts and timeline

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	193a5ec
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/64b819444e4fd4000836791d

[pditommaso]

Using the underlying virtual file system is a good idea. I'm concerned to on 503 slow down error that can be reported when resulting large pipeline because the object storage API will be under pressure.

Also, it would make sense to move this into its own plugin to be activated to enable this feature

[bentsherman]

I'm concerned to on 503 slow down error that can be reported when resulting large pipeline because the object storage API will be under pressure.

Me too. The AWS SDK should be able to handle it with it's built-in adaptive retry, but I don't remember if it's enabled by default.

If that's not enough, we could maintain a local cache and sync with S3 at a controlled rate, at the cost of losing some progress if the workflow is terminated abruptly.

Also, it would make sense to move this into its own plugin to be activated to enable this feature.

I made it enabled with an environment variable NXF_CACHE_STORE='path' so that it doesn't have to reside in a separate plugin. It still uses the default cache store by default.

[bentsherman]

I moved the patch cache to a plugin and added the index file to behave like the default.

One problem with the plugin is that there is no way to enable it with the clean and log commands that use the cache. How do you recommend we instrument those commands?

[pditommaso]

Likely the only way to enable it in those commands is via the NXF_PLUGINS_DEFAULT, which is not very handy. However, the primary need for this plugin is to use it with Tower, therefore I would not care too much

[bentsherman]

Okay, then this PR is ready for review. I tested with rnaseq-nf by running and then resuming the pipeline. Looks like there is still a problem with packing.gradle, but I'm not sure how to fix it.

[pditommaso]

Ok, I've removed the docs from AWS page because it should work for any cloud storage and added a few basic integration tests

[pditommaso]

Ok, those tests works, but it should be added some checks that's effectively resuming the tasks and using the cloud cache. Logging adding some grep in the log file. @bentsherman can you please give it a try? then it can be merged

[pditommaso]

I've removed sonatype-lif, just annoying

[abhi18av]

This is b-e-a-utiful, thanks guys! 🎉

Looking forward to taking this for a spin!

[bentsherman]

@pditommaso is there a way to trigger the cloudcache plugin when the NXF_CLOUDCACHE_PATH variable is set? Currently you have to enable the plugin explicitly, seems like you shouldn't have to do that since it's a core plugin. I'm not sure how to do this through the priority extensions loader.

[pditommaso]

That kind of trick happens here

nextflow/modules/nf-commons/src/main/nextflow/plugin/PluginsFacade.groovy

Lines 351 to 372 in ace32d0

	def specs = parseConf(config)
	if( isSelfContained() && specs ) {
	// custom plugins are not allowed for nextflow self-contained package
	log.warn "Nextflow self-contained distribution allows only core plugins -- User config plugins will be ignored: ${specs.join(',')}"
	return Collections.emptyList()
	}
	if( specs ) {
	log.debug "Plugins declared=$specs"
	}
	if( getPluginsDefault() ){
	final defSpecs = defaultPluginsConf(config)
	specs = mergePluginSpecs(specs, defSpecs)
	log.debug "Plugins default=$defSpecs"
	}
	// add tower plugin when config contains tower options
	if( (Bolts.navigate(config,'tower.enabled') || env.TOWER_ACCESS_TOKEN ) && !specs.find {it.id == 'nf-tower' } ) {
	specs << defaultPlugins.getPlugin('nf-tower')
	}
	if( (Bolts.navigate(config,'wave.enabled') || Bolts.navigate(config,'fusion.enabled')) && !specs.find {it.id == 'nf-wave' } ) {
	specs << defaultPlugins.getPlugin('nf-wave')
	}