Skip cache db locking (optionally?) in "nextflow log"

[jmuhlich]

New feature

Allow nextflow log session_id to run without locking the cache db.

Usage scenario

When I help other users with failed Nextflow runs on our institution's HPC cluster, it is often helpful to run nextflow log session_id on a particular session to get deeper information on the individual processes. I do not have root access on the cluster, so this is not possible unless they grant me (and usually everyone else via the "other" permission bit) write access to their cache db directory so leveldb can take a lock, which is intrusive and not very safe. I could also replicate their .nextflow directory into another location I do have write access to, but this is cumbersome.

Suggest implementation

Can leveldb perform read-only operations without locking? I would accept undefined behavior if the db is being written to concurrently -- in my use case the runs are no longer active so this shouldn't happen anyway. It would be reasonable to require opting in to this behavior via an extra option to nextflow log.

[jmuhlich]

After checking up on leveldb myself, it doesn't appear a read-only operation mode was ever part of their design. Do I have any other options for my use case?

[bentsherman]

As you say, LevelDB does not support multiple readers. However, we recently added a new cache store that uses remote object storage instead. This cloud cache doesn't use a lock file, so you can perform multiple operations on the same cache db. Just keep in mind that the cache is now located in cloud storage instead of local storage.

To enable the cloud cache, set the NXF_CLOUDCACHE_PATH environment var to the desired cache path:

export NXF_VER=23.08.1-edge
export NXF_CLOUDCACHE_PATH=s3://my-bucket/cache