Provide a mechanism for using secrets in onComplete and onError

[Midnighter]

New feature

Usage scenario

Secrets can currently either be used in the configuration (but not assigned to params) or accessed in processes; it would be stellar to also be able to access them in the onComplete (and onError) handlers. That's the typical place to send pipeline reports etc. which will need access to authentication information.

Alternatively, if it was possible to define custom configuration scopes, then I could assign secrets just like to the built-in scopes and use that in my handlers. Maybe this is already possible via plugins?

Suggest implementation

I don't know about the nextflow code itself but the usage could look like either allowing direct usage:

workflow.onComplete {
    sendChat('Pipeline status', secrets.API_TOKEN)
}

or via a custom config scope and handler (possibly defined in a plugin):

nextflow.config:

chat {
    authentication = secrets.API_TOKEN
}

workflow.onComplete = {
    sendChat('Pipeline status')
}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Midnighter]

I learnt in the meantime that I could do this with a plugin that consumes its own config. However, writing a plugin is a fair amount of work so a simpler mechanism may still be worth it.

[mateuszpawlik]

I second this issue. Any new ideas? In my use case I want to report the pipeline progress from every process and once the pipeline completes or fails. I use GitLab API via python-gitlab to create merge request notes with report messages. I was quite happy finding the secrets feature of Nextflow. Sending API requests from processes works with the secret directive. However, I could not find any way to use secrets in workflow handlers. I think I will need to return to using a config file.

[mateuszpawlik]

👍