Allow to publish timeline, trace and DAG files as a Tower report

[jordeu]

New feature

Current Tower plugin is only able to use as reports files that are notified with the onFilePublish event. This excludes some files generated by Nextflow like the timeline, trace and DAG files.

Allow the Tower plugin use these files too.

Usage scenario

• A user wants to add the pipeline timeline as a final report and visualize it directly in Tower
• A user wants to download the trace file from Tower
• A user want to download or visualize the DAG file from Tower.

Suggest implementation

Trigger a onFilePublish event for each of these files.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pditommaso]

You are suggesting the onFilePublish event handle is called so that they are managed like any other publishDir file.

I like the idea, however, it's forcing a bit the semantics of that event handle a because those files are not produced by a publishDir

[bentsherman]

The main problem here is that these reports are rendered by the onFlowComplete event, which happens after all files have been published.

nextflow/modules/nextflow/src/main/groovy/nextflow/Session.groovy

Lines 644 to 647 in 0f4d886

	// shutdown publish dir executor
	publishPoolManager.shutdown(aborted)
	// invoke shutdown callbacks
	shutdown0()

So even if you emit a publish event for the reports, it would either be ignored or cause a race condition.

To make it work, we would need to create another event that happens after the workflow completes but before publishing is finished, and publish the reports then.

[stevekm]

since Tower already is aware (?) of the config of the running pipeline, would it be possible for Tower to just check the config after the pipeline completes and see if there are config entries for the report, DAG, and trace, and check if they exist, and add them to the Tower run output?

so that you would basically offload the task to Tower to try to intelligently find the files, instead of modifying the behavior of Nextflow in order to signal it to Tower

not sure if that would be a better or worse option for this functionality

[bentsherman]

@stevekm thanks for the suggestion, that is indeed much simpler and I believe it is the direction we will pursue with the Platform team