Skip to content

09. Reports and Reproducibility

Jump to bottom
Ian Brennan edited this page Sep 3, 2024 · 1 revision

Reports

pipesnake automatically generates system reports on time and compute usage in the tracing/pipeline_info/ directory. Of particular interest are the execution_timeline.html and execution_report.html.

The execution timeline will show you the linear time used by each process contributing to the total wall time. timeline

The execution report shows more information for each individual process, including CPU and memory usage.

These reports may be useful to establish the pipeline parameters in your config file to optimise performance.

Software Versions

Stored in your --outdir you will find the pipeline_info/ directory. This holds a .yml file that contains information about all the software versions used in your run of pipesnake.

ASTER:
  astral-hybrid: 1.16.3.4
BBMAP_REFORMAT2:
  BBMAP - reformat.sh: '39.01'
CUSTOM_DUMPSOFTWAREVERSIONS:
  python: 3.10.6
  yaml: '6.0'
GBLOCKS:
  gblocks: 0.91b
IQTREE:
  iqtree: 2.2.5
MERGE_TREES:
  BusyBox: v1.22.1
PHYLOGENY_MAKE_ALIGNMENTS:
  python: 3.8.3
SED:
  sed: This is not GNU sed version 4.0
Workflow:
  Nextflow: 23.04.4
  ausarg/pipesnake: '1.0'

Log and History

pipesnake will generate a complete log file for each run. This file is stored in the working directory as a hidden file .nextflow.log. Subsequent runs get a number appended to the filename, e.g. .nextflow.log.1, .nextflow.log.2, ... This log file includes all information printed to the screen as well as a string with all specified parameters of the run, and should look kinda like this:

Feb-01 18:06:38.259 [main] DEBUG nextflow.cli.Launcher - $> nextflow run ausarg/pipesnake --input SampleInfo.csv --outdir TEST1 -profile docker --stage from-prg --blat_db SqCL_Targets.fasta
Feb-01 18:06:38.299 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.04.4
Feb-01 18:06:38.309 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/ianbrennan/.nextflow/plugins; core-plugins: nf-amazon@1.16.2,nf-azure@1.0.1,nf-codecommit@0.1.4,nf-console@1.0.5,nf-ga4gh@1.0.5,nf-google@1.7.3-patch1,nf-tower@1.5.12,nf-wave@0.8.4
Feb-01 18:06:38.313 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Feb-01 18:06:38.314 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Feb-01 18:06:38.315 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Feb-01 18:06:38.319 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
...

In addition to complete log files, we store a workflow command history. This is also saved as a hidden file, but can be found in the working directory in .nextflow/history, and should look like this:

2024-02-01 18:05:56     -       soggy_kay       -       bfa144bff8ff468914acdd990c6976b4d0a4841e        22233d6c-0f74-42a3-8ea7-92604bf08cc1    nextflow run ausarg/pipesnake --input SampleInfo.csv --outdir TEST1 -profile docker --stage from-prg
2024-02-01 18:06:40     8m 26s  cranky_nobel    OK      bfa144bff8ff468914acdd990c6976b4d0a4841e        9084baa2-6295-4d50-81d0-e787e880f668    nextflow run ausarg/pipesnake --input SampleInfo.csv --outdir TEST1 -profile docker --stage from-prg --blat_db SqCL_Targets.fasta

Clone this wiki locally