Q1: Specification - Compute Resources

[mfansler]

Create specification for Q1.

Working definition: Obtain execution time and memory usage in MiB for method execution. If multiple rules or samples for method: sum time and get max MiB.

[dominikburri]

What we need to think about to properly benchmark the methods:

• Which dataset, resp. samples, to use?
• How many replicates to use (if any) and how to report them? E.g. box plots for 10 executions.
• On which platform should it be tested? This also influences how the results can be obtained, e.g. when using AWS there is propably a way to automatically obtain the compute usages.
• The benchmark should be irrespective of the implementation of the workflow. That is, using a nextflow or snakemake specific approach could lead to systematic differences based on their implementation.
• other?