Benchmark figures

[mbhall88]

Create benchmark figures for time and memory usage.

How do we want to do time? CPU time, wall time? I guess this gets complicated by threads, although I am using 2 for each tool at the moment.

[mbhall88]

Okay, so this is what I have for the nanopore data. Anything we want changed?

Runtime

Max. RAM usage

[lachlancoin]

Nice visualisation

…

On Wed, 12 Oct 2022 at 16:30, Michael Hall ***@***.***> wrote: Okay, so this is what I have for the nanopore data. Anything we want changed? Runtime [image: image] <https://user-images.githubusercontent.com/20403931/195257678-cde00e5c-8d00-4b2b-b807-cd266b75e77a.png> Max. RAM usage [image: image] <https://user-images.githubusercontent.com/20403931/195257695-4863394e-9a86-4c5c-9c9c-d1c1365a2854.png> — Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6TKZBCDNI43ONLU5FJM6TWCZEFXANCNFSM6AAAAAARB5M3YM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[iqbal-lab]

Looks great. Nothing to change, though I'm wondering what the p values are testing

[lachlancoin]

Just testing whether the means of the distributions are equal. Pretty clear they are not from the plot but I guess it helps if you want to say "significantly different" . I guess you used an unpaired t-test, but actually they are paired data (same sample using different approaches) If anything that should make the pvalues even more significant.

…

On Wed, 12 Oct 2022 at 17:40, Zamin Iqbal ***@***.***> wrote: Looks great. Nothing to change, though I'm wondering what the p values are testing — Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6TKZATS354REDKE4BQDTLWCZMOZANCNFSM6AAAAAARB5M3YM> . You are receiving this because you commented.Message ID: ***@***.***>

[mbhall88]

I used a wilcoxon signed-rank test. Basically the nonparametric version of a paired t-test (because I'm lazy and couldn't be bothered testing for normality 😅 )

Pretty clear they are not from the plot but I guess it helps if you want to say "significantly different"

It wasn't actually clear to me that the runtimes would be significantly different for drprg and mykrobe

[mbhall88]

Here are the Illumina figures. These aren't final as this is only a subset of 400 samples, but I can't imagine it will change dramatically.

Memory

Runtime

[mbhall88]

The latest figures after adding in the racon denovo from iqbal-lab-org/pandora#299 can be seen in iqbal-lab-org/pandora#303 (comment)