Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BENCHMARK] tool comparison macro benchmarks #156

Merged

Conversation

Irallia
Copy link
Collaborator

@Irallia Irallia commented Sep 15, 2021

Resolves #123

Our first tool comparison:
CallerComparisonPlot.pdf

@Irallia Irallia self-assigned this Sep 15, 2021
@codecov
Copy link

codecov bot commented Sep 15, 2021

Codecov Report

Merging #156 (7c72242) into master (92cc315) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #156   +/-   ##
=======================================
  Coverage   94.93%   94.93%           
=======================================
  Files          18       18           
  Lines         731      731           
=======================================
  Hits          694      694           
  Misses         37       37           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92cc315...7c72242. Read the comment docs.

@Irallia Irallia requested a review from eaasna September 15, 2021 15:11
Copy link
Contributor

@eaasna eaasna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! You already have a nice structure set up, having a separate configuration file and directories for rules and scripts. Might also be good to extract the rules .smk files from /parameter_benchmarks/Snakefile.

I've added some comments, but might have misunderstood your init process :)

params:
sample = config["parameters"]["sample"],
min_qual = config["parameters"]["min_qual"],
min_var_length = config["parameters"]["min_var_length"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the same parameter is used for multiple rules you could define e.g min_var_length globally (before the rules) for the whole .smk file and use it in the shell call as {min_var_length} instead of {params.min_var_length}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm not sure, if this makes it more clear. What do you say @joergi-w ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @eaasna that all parameters that are the same in all the rules, should be defined above, so you don't need to repeat them. If it looks too dirty, you can still group them there (but I guess it's fine).

test/benchmark/iGenVar_init.sh Show resolved Hide resolved
@Irallia Irallia requested a review from eaasna September 16, 2021 11:44
Copy link
Member

@joergi-w joergi-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some ideas:

test/benchmark/caller_comparison/Snakefile Show resolved Hide resolved
test/benchmark/caller_comparison/config/config.yaml Outdated Show resolved Hide resolved
test/benchmark/caller_comparison/config/config.yaml Outdated Show resolved Hide resolved
test/benchmark/caller_comparison/config/config.yaml Outdated Show resolved Hide resolved
params:
sample = config["parameters"]["sample"],
min_qual = config["parameters"]["min_qual"],
min_var_length = config["parameters"]["min_var_length"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @eaasna that all parameters that are the same in all the rules, should be defined above, so you don't need to repeat them. If it looks too dirty, you can still group them there (but I guess it's fine).

test/benchmark/iGenVar_init.sh Show resolved Hide resolved
test/benchmark/parameter_benchmarks/Snakefile Show resolved Hide resolved
Copy link
Member

@joergi-w joergi-w left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Re-run the doxygen CI, as it failed temporarily.

…ing_evaluation for a snakemake workflow.

Signed-off-by: Lydia Buntrock <lydia.buntrock@fu-berlin.de>
Signed-off-by: Lydia Buntrock <lydia.buntrock@fu-berlin.de>
Signed-off-by: Lydia Buntrock <lydia.buntrock@fu-berlin.de>
Signed-off-by: Lydia Buntrock <lydia.buntrock@fu-berlin.de>
Signed-off-by: Lydia Buntrock <lydia.buntrock@fu-berlin.de>
@Irallia Irallia force-pushed the TEST/benchmarks/tool_comparison_macro_benchmarks branch from 4b3eb1c to 7c72242 Compare September 17, 2021 08:30
@Irallia Irallia merged commit 547ce22 into seqan:master Sep 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[TEST] Create some shell scripts for benchmarking
3 participants