-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial implementation benchmarks for the rune cli #240
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall I'm really happy with this PR!
As discussed on Slack we might either choose to ignore build times or only run benchmarks locally before a release. I'm lazy so I'd prefer to just have CI run our benchmarks every time something gets merged into master
, but including it as a manual step in our release process might work too. What do you think will be the best way forward?
There are a couple stylistic comments (e.g. functions take PathBuf
by value or &Path
by reference, but not &PathBuf
because then you've got double indirection), but that's something a quick clippy
run would point out so I'm not overly concerned.
|
||
assert!(blob.len() != 0); | ||
|
||
rune_build_dir.close().expect("Unable to close the rune build directory"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we don't call it then the destructor will make sure our temporary directory is cleaned up so it's up to you whether you want to write an explicit close()
here and blow up if it failed or skip an extra line of code.
From their docs:
Although TempDir removes the directory on drop, in the destructor any errors are ignored. To detect errors cleaning up the temporary directory, call close instead.
let base = example_dir().join("sine"); | ||
let rune = parse_runefile(base.join("Runefile.yml").as_path()).unwrap(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is more personal preference than anything else, but I'd typically write it like this:
let base = example_dir().join("sine"); | |
let rune = parse_runefile(base.join("Runefile.yml").as_path()).unwrap(); | |
let base = example_dir().join("sine"); | |
let runefile = base.join("Runefile.yml"); | |
let rune = parse_runefile(&runefile).unwrap(); |
Or alternatively you could make parse_runefile()
accept anything which is impl AsRef<Path>
. That would also let us accept string literals if we were using it for other tests.
TODO: Refactor out the project_root() code into utils
Since Just this one benchmark is taking an average of 3 minutes on my machine, I haven't added benchmarks that test out building other runes.
Co-authored-by: Michael Bryan <michael@hotg.ai>
Co-authored-by: Michael Bryan <michael@hotg.ai>
Co-authored-by: Michael Bryan <michael@hotg.ai>
This is a long time coming!! |
We use the criterion benchmark framework for this task, as the built in rust benchmarks aren't yet available in rust stable.
Each of the benchmarks use a sample size of 10 instead of the default 100.
These benchmarks measure the time taken for:
The results on my Linux Ryzen 7 1700 machine look like:
TODO: Try to enable Github Actions for these benchmarksThese benchmarks can be run manually before each release, so we aren't adding them to github actions for now