Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Benchmark PRs and releases and display well formatted results #322

Closed
Robbepop opened this issue Jan 10, 2022 · 6 comments
Closed

CI: Benchmark PRs and releases and display well formatted results #322

Robbepop opened this issue Jan 10, 2022 · 6 comments
Assignees
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@Robbepop
Copy link
Member

After #304 has been implemented it would be nice to have benchmarks run automatically for every PR and PR update similar to how it is already done with test coverage reporting via codecov.io.

I personally do not know what needs to be done with GitHub Actions CI or if that's actually feasible without a ton of work.

@Robbepop Robbepop added enhancement New feature or request help wanted Extra attention is needed labels Jan 10, 2022
@Robbepop
Copy link
Member Author

I found this GitHub Actions and will try to make it work for wasmi.

@Robbepop
Copy link
Member Author

My personal summary is that https://github.com/marketplace/actions/criterion-compare-prs is not a fit to close the linked issue since the GitHub Actions unfortunately is heavily outdated and has outdated documentation which is a red flag anyways.

We need to look for another solution or build our own GitHub Actions for comparing Citerion benchmarks.

Closed.

@Robbepop Robbepop reopened this Jan 11, 2022
@Robbepop
Copy link
Member Author

Did not want to close this issue but the linked PR: #324

@Robbepop
Copy link
Member Author

Maybe running Criterion benchmarks on GitHub Actions CI is too noisy.
The well known Cachegrind tool is made exactly for usage scenarios like these.
The experimental (unreleased) lai tool from the same author as Criterion might help here.

@Robbepop Robbepop changed the title Add CI that automatically runs benchmarks and posts them to the PR CI: Benchmark PRs and releases and display well formatted results Aug 22, 2022
@Robbepop
Copy link
Member Author

Robbepop commented Sep 9, 2022

This is being worked on by @sergejparity here #422.

@Robbepop
Copy link
Member Author

Implemented in #422. Closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants