Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a (automatic?) process for updating the benchmark results #18

Open
kevlar1818 opened this Issue Sep 2, 2015 · 1 comment

Comments

Projects
None yet
2 participants
@kevlar1818
Copy link

kevlar1818 commented Sep 2, 2015

It would be nice to have the "Results" table in the README kept up to date. For instance, a Rust implementation was recently added (#14), and there's a pull request to improve said implementation (#5).

Obviously, this could be a CI task, but maybe the simpler thing to do is update the README in any pull request which changes/adds implementation(s)? For consistency across machines, probably a complete rewrite of the benchmark results table would be required, perhaps noting the machine specs as well.

This is just a thought and is open for discussion.

@dimroc

This comment has been minimized.

Copy link
Owner

dimroc commented Sep 2, 2015

It's a great idea and I have thought of automating the calculation of each implementation's runtime. I will be manually updating the README later this week.

We have come to a point though where many of these implementations are no longer apples to apples comparisons. They vary in some subtle and not so subtle ways:

  • Holding contents in memory rather than streaming in lines
  • Using regex or using substring
  • Handling ASCII or Unicode
  • Others.

This repo is becoming more a source of idiomatic implementations rather than a fair speed comparison. I'll be including this information in the README.

Your idea is still very valid though, we could still use an automatic way of running and tracking benchmarks across languages. It could spur more community involvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.