Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a benchmark for endoding/decoding #24

Closed
judah opened this issue Jul 9, 2016 · 4 comments
Closed

Add a benchmark for endoding/decoding #24

judah opened this issue Jul 9, 2016 · 4 comments

Comments

@judah
Copy link
Collaborator

judah commented Jul 9, 2016

We should have a benchmark for encoding to/from the wire format, to make sure that our reflection and abstraction in the parser doesn't cause a significant slowdown compared to other Haskell code, and to give us more confidence when refactoring. (So far, we haven't done any performance tuning of the code.)

One arbitrary data point: decoding a 1MB proto took ~60ms on my desktop, and decoding followed by encoding (which includes forcing all the fields) took ~230ms. (The code was compiled with -O2.)

We should probably also benchmark the text format, though that's usually less performance-critical.

@agrue
Copy link
Contributor

agrue commented Aug 26, 2016

I am working on a set of criterion benchmarks.

@agrue
Copy link
Contributor

agrue commented Aug 31, 2016

Some updates... adding criterion benchmarks was quite simple. So I'm currently trying to figure out what a reasonable initial set of benchmarks looks like. The only thing remotely like a "standard benchmark" I could find was here: https://github.com/google/protobuf/tree/master/benchmarks. But I'm not sure we benefit much from imitating it; it looks like a raw speed test for a single large, heterogeneous proto. It seems more interesting to me to have a larger number of small, focused test cases to help uncover pathological cases (at least right now, with our largely unoptimized code).

So for the initial PR I guess I'll just come up with a small set of test cases that make sense to me. If anyone disagrees with the above or has any requests for specific benchmarks, please respond here. :)

@judah
Copy link
Collaborator Author

judah commented Aug 31, 2016

That plan sounds great; thanks for taking this on!

Eventually we should test the effect of proto-lens putting all fields in a single, immutable, flat Haskell record. For large protos my guess is it's significantly less efficient (in particular, for decoding) than languages which can do updates by mutating a single field. The standard benchmark might help us examine that use case.

@judah
Copy link
Collaborator Author

judah commented Jul 31, 2017

Closing since the current set of benchmarks has proven useful so far. If we run into any particular bottlenecks in the future, we can add more specific benchmarks as needed.

@judah judah closed this as completed Jul 31, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants