Skip to content

Conversation

@britto
Copy link
Collaborator

@britto britto commented Jul 8, 2020

Protobuf includes benchmarks for its official language implementations, such as Python, C++ and Golang. They measure average encode and decode throughput for each built-in dataset. This is useful to check how Elixir matches up with them.

https://github.com/protocolbuffers/protobuf/blob/master/benchmarks/README.md

For each dataset, it runs decode (then encode) on all the payloads. If that dry run takes longer than 3s, its time is taken as the average. Otherwise, repeat a few runs and average their times.

Caveat

Throughput is measured in MB/s, always based on the binary payload size from the dataset, even when the output size doesn't match – which is the case when the messages include proto2 groups, which are currently skipped. This can give unreliable results, so it's important to take that into account when comparing with feature-complete implementations.

Sample output

Message benchmarks.proto2.GoogleMessage1 of dataset file data/dataset.google_message1_proto2.pb
Average throughput for parse_from_benchmark: 18.48 MB/s
Average throughput for serialize_to_benchmark: 6.19 MB/s

Message benchmarks.proto3.GoogleMessage1 of dataset file data/dataset.google_message1_proto3.pb
Average throughput for parse_from_benchmark: 18.4 MB/s
Average throughput for serialize_to_benchmark: 11.1 MB/s

Message benchmarks.proto2.GoogleMessage2 of dataset file data/dataset.google_message2.pb
Average throughput for parse_from_benchmark: 47.82 MB/s
Average throughput for serialize_to_benchmark: 5656.75 MB/s

Message benchmarks.google_message3.GoogleMessage3 of dataset file data/dataset.google_message3_1.pb
Average throughput for parse_from_benchmark: 19.94 MB/s
Average throughput for serialize_to_benchmark: 45.5 MB/s

Message benchmarks.google_message3.GoogleMessage3 of dataset file data/dataset.google_message3_2.pb
Average throughput for parse_from_benchmark: 110.65 MB/s
Average throughput for serialize_to_benchmark: 164.96 MB/s

Message benchmarks.google_message3.GoogleMessage3 of dataset file data/dataset.google_message3_3.pb
Average throughput for parse_from_benchmark: 9.8 MB/s
Average throughput for serialize_to_benchmark: 6.84 MB/s

Message benchmarks.google_message3.GoogleMessage3 of dataset file data/dataset.google_message3_4.pb
Average throughput for parse_from_benchmark: 5254.14 MB/s
Average throughput for serialize_to_benchmark: 737.71 MB/s

Message benchmarks.google_message3.GoogleMessage3 of dataset file data/dataset.google_message3_5.pb
Average throughput for parse_from_benchmark: 3.77 MB/s
Average throughput for serialize_to_benchmark: 3.29 MB/s

Message benchmarks.google_message4.GoogleMessage4 of dataset file data/dataset.google_message4.pb
Average throughput for parse_from_benchmark: 20.06 MB/s
Average throughput for serialize_to_benchmark: 32.46 MB/s

@whatyouhide whatyouhide merged commit a5cfd46 into elixir-protobuf:master Jul 20, 2020
@britto britto deleted the standard-benchmarks branch October 21, 2021 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants