Add benchmarks to compare text and protobuf parsing. #53
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
text/bench_test.go
Outdated
|
👍 otherwise |
|
That'd indicate ~5us per sample in text format. With a 60s interval and 500k timeseries (rough m1.large limit), that's only .04 of a core on a 2 core box. I don't think we need to worry about performance in the general case, the rules/graphs/storage will dominate. |
added 2 commits
January 27, 2015 17:16
|
@juliusv Oh my... I hope my typos are not a representation for my mental state... |
|
@brian-brazil Wait for fish running Prometheus on a Raspberry Pi... :) But you are right, not our most pressing concern right now. The gzipped text format is actually not much larger than the gzipped protobuf, which gets rid of the bandwidth concern. CPU-wise: Even the 50k samples/s scenario will only need 0.25 cores. GC-pressure might hurt more, here... but who knows. |
beorn7
added a commit
that referenced
this pull request
Jan 27, 2015
Add benchmarks to compare text and protobuf parsing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@brian-brazil @juliusv
I guess that means for high sample ingestion rates, we should aim for (optionally) supporting protobuf in the new Java client, too...
On the other hand: As the parsing is happening in parallel with many scrapes, the bottleneck for sample ingestion on spinning disk machines is persisting the chunks to disk. (With more efficient compression for chunks, that might change, though.) On SSD, the bottleneck (with protobuf ingestion) appeared to be the fingerprint calculation, which happens serially at the moment. Should anybody ever want more than 50k samples/sec ingestion, we would improve the fingerprint calculation first.
Still, especially for situations where CPU is tighter (and the server is also serving expensive queries and such), a 4x speed-up on parsing seems worth the deal...