forked from elastic/elasticsearch
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Avoid more time-consuming calls (elastic#70)
With this commit we reduce usage of two time-consuming methods on the hot code path: generating random numbers and determining the current timestamp. This also changes behavior slightly: * The `hostname` field will vary host names from 1 - 3 in a regular pattern instead of randomly (however, as the host name is e.g. `web-EU-1.elastic.co` and only the number changes non-randomly we deem this change ok) * The `offset` will change be more realistic now: Before it changed randomly and now the offset increases by the average event size up to a certain maximum. * The current `@timestamp` will be retrieved only once per bulk. For documents within a bulk we'll advance the microsecond portion by `1 / bulk size` microseconds. We've measured the performance impact of this change by stubbing out Elasticsearch with nginx and running the `index-logs-fixed-daily-volume` challenge with the following track parameters: * `bulk_size`: 20000 * `bulk_indexing_clients`: 16 * `number_of_days`: 1 * `daily_logging_volume`: "20GB" We have measured the following median indexing throughput in our test environment: * baseline (master): 153476 docs/s * Using a deterministic `hostname` and `offset`: 174371 docs/s * All three measures together: 222611 docs/s This means we improve the maximum achievable indexing throughput by 45% in this configuration.
- Loading branch information
1 parent
146a372
commit 8efa5cd
Showing
6 changed files
with
119 additions
and
17 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters