Set field limit, after testing with rally. #1444
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add rally challenge to test mapping explosion limits for APM. implements elastic#1291
|
After discussing the timeouts with @danielmitterdorfer I changed the on a 4gb RAM, 1 node, 1 zone cloud instance leads to following results: [updated results] increasing warmup time to
Performance overview of the ES instance: |
Reduce bulk size to apm-server default of 50. Increase warmup period to reduce measured latency.
|
After rerunning the tests (see results above) I suggest to set the @roncohen what's your opinion on that? |
|
As you've shown, with the right size machines and configuration, it could work to have a higher number of columns than 2000. Se let me see if this is your line of thinking: if you have 2000+ tags you're basically doing it wrong. So in that case it's better to loudly complain and stop indexing, even in cases where the cluster could actually handle more tags, because we would expect the number of tags to keep increasing beyond that because the feature is not being used correctly. |
|
APM Server's default settings are generally rather conservative to fit for a small APM machine. So my suggestion is to align the default setting for (Note: ES default value for total_fields is 1_000, libbeat default value is 10_000). |
|
OK. Sounds good 👍 |
|
jenkins, retest this please |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add rally challenge to test mapping explosion limits for APM.
implements #1291
I did some tests using the APM rally challenge
ingest-field-explosionon ES instances with the following results:The amount of fields other than
tagsis roughly 200. My suggestion is to set1,000as default value fortotal_fieldsfor6.5. Setting it too low would cause errors when trying to write although the ES instance might be able to handle it. Setting the value too high would risk memory issues with the whole ES instance.