Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support performance presets in the Elasticsearch output #37259

Merged
merged 17 commits into from
Dec 5, 2023
Merged

Support performance presets in the Elasticsearch output #37259

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
cac8dff
Add configuration presets to the ES output
faec Nov 30, 2023
20b495b
Log when performance preset is applied
faec Nov 30, 2023
97c285d
Adding tests, fixing bugs
faec Dec 1, 2023
3ebf0b1
rewrite applyPreset and tests to handle worker count correctly
faec Dec 4, 2023
80e851d
add license header
faec Dec 4, 2023
e6ec2a6
Merge branch 'main' of github.com:elastic/beats into config-presets
faec Dec 4, 2023
710eb2f
adjust header order
faec Dec 4, 2023
0b8c5ec
Adjust logging for presets
faec Dec 4, 2023
ddd081e
add changelog
faec Dec 4, 2023
4a90d8a
Fix override checking of user keys
faec Dec 5, 2023
d279a5c
Add presets to default / reference beat configs
faec Dec 5, 2023
8af75ff
Add a unit test for the config namespace corner case
faec Dec 5, 2023
86876da
make update
faec Dec 5, 2023
125ca9e
make update
faec Dec 5, 2023
54767a9
Mention preset overrides in queue settings, change phrasing
faec Dec 5, 2023
5c7776d
Merge branch 'main' of github.com:elastic/beats into config-presets
faec Dec 5, 2023
1de3800
make update again
faec Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
21 changes: 18 additions & 3 deletions libbeat/_meta/config/output-elasticsearch.reference.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,15 @@ output.elasticsearch:
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["localhost:9200"]

# Performance presets configure other output fields to recommended values
# based on a performance priority.
# Options are "balanced", "throughput", "scale", "latency" and "custom".
# Default if unspecified: "custom"
preset: balanced

# Set gzip compression level. Set to 0 to disable compression.
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 1.
#compression_level: 1

Expand All @@ -30,6 +38,8 @@ output.elasticsearch:
#param2: value2

# Number of workers per Elasticsearch host.
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
#worker: 1

# If set to true and multiple hosts are configured, the output plugin load
Expand Down Expand Up @@ -67,8 +77,10 @@ output.elasticsearch:
#max_retries: 3

# The maximum number of events to bulk in a single Elasticsearch bulk API index request.
# The default is 50.
#bulk_max_size: 50
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 1600.
#bulk_max_size: 1600

# The number of seconds to wait before trying to reconnect to Elasticsearch
# after a network error. After waiting backoff.init seconds, the Beat
Expand All @@ -84,7 +96,10 @@ output.elasticsearch:
# The maximum amount of time an idle connection will remain idle
# before closing itself. Zero means use the default of 60s. The
# format is a Go language duration (example 60s is 60 seconds).
# idle_connection_timeout: 60s
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 3s.
# idle_connection_timeout: 3s

# Configure HTTP request timeout before failing a request to Elasticsearch.
#timeout: 90
Expand Down
4 changes: 4 additions & 0 deletions libbeat/_meta/config/output-elasticsearch.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]

# Performance preset - one of "balanced", "throughput", "scale",
# "latency", or "custom".
preset: balanced

# Protocol - either `http` (default) or `https`.
#protocol: "https"

Expand Down
21 changes: 18 additions & 3 deletions x-pack/filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3922,7 +3922,15 @@ output.elasticsearch:
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["localhost:9200"]

# Performance presets configure other output fields to recommended values
# based on a performance priority.
# Options are "balanced", "throughput", "scale", "latency" and "custom".
# Default if unspecified: "custom"
preset: balanced

# Set gzip compression level. Set to 0 to disable compression.
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 1.
#compression_level: 1

Expand All @@ -3943,6 +3951,8 @@ output.elasticsearch:
#param2: value2

# Number of workers per Elasticsearch host.
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
#worker: 1

# If set to true and multiple hosts are configured, the output plugin load
Expand Down Expand Up @@ -3980,8 +3990,10 @@ output.elasticsearch:
#max_retries: 3

# The maximum number of events to bulk in a single Elasticsearch bulk API index request.
# The default is 50.
#bulk_max_size: 50
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 1600.
#bulk_max_size: 1600

# The number of seconds to wait before trying to reconnect to Elasticsearch
# after a network error. After waiting backoff.init seconds, the Beat
Expand All @@ -3997,7 +4009,10 @@ output.elasticsearch:
# The maximum amount of time an idle connection will remain idle
# before closing itself. Zero means use the default of 60s. The
# format is a Go language duration (example 60s is 60 seconds).
# idle_connection_timeout: 60s
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 3s.
# idle_connection_timeout: 3s

# Configure HTTP request timeout before failing a request to Elasticsearch.
#timeout: 90
Expand Down
4 changes: 4 additions & 0 deletions x-pack/filebeat/filebeat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,10 @@ output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]

# Performance preset - one of "balanced", "throughput", "scale",
# "latency", or "custom".
preset: balanced

# Protocol - either `http` (default) or `https`.
#protocol: "https"

Expand Down
21 changes: 18 additions & 3 deletions x-pack/heartbeat/heartbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -542,7 +542,15 @@ output.elasticsearch:
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["localhost:9200"]

# Performance presets configure other output fields to recommended values
# based on a performance priority.
# Options are "balanced", "throughput", "scale", "latency" and "custom".
# Default if unspecified: "custom"
preset: balanced

# Set gzip compression level. Set to 0 to disable compression.
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 1.
#compression_level: 1

Expand All @@ -563,6 +571,8 @@ output.elasticsearch:
#param2: value2

# Number of workers per Elasticsearch host.
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
#worker: 1

# If set to true and multiple hosts are configured, the output plugin load
Expand Down Expand Up @@ -600,8 +610,10 @@ output.elasticsearch:
#max_retries: 3

# The maximum number of events to bulk in a single Elasticsearch bulk API index request.
# The default is 50.
#bulk_max_size: 50
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 1600.
#bulk_max_size: 1600

# The number of seconds to wait before trying to reconnect to Elasticsearch
# after a network error. After waiting backoff.init seconds, the Beat
Expand All @@ -617,7 +629,10 @@ output.elasticsearch:
# The maximum amount of time an idle connection will remain idle
# before closing itself. Zero means use the default of 60s. The
# format is a Go language duration (example 60s is 60 seconds).
# idle_connection_timeout: 60s
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 3s.
# idle_connection_timeout: 3s

# Configure HTTP request timeout before failing a request to Elasticsearch.
#timeout: 90
Expand Down
4 changes: 4 additions & 0 deletions x-pack/heartbeat/heartbeat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,10 @@ output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]

# Performance preset - one of "balanced", "throughput", "scale",
# "latency", or "custom".
preset: balanced

# Protocol - either `http` (default) or `https`.
#protocol: "https"

Expand Down
21 changes: 18 additions & 3 deletions x-pack/metricbeat/metricbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1846,7 +1846,15 @@ output.elasticsearch:
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["localhost:9200"]

# Performance presets configure other output fields to recommended values
# based on a performance priority.
# Options are "balanced", "throughput", "scale", "latency" and "custom".
# Default if unspecified: "custom"
preset: balanced

# Set gzip compression level. Set to 0 to disable compression.
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 1.
#compression_level: 1

Expand All @@ -1867,6 +1875,8 @@ output.elasticsearch:
#param2: value2

# Number of workers per Elasticsearch host.
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
#worker: 1

# If set to true and multiple hosts are configured, the output plugin load
Expand Down Expand Up @@ -1904,8 +1914,10 @@ output.elasticsearch:
#max_retries: 3

# The maximum number of events to bulk in a single Elasticsearch bulk API index request.
# The default is 50.
#bulk_max_size: 50
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 1600.
#bulk_max_size: 1600

# The number of seconds to wait before trying to reconnect to Elasticsearch
# after a network error. After waiting backoff.init seconds, the Beat
Expand All @@ -1921,7 +1933,10 @@ output.elasticsearch:
# The maximum amount of time an idle connection will remain idle
# before closing itself. Zero means use the default of 60s. The
# format is a Go language duration (example 60s is 60 seconds).
# idle_connection_timeout: 60s
# This field may conflict with performance presets. To set it
# manually use "preset: custom".
# The default is 3s.
# idle_connection_timeout: 3s

# Configure HTTP request timeout before failing a request to Elasticsearch.
#timeout: 90
Expand Down
4 changes: 4 additions & 0 deletions x-pack/metricbeat/metricbeat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,10 @@ output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]

# Performance preset - one of "balanced", "throughput", "scale",
# "latency", or "custom".
preset: balanced

# Protocol - either `http` (default) or `https`.
#protocol: "https"

Expand Down
11 changes: 0 additions & 11 deletions x-pack/metricbeat/modules.d/enterprisesearch-xpack.yml.disabled
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This module removal intentional? Didn't see a mention about the removal in the commit or description, but I may have missed it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm -- not intentional on my part, but it looks like it was one of the side effects of doing a make update on the full repo. Looking closer, I see that there's an identical file already in that directory without the -xpack suffix, and that every other module in that directory has only a single config file with no -xpack suffix, so I think this behavior is correct and maybe it got missed because the CI's make check wasn't strict enough to enforce that regeneration. Good catch, thanks :-)

This file was deleted.