Skip to content

Latest commit

 

History

History
314 lines (237 loc) · 6.9 KB

fleet-settings-output-elasticsearch.asciidoc

File metadata and controls

314 lines (237 loc) · 6.9 KB
Raw

{es} output settings

Specify these settings to send data over a secure connection to {es}. In the {fleet} Output settings, make sure that {es} output type is selected.

Hosts

The {es} URLs where {agent}s will send data. By default, {es} is exposed on the following ports:

9200

Default {es} port for self-managed clusters

443

Default {es} port for {ecloud}

Examples:

Refer to the {fleet-server} documentation for default ports and other configuration details.

{es} CA trusted fingerprint

HEX encoded SHA-256 of a CA certificate. If this certificate is present in the chain during the handshake, it will be added to the certificate_authorities list and the handshake will continue normally. To learn more about trusted fingerprints, refer to the {ref}/configuring-stack-security.html[{es} security documentation].

Proxy

Select a proxy URL for {agent} to connect to {es}. To learn about proxy configuration, refer to [fleet-agent-proxy-support].

Advanced YAML configuration

YAML settings that will be added to the {es} output section of each policy that uses this output. Make sure you specify valid YAML. The UI does not currently provide validation.

See Advanced YAML configuration for descriptions of the available settings.

Make this output the default for agent integrations

When this setting is on, {agent}s use this output to send data if no other output is set in the agent policy.

Make this output the default for agent monitoring

When this setting is on, {agent}s use this output to send agent monitoring data if no other output is set in the agent policy.

Performance tuning

Choose one of the menu options to tune your {agent} performance when sending data to an {es} output. You can optimize for throughput, scale, latency, or you can choose a balanced (the default) set of performance specifications. Refer to Performance tuning settings for details about the setting values and their potential impact on performance.

You can also use the Advanced YAML configuration field to set custom values. Note that if you adjust any of the performance settings described in the following Advanced YAML configuration section, the Performance tuning option automatically changes to Custom and cannot be changed.

Performance tuning settings

Table 1. Performance tuning preset values
Configuration Balanced Optimized for Throughput Optimized for Scale Optimized for Latency

bulk_max_size

1600

1600

1600

50

worker

1

4

1

1

queue.mem.events

3200

12800

3200

4100

flush.min_events

1600

1600

1600

2050

flush.timeout

10

5

20

1

compression

1

1

1

1

idle_timeout

3

15

1

60

For descriptions of each setting, refer to Advanced YAML configuration. For the queue.mem.events, flush.min_events and flush.timeout settings, refer to the {filebeat-ref}/configuring-internal-queue.html[internal queue configuration settings] in the {filebeat} documentation.

Balanced represents the new default setting (out of the box behaviour). Relative to Balanced, Optimized for throughput setting will improve EPS by 4 times, Optimized for Scale will perform on par and Optimized for Latency will show a 20% degredation in EPS (Events Per Second). These relative performance numbers were calculated from a performance testbed which operates in a controlled setting ingesting a large log file.

As mentioned, the custom preset allows you to input your own set of parameters for a finer tuning of performance. The following table is a summary of a few data points and how the resulting EPS compares to the Balanced setting mentioned above.

These presets apply only to agents on version 8.12.0 or later.

Table 2. Performance tuning: EPS data
worker bulk_max_size queue.mem_events flush.min_events flush.timeout idle_timeout Relative EPS

1

1600

3200

1600

5

15

1x

1

2048

4096

2048

5

15

1x

1

4096

8192

4096

5

15

1x

2

1600

6400

1600

5

15

2x

2

2048

8192

2048

5

15

2x

2

4096

16384

4096

5

15

2x

4

1600

12800

1600

5

15

3.6x

4

2048

16384

2048

5

15

3.6x

4

4096

32768

4096

5

15

3.6x

8

1600

25600

1600

5

15

5.3x

8

2048

32768

2048

5

15

5.1x

8

4096

65536

4096

5

15

5.2x

16

1600

51200

1600

5

15

5.3x

16

2048

65536

2048

5

15

5.2x

16

4096

131072

4096

5

15

5.3x