[Fleet] Add option to enable disk queue for elastic-agent-shipper

[leehinman]

Part of #elastic/elastic-agent-shipper#118

Description

We need to support the new Elastic Agent Shipper in Fleet, that is going to replace the beats options currently configured in settings/outputs through YML editor. The aim is to allow the user to configure the new options from the Settings Output Flyout.

• Options from add disk queue configuration to shipper configuration should be user configurable. The parameters shown in the UI are going to map those found in the reference configuration file.
  • Note that these are still in preliminary definition and might be changed. They are the same as current beats configs except for use_compression and  encryption_password.
  • The maximum batch size and flush timeout parameters are not yet defined for the shipper, so we're going to have placeholders (maximum_batch_bytes and output_queue_flush_timeout) until they get implemented.
  • Memory queue size for now is going to be Memory queue events. Copy: "Maximum number of events that can be stored in the queue. Default is set to 4096"
  • The encryption password doesn't need to be exposed in the UI as users don't need to be aware of it
• Show the new options only when the Shipper feature flag is enabled per output (see comment)
  • The options should be visible for both Logstash and ES outputs
• Implement toggle for disk queue vs memory queue (memory queue is the default). For the shipper the memory queue parameters are defined here.
  • Queue option is persistent between agent upgrade/downgrade
  • If downgrade is to 8.5.x or older, memory queue option is used (as disk queue wasn't available) and the new configuration under the shipper: object should be ignored
• Disk Queue section should be collapsed when Disk Queue toggle is not enabled. Once enabled, all the options are available to the user to configure.
• If Disk Queue and Encryption are enabled due to an integration, then Disk Queue and Encryption should be grayed out.
• If Compression is disabled, do not display the compression levels.
  • Once compression is enabled, the compression levels (1 - 9) should be configurable.
  • All other options are available for configuration
• Load balancing and compression options are unique to each output type. They'll be always visible in the case of Logstash/ES, however when the Kafka output will be added won't be applicable and will need to be hidden/grayed out
• Add new shipper section to agent policy. The exact format is still being discussed ( see Namespace config values in fleet output elastic-agent#1729)
• Write migration for new output format

Demo

• Prepare recorded demo

Updated Mockups

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[cmacknz]

@nimarezainia to follow up with more details here on what should be configurable and make sure we get UI mocks created.

[nimarezainia]

@cmacknz @ @leehinman I discussed this with @dborodyansky and the UI mock up we have in the definition document seems pretty straight forward to use for all the configurations discussed for the new shipper. Higher fidelity examples can be created and we will get someone from docs to look over the copy as there are some text boxes to consider.

@kpollich added this to our design discussions. Let's clarify then.

[criamico]

@nimarezainia i'm starting to look at this and I have some questions about the designs.

Comparing the current output settings and this desired UI, it's not completely clear what to do of the YML editor. Should we keep it, since it allows to configure many more features?
If it stays, how should we handle any potential conflicts between the YML settings and the new UI configurations? Which one should override the other?

[nchaulet]

Comparing the current output settings and this desired UI, it's not completely clear what to do of the YML editor. Should we keep it, since it allows to configure many more features?

Yes it's used to configure some ssl configuration, we should probably keep it.

if downgrade is to 8.5 or older memory queue is used (disk queue wasn't available)

The agent policies are not specific for an agent version, does the agent < 8.6 will handle the option and use memory queue if we pass disk queue or it will crash?

[criamico]

Yes it's used to configure some ssl configuration, we should probably keep it.
@nchaulet Thanks for this clarification. We just need to determine if:

1. we want to give more weight to the YML content hence overriding the UI settings
   OR
2. we want to discard the content of the YML based on what's in the UI settings.

[cmacknz]

Drive by comment tangentially related to the thread above:

One thing that is not captured in this issue is that the agent functionality that supports the new configurations here will be behind a feature flag that is disabled by default. Specifically, users would need to opt in to using the Elastic agent data shipper for each output they have.

I don't think we would want to present this UI unless a user has enabled the shipper feature flag in the agent policy. Without enabling the shipper these configuration options do nothing, and I don't think we should present them to users if they have no effect. Presenting these options by default would be very confusing, as users could configure a disk queue but the agent would ignore the configuration option.

Ideally we should also have checkbox or similar toggle to opt in to the new elastic agent shipper on a per output basis. Only if that option is selected should these new UI elements be shown. Only once we get out of Beta we would default to having this flag enabled by default and the new queue configuration UI visible default.

Possibly this context aws intended to be provided in #142649 which isn't complete yet. Pinging @nimarezainia for additional input here.

[cmacknz]

One exception to my comment above, the Compression and Loadbalancing options here are always valid, regardless of if the agent is using the data shipper or the existing output architecture.

[criamico]

Thanks @cmacknz, I'll take in account that these features are available under a feature flag.

[kpollich]

I'll take a pass at rewriting this issue description to be less of a placeholder today. I will be sure to include the feature flag functionality as a requirement here.

[criamico]

I'm currently gathering more information about this ticket and I'd like to have confirmation that I got them right:

• Load balancing and compression should always be visible and they are the parameters described here:

  • https://www.elastic.co/guide/en/beats/filebeat/current/load-balancing.html
  • https://www.elastic.co/guide/en/beats/filebeat/8.5/configuration-monitor.html#_compression_level_4

• The options in the first section (Maximum batch size, flush timeout and memory queue size) already exist. I connected them to these three parameters, but I'd like to make sure that they're correct ones:

  • Maximum batch size: https://www.elastic.co/guide/en/beats/filebeat/8.5/elasticsearch-output.html#_bulk_max_size
  • Flush timeout: https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html#configuration-internal-queue-memory
  • Memory Queue size: https://www.elastic.co/guide/en/beats/filebeat/current/configuring-internal-queue.html#configuration-internal-queue-disk

• The only new options (under feature flag) would then be Disk Queue and Encryption. I'm trying to find the correct parameters to be listed. @cmacknz do you have any docs that point to them?
  I'd also need to know how to enable/run the shipper. @leehinman Could you help with these questions?