[Fleet] Add a Logstash output type in Fleet settings

[mostlyjason]

Allow users to add and edit outputs for Logstash in Fleet settings. It should contain a list of hosts and a YAML configuration block for more settings. The settings should match those provided by Beats https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html. We also want to require/encourage users to set up Logstash with minimal privileges and mTLS for good security. Please see the google doc for more information about the UX.

Goals

1. When a user is adding a LS output to Fleet, we should prompt the user to update their Logstash config to use an API key with an appropriate set of access permissions. We should generate this key on behalf of the user to make it easy for them to make this change.
2. Users should see instructions on how to enable the Elastic Agent input in Logstash, and enable data streams in the Elasticsearch output.
3. Users should be able to update the credentials for Logstash if needed, without downtime for their existing data ingestion
4. Users should have an option to add new integrations without manually reconfiguring Logstash each time.
5. We should require users to enable mutual TLS because plaintext connections could expose user data, and it's the only way Logstash knows it's connected to a trusted source.
6. We should prevent users from using the Logstash output with APM until proper support is added.

Requirements

Fleet UI requirements:

• https://github.com/elastic/ingest-dev/issues/1047
• [Fleet] Add logstash output type in output API and preconfiguration #125792
• [Fleet] Implement logstash output form  #125666
• [Fleet] Do not allow to use logstash output with APM integration #125994
• Document how users configure a Logstash output, including SSL certs [Request] Document how to use logstash in Fleet managed mode observability-docs#1691

Control team requirements:

• https://github.com/elastic/obs-dc-team/issues/601
• [Elastic Agent] Infrastructure to support multiple outputs in a policy beats#27442 - unsure if needed
• Require mTLS for Logstash output, with opt-out

Kibana requirements:

• System Indices elasticsearch#50251

Endpoint security requirements:

• See private Endpoint security issue

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ThomSwiss]

For our company, the output to kafka or at least to logstash is realy important. I would like to deploy more than 10'000 Elastic Agents in our company but I don't like to send the data directly as elasticsearch outplut. Is there a plan to implement this?

[mostlyjason]

@ThomSwiss yes this is a near term priority for our team to implement

[jen-huang]

Closing as complete, docs requirement tracked in elastic/observability-docs#1691.

[amolnater-qasource]

Hi @nchaulet
We have created 17 testcases for this feature under our Fleet test suite at link fleet_suite.

Please let us know if we are missing anything.
Thanks

[amolnater-qasource]

Hi Team
We have also created 01 testcase covering detailed steps for logstash output setup and checking the data under our Fleet test suite at link:

• [Self-Managed]: Validate user is able to setup logstash output and data is available under Data Streams tab.

Please let us know if anything else is required from our end.
Thanks

[amolnater-qasource]

Hi Team
We have executed 18 testcases for this feature under our Fleet Test plan at links:

• Policy Output Settings
• Logstash Output Settings

Hence marking this as QA:Validated.
Please let us know if anything else is required from our end.
Thanks

[amolnater-qasource]

Hi @jlind23 @joshdover
We have attempted to validate logstash output feature on 8.2 BC4 Kibana cloud environment.
Earlier we had success on self-managed only, however now we are also able to run this feature on cloud environments too.

Build details:
VERSION: 8.2.0 BC-4 cloud-production
BUILD: 52005
COMMIT: 9a5003d
Artifact Link: https://staging.elastic.co/8.2.0-3b2b9b86/summary-8.2.0.html

Screenshots:

Thanks