[Meta][Project] Implement the Elastic Agent Data Shipper

[cmacknz]

Goal

The is the project issue tracking the implementation of the Elastic agent data shipper, as designed in the proposal.

Motivation

The shipper is part of the Elastic Agent v2 project: elastic/elastic-agent#189

The goals of the Elastic agent data shipper are to:

• Remove the need for processing, queueing, and output protocols to be reimplemented in each input.
• Minimize the number of output connections required in Elastic agent deployments.
• Simplify configuration and performance tuning of Elastic agent output pipelines.
• Make Elastic agent output pipelines more observable and easier to debug.
• Define the event publishing interface all Elastic agent inputs will use.
• Improve on or maintain the performance of the existing beats outputs.

Development Plan

The rollout of the shipper to customers will occur in incremental phases:

• [Meta] Shipper 8.5 - Experimental integration with Filebeat and Metricbeat #15
• [Meta] Elastic Agent Shipper Project #16
• Phase 3: Migrate remaining beats and agent inputs. TBD.

[cmacknz]

Closing, moving to meta issues that are aligned with the release cycle for tracking.