[Meta] Shipper 8.5 - Experimental integration with Filebeat and Metricbeat

[cmacknz]

The 8.5 release milestone minimum feature set necessary to use the shipper with metricbeat and filebeat: only support the memory queue and the Elasticsearch output, leaving processors in the beats. The shipper can be enabled by setting an environment variable when starting the agent. A feature flag will be added to the agent policy and Fleet UI in a future iteration.

Dependencies

The primary dependency of the phase 1 data shipper is the agent control plane. The control plane work to support the phase 1 shipper is tracked in:

• [Meta] V2 Feature Architecture (8.5/8.6) elastic-agent#836
  • [Meta][Shipper] Support the Elastic Agent Data Shipper elastic-agent#214

Implementation

The implementation tasks for the shipper are broken into several independent feature groups, listed below. Tasks will be converted to issues as the implementation progresses.

• Setup the shipper's build, lint, and CI process #4
• Define the initial shipper gRPC interface #5
• Create a skeleton shipper executable #6
• Allow the Elastic Agent to run the shipper #13
  • Prototype running the shipper under agent using the V1 protocol #46
  • Update with the latest changes to the V2 agent protocol #101
  • Test the shipper's V2 agent control client once the server is complete.
• [Meta][Feature] Implement the memory queue and output pipeline #7
• Allow running the shipper in standalone mode independently of the agent #83
• [Meta][Feature] Enable filebeat and metricbeat to publish data to the shipper #8
  • Implement and document the basic experimental shipper output type #22
  • Add integration tests for the shipper output #23
  • Add support for input and data stream identification when sending events #24
  • Implement the shipper's gRPC API #34
    • The PublishEvents RPC should block when the queue is full #84
    • The API specification is updated with any changes or improvements that were necessary.
    • Allow google.protobuf.Timestamp in shipper Event fields and metadata #62
    • A reference API client implementation exists and integrates with the server side implementation.
    • Allow the Beats' shipper output to depend on the shipper client directly. #64
    • Benchmark the shipper output's conversion from publisher.Event beats#32081
    • Implement validation for incoming events #79
    • Build 799 for main with status FAILURE beats#32814
    • An integration test suite exists to verify that the client and server implementations interact correctly, with emphasis on error conditions: partially accepted batches, full queues, server and client side restarts, etc.
      • Make the shipper output to acknowledge batches based on PersistedIndex not AcceptedCount beats#32329
      • Beats shipper output should not wait for full acknowledgment before sending more batches #98
• [Meta][Feature] Implement end to end acknowledgement #9
  • Add entry IDs to the memory queue beats#32541
  • Make the shipper output to acknowledge batches based on PersistedIndex not AcceptedCount beats#32329
  • Beats shipper output should not wait for full acknowledgment before sending more batches #98

QA / Testing Needs

Deferred to 8.6

Documentation

N/A - The shipper will be experimental and enabled via an environment variable when starting the agent.

[nimarezainia]

@cmacknz @pierrehilbert is this meta issue what we should track for experimental availability of shipper in 8.5?

[cmacknz]

Yes this is the issue for the 8.5 milestone.

[cmacknz]

Only outstanding issues to close this is the Elasticsearch output #10 and standalone mode which is already in review #83

[cmacknz]

Closing, all outstanding work moved to the next milestone: #16