[Meta] Elastic Agent Shipper Project

[cmacknz]

Experimental, Beta, and GA Criteria

• [Meta] elastic-agent-shipper journey to GA #197

Blockers for Beta Milestone 1 - Elasticsearch output and memory queue

NEW - Convert the shipper to a Filebeat input

• [Elastic Agent] Prototype a shipper gRPC input for Filebeat beats#35135
• Support the shipper as an instance of Filebeat configured to use the shipper protocol input elastic-agent#2521
• Allow queue configuration to be specified under the output type beats#35615 (enables setting shipper queue config via agent policy)
• Each shipper input unit should have a dedicated unix socket for inter-process communication elastic-agent#2581
• Add elastic_agent.shipper to the list of known monitoring datasets in Fleet elastic-agent#2654
• Update the shipper monitoring implementation to be a variant of Beat monitoring elastic-agent#2580
• [Elastic Agent] Implement the shipper PersistedIndex API in the Filebeat shipper input beats#35266
• [Elastic Agent] Implement input metrics for the shipper Filebeat input beats#35267
• Archive this repository and remove the elastic-agent-shipper binary from the release process #306
• Allow queue configuration to be specified under the output type beats#35615

Bugs

• github.com/elastic/beats/v7/libbeat/outputs/shipper tests are flaky beats#34984
• The shipper should report itself as FAILED when multiple outputs are configured. #171
• TestPersistedIndex/server_should_properly_shutdown is flaky #300
• Fail to start if the shipper is enabled but the shipper executable cannot be found elastic-agent#2446
• Disk queue re-sends last segment file when there is more than on segment on disk beats#32560
• Fix mage test:integration #244
• Fix so shipper doesn't stall while sending via elasticsearch output #255
• Fix elasticsearch output adding bulk index item #253
• include @timestamp in events sent via elasticsearch output #256
• Shipper output fails on large events/batches beats#34695
• Shipper output retries forever on all errors regardless of actual retry settings beats#34700
• Beats pipeline doesn't respect configured batch sizes on startup under agent beats#34703
• The Elasticsearch output should not report itself as degraded based only on the time between events #301

Features

• Implement design to handle queued events on policy update #286
• Investigate way to prove events ingested with & without shipper are functionally the same #257
• Support monitoring the shipper from the Elastic Agent #267
• Beats should use the proxy queue when the shipper output is active beats#34396

Elasticsearch Output V2

• Split large batches of documents if received 413 from Elasticsearch beats#29778
• Shipper doesn't support infinite retry / robust failure handling #262
• Allow the Elasticsearch output to load balance between multiple hosts #152
• Support output proxy configuration #172
• Expose go-elasticsearch BulkIndexer output performance metrics #153
• Decide what the default shipper Elasticsearch output configuration should be #291
• Tune / finalize the queue's batch and timing parameters #58

Monitoring

• Add a queue utilization metric to report how full the queue is as a percentage #265
• Add a counter for the number of bytes transmitted by the shipper #271
• Elastic agent metric dashboard for the shipper #54

Performance

• Onboard the shipper into the automated ingest performance testing framework #266
• The shipper event protocol should be a wrapper for JSON bytes #288

Testing

• Add document equivalency test case to end to end tests #290
• Add integration tests between the Beats shipper client and the shipper server beats#33205
• Add an end to end test suite for the shipper. #173
• Add integration tests for end to end acknowledgement #264

Technical Debt

• Move batch assembly / acknowledgment out of the Beats queue #186
• Shipper configuration tests are run against invalid configurations elastic-agent#2147
• Config parameters aren't passed through to the shipper output in Beats elastic-agent#2351
• Combine the shipper client into https://github.com/elastic/elastic-agent-client elastic-agent-shipper-client#20
• Meta Issue: Shipper configuration and I/O #226
  • Namespace config values in fleet output elastic-agent#1729
  • Input configs from units should hard-code shipper connection settings elastic-agent#1744
  • Standardize the shipper's gRPC TLS setup process #224
  • Look into changing how the shipper recieves gRPC connection config #225
• Investigate use of vtprotobuf to generate the shipper gRPC API #263

Documentation

• Document the shipper process and how it interacts with agent policy:
• Document the shipper feature flag and the current stability guarantees.
• Document why a user would want to enable the shipper.
• Document how the number of shippers (and therefore output connections) can be controlled via the agent policy.
• Document configuration of the shipper queues.
• Document any backwards compatibility or migration issues when moving from per beat outputs and queues to the centralized shipper.

Future Feature Milestones

Kafka Output

• [Fleet] Implement Kafka output form UI kibana#143324
• Make Kerberos support GA for Kafka output #168
• Performance test the shipper Kafka output #232
• Add "real" selector support for kafka output #203
• Add support for metrics observers, add output metrics:

  elastic-agent-shipper/output/kafka/client.go

  Line 41 in de8cd74

  //observer outputs.Observer TODO: what to do with observers?

• Add support for the retrying failed events:

  elastic-agent-shipper/output/kafka/client.go

  Lines 48 to 49 in de8cd74

  	failed []*messages.Event // TODO: Need to know how to deal with failed events
  	// batch publisher.Batch // and how to retry after a batch is complete.

• Support event codecs:

  elastic-agent-shipper/output/kafka/client.go

  Line 63 in de8cd74

  writer codec.Codec, // TODO: Proper codec support

• Support partition caching:

  elastic-agent-shipper/output/kafka/partition.go

  Lines 122 to 125 in de8cd74

  	// TODO: Do we need to cache the partition?
  	//if _, err := msg.data.Cache.Put("partition", partition); err != nil {
  	// return 0, fmt.Errorf("setting kafka partition in publisher event failed: %v", err)
  	//}

• Update sarama to the latest version
• Compatibility testing between existing Beats Kafka output configurations and the shipper Kafka output implementation.

Logstash Output

• Implement the Logstash output #129
• Add support for configuring the retry algorithm:

  elastic-agent-shipper/output/logstash/config.go

  Line 29 in de8cd74

  // Backoff Backoff `config:"backoff"` TODO

• Support configuring bulk max size:

  elastic-agent-shipper/output/logstash/config.go

  Line 21 in de8cd74

  // BulkMaxSize int `config:"bulk_max_size"` TODO

• Support load balancing between hosts:

  elastic-agent-shipper/output/logstash/config.go

  Line 20 in de8cd74

  // LoadBalance bool `config:"loadbalance"` TODO

• Support pipeline with the async client:

  elastic-agent-shipper/output/logstash/config.go

  Line 25 in de8cd74

  // Pipelining int `config:"pipelining" validate:"min=0"` TODO support async client

• Support slow start mode:

  elastic-agent-shipper/output/logstash/config.go

  Line 22 in de8cd74

  // SlowStart bool `config:"slow_start"` TODO

• Add event codec support: https://github.com/cmacknz/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/logstash/encode.go#L11-L12
• Compatibility testing between existing Logstash output configurations and the shipper Logstash output implementation.
• Performance test the Logstash output.

Global Processors

• Synchronize the shipper configuration model with what is implemented in the agent #161
• Prototype migrating a single processor from a beat to the shipper elastic-agent#1688
• Migrate the default global Beat processors to the shipper #292
• Migrate all processing from the inputs to the shipper elastic-agent#1689
• Improve processor error handling elastic-agent#1690
• Allow configuring global processors that apply to all events: https://github.com/elastic/ingest-dev/issues/2442.

Endpoint Security Support

• A misbehaving input should not be able to flood the shipper and prevent publishing the events of other inputs #213
• Enable mTLS on connections between inputs and the shipper #275
• Require use of an encrypted disk queue when Elastic Defend is installed

Disk Queue

• enrich diskqueue configuration with encryption_password when disk queue is encryption is enabled elastic-agent#1656
• Add disk queue options and operation to documentation #122
• The disk queue must assign an auto-incrementing ID to each element #27
• Immediately acknowledge events persisted to the disk queue when an input is subscribed to the acknowledgement stream.
• The disk queue must assign an auto-incrementing ID to each element #27
• Correctly handle enqueued events affected by agent policy changes #49
• Add explicit memory usage cap for the disk queue beats#30069
• Add disk queue metrics to shipper metrics #125
• Implement queue metrics for the disk queue #145

Future Enhancements - Not Yet Planned

• Explore an autoscaling Elasticsearch output based on the APM Server autoscaling work #175
• Investigate changing the pipeline order to improve performance #44
• Deadletter indexing should be taken care of automatically #245

[softengchick]

Brian McKinney and Michelle Bennett will be working this for Endpoint. @brian-mckinney @softengchick

[cmacknz]

@softengchick @brian-mckinney is there an issue for this on the endpoint side we can follow?

If you have not seen it recently, the gRPC API for the shipper is defined at https://github.com/elastic/elastic-agent-shipper-client/tree/main/api. The current API is meant to be generic but it is tailored to the needs of Beats since that is what we started with. It is possible we missed something that could make integrating with Endpoint easier.

Reach out once you are ready to start work with any questions you have. I expect this will be a larger change for Endpoint than the move to the V2 control protocol since it isn't just iterating on something Endpoint already supports.