Provide the relevant subset of the agent policy to the shipper as its configuration

[cmacknz]

An initial configuration file for the shipper has been defined at https://github.com/elastic/elastic-agent-shipper/blob/main/elastic-agent-shipper.yml, but it does not yet include all of the information required for the shipper to run.

The shipper should receive the subset of the agent policy that applies to each instance of a shipper process started by the agent.

Consider a simple agent policy like https://gist.github.com/blakerouse/88c9e461c031fc089e0f8601b78b48de. Then each defined output will map to an instance of the shipper process that should be provided with:

• The contents of the output section for the started shipper.
• The contents of the agent.monitoring section to control shipper monitoring.
• For each input configured to use the the applicable shipper instance that defines a processor, the relevant sub-set of the input definition. The shipper needs at least the input ID, data stream identifiers, and processor configurations.

For the sample agent policy above, we should receive at least the following configuration almost verbatim from the agent policy.

outputs:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: default-system-policy
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth-default-system-policy
        data_stream:
          dataset: system.auth
          type: logs
        processors:
          - add_locale: null
      - id: logfile-system.syslog-default-system-policy
        data_stream:
          dataset: system.syslog
          type: logs
        processors:
          - add_locale: null
  - id: default-system-policy
    data_stream:
      namespace: default
    streams:
      - id: system/metrics-system.filesystem-default-system-policy
        data_stream:
          dataset: system.filesystem
          type: metrics
        processors:
          - drop_event.when.regexp:
              system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.fsstat-default-system-policy
        data_stream:
          dataset: system.fsstat
          type: metrics
        processors:
          - drop_event.when.regexp:
              system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)

Relates to:

• Allow the Elastic Agent to run the shipper elastic-agent-shipper#13
• Inject Beat gRPC stream identifiers without relying on agent specification file transformations beats#33135
• [Meta][Feature] The agent policy's input processor configurations must be provided to the shipper. #220
• [Meta][Feature] Start the shipper process(es) and configure inputs to write to them when the shipper feature flag is enabled. #219

[cmacknz]

CC @rdner as this will likely define how the shipper receives the processor list from the agent.

CC @fearful-symmetry since the configuration format should be finalized to complete elastic/elastic-agent-shipper#13

[rdner]

In my proposal document I had the following example:

outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
inputs:
  - id: default-system-policy # unique for the entire policy
    use_output: default       # output is declared only on the input level
    processors:               # list of processors for the input level
      - add_locale: null
    data_stream:              # data stream parameters of the input level 
                              # (can be overridden by streams)
      namespace: default
      dataset: system.auth
      type: logs
    streams:                  # optional, some inputs don’t have streams
      - id: logfile-system.auth-default-system-policy
        data_stream:          # overrides the input level `data_stream`
          dataset: system.auth
          type: logs
        processors:           # additional processors to what declared 
                              # on the input level
          - add_locale: null

This is the bare minimum we would need in the shipper, it does not have to be exactly that but it's something we could start the discussion with.

[fearful-symmetry]

Also don't forget monitoring/debug config, which will need to live somewhere in here as well.

Also an interesting question is where the source of truth for this configure structure will live, if there is one. Ideally, there would be a single protobuf file that lives somewhere, and is imported by the shipper, and agent, and whatever components care about sending/receiving that config. Not sure if that's possible with the current stack of the entire stack.

Also, if this config is (primarily) generated based on some web input in Kibana's Fleet UI, I'm wondering at which point it doesn't really make sense to be storing all this in YAML, which is considerably more cumbersome to deal with in debug environments compared to something non-whitespace sensitive like JSON.

[cmacknz]

OK, I think it's clear we need the output, agent.monitoring, and a slightly stripped down section of the inputs section that includes the input IDs, and stream IDs with their data streams and processors per Denis' comment above.

The snippet in the issue description looks like it captures all of this now.

[cmacknz]

@fearful-symmetry I believe the agent policy is stored as a Kibana saved object by fleet, see https://github.com/elastic/kibana/blob/60e7f1dcd120c5512506050611d99c3455cbd912/x-pack/plugins/fleet/server/saved_objects/index.ts#L131 for example.

If you look around https://github.com/elastic/kibana/tree/main/x-pack/plugins/fleet/dev_docs and the fleet packages you can follow how fleet handles policies.

[cmacknz]

Unless there's anything else we want to add to this I'll convert this into an issue in the agent repository for the control plane team to send us the agent policy sections we care about.

[cmacknz]

This issue is now the specification of what the agent should provide to the shipper. I've transferred to the agent repository and tracked it under #214.

CC @ph @blakerouse

[cmacknz]

We spoke today and decided the simplest initial path is for the agent to provide the shipper with the entire agent policy, and the shipper can parse the sections it requires.

[jlind23]

Closing this as the shipper project has been revisited.