Designing the input configuration

[jasonrhodes]

We discussed this in another repo and came up with the following examples.

Previously Specified Options

There are multiple ways how behind the scene asset collection can be implemented (cloudbeat, metricbeat, assetbeat, inputs v2). No matter the implementation, the elastic agent configs for asset collection should always look the same. This issue is to discuss how the config should look like.

Config proposals

The following is a list of potential options to get the discussion started.

A) Single asset input

A single input is configured to collect assets and with it the types of assets that should be collected:

inputs:
- type: assets
  asset_types: ["k8s/pods", "k8s/nodes"]

B) Input per collection target

An input is created per collection target. The assumption is, that this target has shared configuration options like access key / secret for AWS, simliar for k8s.

inputs:
- type: assets/k8s
  asset_types: ["pods", "nodes"]
- type: assets/aws
  asset_types: ["s3", "ec2"]

This also allows configs to collect all assets by default:

inputs:
- type: assets/k8s
  # This would be the default an not needed
  asset_types: "*"

C) Input per asset

Each asset collection has an input specified:

inputs:
- type: assets/k8s/pods
- type: assets/k8s/nodes
- type: assets/aws/s3

Personal preference

My personal preference is with B as it is a good middle ground. It allows to share connection configs in an easy way and to still specify * patterns to collect all assets (not possible with C).

There is a good chance that not all assets should be collected with the same frequency. This leads to a config as following:

inputs:
- type: assets/k8s
  asset_types: ["nodes"]
  period: 1m
- type: assets/k8s
  asset_types: ["pods"]
  period: 10s
- type: assets/aws
  asset_types: ["s3", "ec2"]

My ideal scenario is that eventually there is a subscribe mechanism for asset changes which means period becomes obsolete.

Examples

Here's a more complete config example, mostly drawing from Option B above, with some inline questions about how we might want to adjust. I've shown some credentials examples here based on what I see in existing AWS integrations, just for example's sake.

inputs:
  # Note: you would never have more than one of k8s-api input in a given policy, but you might have
  # them in different policies for different agents, deployed into your different K8s clusters.
  - type: assets/k8s-api
    name: assets-k8s-api
    asset_types:
      - k8s.pod
      - k8s.node
      - k8s.cluster
      - container
    api_host: 'https://${env.NODE_NAME}:10250'
    period: 1h
  # TBD: all AWS assets in a single input, or separate AWS K8s vs AWS ... other?
  - type: assets/aws
    name: assets-aws
    asset_types:
      - k8s.cluster
      - host_node # aws ec2
      - vpc
      - s3.bucket # and/or others?
    access_key_id: jason-access-key-id-test
    secret_access_key: jason-secret-access-key-test
    session_token: jason-session-token-test
    project: 
      name: 'elastic-observability'
      id: 12345678
    regions:
      - us-east-1
      - us-west-1
    period: 2h
  - type: assets/aws
    name: assets-aws
    asset_types:
      - k8s.cluster
    access_key_id: jason-access-key-id-test2
    secret_access_key: jason-secret-access-key-test2
    session_token: jason-session-token-test2
    project: 
      name: 'other-aws-project'
      id: 98765432
    regions:
      - us-west-1
    period: 2h
  - type: assets/azure
    name: assets-azure
    asset_types:
      - k8s.cluster
      - host_node # azure vm
      - vpc
    credentials: whatever
    subscription_id: 12341234xyz
    period: 4h

I started an example of the other options, e.g. Option C:

inputs:
  - type: assets/k8s.pods
    name: assets-k8s-pods
    api_host: 'https://${env.NODE_NAME}:10250'
    period: 1h
  - type: assets/k8s.nodes
    name: assets-k8s-nodes
    api_host: 'https://${env.NODE_NAME}:10250'
    period: 4h
  - type: assets/k8s.cluster
    name: assets-k8s-cluster
    api_host: 'https://${env.NODE_NAME}:10250'
    period: 4h
  - type: assets/aws:k8s.clusters
    name: assets-aws-k8s-clusters
    access_key_id: jason-access-key-id-test
    secret_access_key: jason-secret-access-key-test
    session_token: jason-session-token-test
    project: 
      name: 'elastic-observability'
      id: 12345678
    regions:
      - us-east-1
      - us-west-1
    period: 2h
  - type: assets/aws:k8s.clusters
    name: assets-aws-k8s-clusters
    access_key_id: jason-access-key-id-test2
    secret_access_key: jason-secret-access-key-test2
    session_token: jason-session-token-test2
    project: 
      name: 'other-aws-project'
      id: 98765432
    regions:
      - us-west-1
    period: 2h
  - type: assets/azure:k8s.clusters
    name: assets-azure-k8s-clusters
    credentials: whatever
    subscription_id: 12341234xyz
    period: 4h

[tommyers-elastic]

agree with option B @jasonrhodes.

is there anything we need to be aware from the agent side to implement this correctly? AFAICT, once we write a shared config struct for assets it should 'just work'?