New aws_ecs_metadata transform / enrichment

[binarylogic]

As a follow up to #736 (comment), we should provide the ability to enrich logs and metrics data with AWS ECS environment data.

[LucioFranco]

Prior art

https://github.com/michaelgruber/fluent-plugin-ecs-metadata-filter

Configuration

This config should be similar to the aws_ec2_metadata config except it pulls data from the ecs metadata endpoint described here. This proposal will start out to be simple but we can expand it later. There are a couple issues around what docker stats are available depend on deploying ECS via EC2 or Fargate.

That said, initially we will only support a small subset as described below. We should be hitting this endpoint ${ECS_CONTAINER_METADATA_URI}/task that returns JSON.

fields

• cluster
• family
• availability-zone (only available on ec2 launch types)
• container.name
• container.image
• container.label[..] (this should be an array of container labels)

namespace

The namespace field prefixes the added metadata fields with the value provided. The default is no namespace, but if a namespace is provided it should be delimited with a .. Ex:

• Not provided => no namespace

• aws => aws.*

• aws.ecs => aws.ecs.*

• aws_ecs. => aws.ecs..*

container_tag

This should be a string typed option that allows the user to specify what field on the incoming event should be used to correlate container metadata with. By default, this should be tag since this is what the splunk_hec source provides the container tag as.

Output

The output should look something like this:

{
	"cluster": "...",
	"family": "...",
	"availability-zone": "..."
}

Nesting is allowed by using the namespace option with a "." included.

Docs

There are a couple large differences between deploying onto EC2 versus Fargate we should ensure that our documentation makes these differences very clear.

Questions

We also get provided a list of containers but these containers are only associated with a task which means it can vary what is available in this list depending on how users deploy vector. In the common case of using a central vector instance this information is not very useful. So I am curious what might be beneficial here?

cc @ktff

[ktff]

Looks good, especially the idea with the namespace field.

Regarding the question, I think adding extra info for task associated containers and mentioning it, will be fine. As both alternatives, either not adding it or having yet another transform for it, are worse.

[LucioFranco]

Regarding the question, I think adding extra info for task associated containers and mentioning it, will be fine. As both alternatives, either not adding it or having yet another transform for it, are worse.

Agreed, I will update it based on this and what you told me on slack.

[binarylogic]

Closed via #1325