EKS control plane logs should have their source based on the log stream name

[robmp]

While the changes introduced in #371 and #365 that set EKS control plane logs source to eks are an improvement, I think they're not sufficient.

EKS log groups in CloudWatch have multiple log streams, e.g:

These come from different control plane components and have different formats (some glog, others json).

I think it would make sense to have the log source based on the log stream name, e.g. kube-apiserver, kube-scheduler etc.

[tianchu]

@robmp are you suggesting have the source to be kube-apiserver or kube-scheduler?

[robmp]

@robmp are you suggesting have the source to be kube-apiserver or kube-scheduler?

I'm suggesting that the source should be the same as the log stream name (without the random suffix).

[tianchu]

@robmp Thanks for your clarification! I think it makes more sense to keep source being eks, while have what you proposed in a new attribute, e.g., kube_service: kube-apiserver (just an example).

In fact, it would be better to add this in the default eks log pipeline as a Grok parser rule, which would provide a consistent user experience for eks logs ingested through other means (e.g., datadog kinesis firehose). In fact, you can update the log pipeline in your Datadog account that processes eks logs, add a Grok parser rule to parse the log stream name and get the result to a new field for query. To make it an OOB user experience, please submit a FR to support@datadoghq.com.

I'm going to close the issue for now, since I believe we would NOT make this change in the forwarder.

[pawelpesz]

The current handling of EKS CloudWatch logs is really inconsistent across the Datadog infrastructure. Only the audit and scheduler logs are correctly recognised and a corresponding built-in pipeline is added for them, respectively "Kubernetes audit" and "Kube Scheduler (glog)". This is covered by the #406 PR which you @tianchu reviewed some months after this issued was closed.

Datadog expects a source:kube-apiserver in at least two places I was able to locate:

• Default glog pipeline
• Kubernetes API Server Overview dashboard

It would be great if you could take another look at this issue.