Duplicate K8s API Logs

[salilgupta1]

We have an issue where we are collecting api logs twice. This is because our K8s API servers are deployed a containers on the control plane. Therefore, fluentd collects the container logs from /var/log/containers and because of this source https://github.com/SumoLogic/fluentd-kubernetes-sumologic/blob/master/conf.d/file/source.kubernetes.conf#L74 fluentd collects the api server logs from a log file. This is a problem since it significantly increases our intake and we're hitting overage fees.

We can do the following but none are ideal:

1. We could add the path to the api server logs in our EXCLUDE_PATHS env var. However, I prefer those logs since they are enriched with Kubernetes metadata, which makes them much more useful for debugging
2. We could fork the repo and make the necessary changes ourselves. I think the solution would be beneficial to more than us.

Maybe there is something else I can do?

[frankreno]

@salilgupta1 thank you for the issue. I agree 1 is not ideal, we need a better way to solve this to prevent the duplication. Let me see what I can do here.

[frankreno]

@salilgupta1 ok wanted to give you an update on this issue. In order to prevent this duplication, and ensure the logs we capture have all the enriched metadata, I am going to remove the duplicated sources from the appropriate configs. However, this constitutes a breaking change as 1) some folks may be using those log lines and 2) our current Kubernetes app uses these log lines and not the container log lines.

So the plan to address this is the following:

1. I will work on fixing our application to use the container log lines instead of the current source. There are some parsing changes that need to happen as the log format is slightly different.

2. When the app changes are done, I will fix the FluentD plugin to remove these duplicated sources and cut a major release for the plugin (v2.0.0).

3. I will add notes to our docs and the readme detailing the change and the impact and how to correct any searches/content you may have looking at the sources we are going to remove.

The plan is to have this all complete by next week.

[salilgupta1]

@frankreno any updates on your end here? Could we release this fix and then you all can migrate your Kubernetes integrations onto the new version?

[frankreno]

@salilgupta1 I apologize for the delay. We will be pushing the changes out in the next couple of days.

[frankreno]

This has been fixed in v2.0.0