exclude namespace kube-system to send logs to ElasticSearch

[viquar22]

Is there a way to have fluentd to exclude namespace "kube-system" not to send logs to Elasticsearch so that we don't see logs from the namespace(kube-system) in Kibana.

I'm trying to add into td-agent.conf so that it will be updated and stop sending logs from the namespace (kube-system) to ES and we will only have logs from other namespaces but from kube-system in Kibana.

Thanks in advance.

[richm]

@viquar22 yes - add a match like this:

<match kubernetes.var.log.containers.**_kube-system_**>
  @type null
</match>

assuming you are reading container log files written by docker --log-driver=json-file

[viquar22]

Yes.
i use gitlab for deployment.
I updated my td-agent with the above config and deployed but still see the logs from "kube-system" in Kibana.

[richm]

@viquar22 not sure - this is a general fluentd problem, not a k8s meta plugin problem - you should ask how to debug this issue in a fluentd forum

[viquar22]

alright. Thanks for your quick response @richm

[viquar22]

@richm Hey your config works for me.
It has stopped sending logs from namespace (kube-system).
But i see logs in Kibana from same namespace (kube-system) but the pods are different.
When i try to describe (kubectl describe . . ) it doesn't exist.. i see those pod logs in the latest time stamp.

The pods i see in Kibana do not match with the ones i see in the Terminal (kubectl -n kube-system get pods). i am able to describe and login to the pods i see in the terminal and they have updated td-agent configuration.. But Kibana pods doesn't exist... how come it's possible...

I believe those Pods in Kibana are old pods that are still exist somewhere in the buffer(don't know where) and getting logs from them with latest timestamp.

Please suggest. Thanks in advance..

[richm]

@viquar22 I don't know - it could be many things - but I don't think this closed issue is the right place to discuss - try a kubernetes forum or a fluentd forum