Fluentd filter to process Kubernetes logs

[amitkumarj441]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind feature

What happened:
I and @mperezco are working on having a Fluentd filter to process kubernetes logs, we created the ConfigMap to add two files (in ingress):
@include configs.d/user/filter-k8s-core-rewrite.conf [First file is to rewrite the logs coming from services "origin-master" and "origin-node"]
@include configs.d/user/filter-k8s-core-data.conf [Second file is to tag all kubernetes namespaces]

We already have a common data model namespace for kubernetes https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml#L22
So we could do something like this instead:
<record>
kubernetes {"namespace_name":"${record['MESSAGE'].match/.*GET\s\/api.*namespaces\/(\w*)\//}"}
</record>
But the problem is - what if there is no match for /.*GET/ ? What will the namespace name be?

Perhaps record['MESSAGE'] is not coming through as expected?

And we're not getting any logs under Kibana with API tags?

What you expected to happen:
To get Kubernetes components logs (kubeapi, kubelet etc.) in journald

How to reproduce it (as minimally and precisely as possible):
We'll try another way to conditionally add a field:
<record>
unused ${record['MESSAGE'].match(/.*GET\s\/api.*namespaces\/(\w*)\//) do |md| record['kubernetes']=Hash.new; record['kubernetes']['namespace_name']=md[1] end}
</record>
remove_keys unused

We tested the regular expression in Rubular as well as in Fluentular
The intention is that in a line that comes from journald like this:
ago 21 18:19:12 viaq.logging.test origin-master[1194]: I0821 18:19:12.249226 1194 panics.go:76] GET /api/v1/namespaces/logging/endpoints/logging-kibana: (10.89128ms) 200 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4 system:serviceaccount:openshift-infra:endpoint-controller] 192.168.122.5:47476 http://192.168.122.5:47476]`
assigns the record "logging" to "k8s_namespace" ... and shows it in kibana ...
Anything else we need to know?:
We checked our expression according to ruby console which is correct though :

pry(main)> msg = 'ago 21 18:19:12 viaq.logging.test origin-master[1194]: I0821 18:19:12.249226 1194 panics.go:76] GET /api/v1/namespaces/logging/endpoints/logging-kibana: (10.89128ms) 200 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4 system:serviceaccount:openshift-infra:endpoint-controller] 192.168.122.5:47476 http://192.168.122.5:47476]' => "ago 21 18:19:12 viaq.logging.test origin-master[1194]: I0821 18:19:12.249226 1194 panics.go:76] GET /api/v1/namespaces/logging/endpoints/logging-kibana: (10.89128ms) 200 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4 system:serviceaccount:openshift-infra:endpoint-controller] 192.168.122.5:47476 http://192.168.122.5:47476]"`

pry(main)> msg.match(/.*GET\s\/api.*namespaces\/(\w*)\//) do |md| puts md[1] end
logging
=> nil

Other solution could be to add out_stdout in many cases to debug.

Environment:

• Kubernetes version (use kubectl version):

• Cloud provider or hardware configuration**:

• OS (e.g. from /etc/os-release): CentOS 7.3

• Kernel (e.g. uname -a): Linux ith-ThinkPad-W520 4.10.0-32-generic Expand testing of the util package. Now 70% #36~16.04.1-Ubuntu SMP Wed Aug 9 09:19:02 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: ViaQ wrapper

• Others:

[amitkumarj441]

/area logging
/sig cluster-ops
/kind feature

[crassirostris]

@amitkumarj441 Sorry, I don't understand the question/ask :(

[amitkumarj441]

@crassirostris I want kubernetes components logs like kube-apiserver, kube-proxy, kube-dns, kubelet, kube-controller-manager, or kube-scheduler in journald basically I have to find where are getting their logs sent (at least some to journald) and to structure Kubernetes logs .
Please have a look in Kibana (on the sidebar), here I'm not getting any API tags(for kube-apiserver).

P.S.: You can find further explanation in description above in the issue itself.

[crassirostris]

See #39800 for details why it's not currently possible in some setups

You can always configure system components to log to stdout in your setup and configure the container runtime to send logs from containers to journald

[mmmmmmpc]

In the current deployment the services are started by systemd as "origin-node" and "origin-master".
The logs can be seen in journald, as for example all access to API to kubernetes master.
Please note that in this case we are not looking for any log coming from a container, but from a service, and that they are all delivered via journald.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close