Kubernetes Pod name and container ID

[cricketfan5]

I am new to splunk and kubernetes. I am able to setup a kubernetes with universal splunk forwarder. I am able to see logs in the UI. I don't see the name of the pod or container from which it is forwarding. Is there a way to configure to get the pod name and container id in the logs?.

[ae6rt]

One possible way is to use search-time extracted fields in your props.conf file on the indexer:

EXTRACT-sourcefields = /var/log/containers/(?<pod>.*)_(?<namespace>.*)_(?<containerName>.*)-(?<containerID>.*)\.log in source

[cricketfan5]

@ae6rt Thanks for the response. I have followed the kubernetes readme file and I have installed a daemonset as per the document. where would be the props.conf file.

[ae6rt]

If you are starting the indexer as a DaemonSet, you would include, most likely, a props.conf file as part of a ConfigMap for the indexer pod. You would then mount the ConfigMap in a location where the indexer expects its config files.

[cricketfan5]

I followed the same yaml file for creating splunk forwarder as a daemon set. But don't see anything as a indexer in the file. Can you please help me figuring this?. Thanks

[cricketfan5]

I see a props.conf file in the splunkforwarder/apps-k8 folder. Is that the one are you talking about ?.

[ae6rt]

This one: https://github.com/splunk/docker-itmonitoring/blob/master/app-k8s/default/props.conf#L7. Put your extracted fields instruction within this kubernetes sourcetype.

[cricketfan5]

Thanks

[cricketfan5]

I tried updating the values in the props.conf. But still not able to see the host name and pod name and container name anything as such. What might the wrong ?. Thanks

[matthewmodestino]

Hey crickerfan5,

Where are you sending your events to? The Splunk instance in the full demo yaml? Or one outside the cluster? The props/transforms need to be on the indexer, so make sure you are updating them there.

I'll look at updating the repo with the props.conf/transforms to create indextime fields for pod, namespace, container_id, which would look something like this:

in transforms.conf ON YOUR INDEXER:

[k8s_extractions]
SOURCE_KEY = MetaData:Source
DEST_KEY = _meta
REGEX = ^source::/var/log/containers/([^_]+)_([^_]+)_(.*)-([^\-]+)\.log$
FORMAT = $0 pod::$1 namespace::$2 container::$3_$4
WRITE_META = true

in the Props.conf ON YOUR INDEXER add this line pointing to the transform:

TRANSFORMS-k8s_extractions = k8s_extractions

I would suggest you check out our recently released Splunk Connect for Kubernetes here:

https://github.com/splunk/splunk-connect-for-kubernetes

As this is our currently supported method of getting info out of k8s and will take care of said extractions for you.

[cricketfan5]

@matthewmodestino Thanks for the response. Do we need to create a new transforms.conf file?. And I am sending the logs outside the cluster.

[cricketfan5]

@matthewmodestino I have created a transforms.conf and added the text and updated the props.conf file too. But still I don't see any changes in the log in splunk UI. Any help is appreciated. Thanks

[matthewmodestino]

Where are you making these changes? Directly in the container? Making these changes will likely require you to delete the pod and re-deploy.

join us in #kubernetes on slack at splk.it/slack and we can help in more real-time. my username is @mattymo

[cricketfan5]

@matthewmodestino I am not able to find the slack channel. I signup into splk.it/slack.
I made changes on the indexer side and created a new image built the daemonset.

[matthewmodestino]

visit splk.it/slack in your browser, it will take you to a google form for you to request access.
The actual Slack channel is splunk-usergroups.slack.com

When you say indexer side, where is that indexer? did you only deploy the daemonset? to k8s, or is the indexer inside the cluster too?