Attach original tag for 'mounted-file'

[cw-sakamoto]

When using mounted-file it will be tag kube.{namespace}.{pod_name}.{container_name}, but I'd like to attach original tag(kube.{namespace}.{pod_name}.{container_name}.mytag).

example:

<source>
  @type mounted-file
  path /var/log/hello.log
  labels msg=hello
  tag hello-log
</source>

-> kube.{namespace}.{pod_name}.{container_name}.hello-log

Why?

I'd like to separate output process in the container with each tags.

example:

namespace: test
<source>
  @type mounted-file
  path /var/log/hello1.log
  labels msg=hello
  tag hello1-log
</source>
<source>
  @type mounted-file
  path /var/log/hello2.log
  labels msg=hello
  tag hello2-log
</source>

<match **.hello1-log>
  @type test123
</match>

<match **.hello2-log>
  @type test456
</match>

namespace: kube-system
<plugin test123>
  @type logzio_buffered
  endpoint_url https://listener.logz.io:8071?token=TOKEN&type=log-router
  buffer_path /some/path
</plugin>

<plugin test456>
  @type logzio_buffered
  endpoint_url https://listener.logz.io:8071?token=TOKEN&type=log-router
  buffer_path /some/path456
  buffer_size 1m
</plugin>

If you can do with the current specifications, please let me know.

[jvassev]

Hi,
In general kube-fluentd-operator tries to step away from the fluentd tag. Instead, it promotes Kubernetes labels.

In your case however all the logevents come from a single pod meaning they share the same labels. So what can be done here is let users attach new labels to the @type mounted-file logs ike this:

<source>
  @type mounted-file
  path /var/log/hello1.log
  labels msg=hello
  add_labels app=app1
  tag hello1-log
</source>

<source>
  @type mounted-file
  path /var/log/hello2.log
  labels msg=hello
  add_labels app=app2
  tag hello2-log
</source>

Then you use the $labels macro to select the logs:

<match $labels(app=app1)>
  @type test123
</match>

<match $labels(app=app2)>
  @type test456
</match>

Indeed, this would be easy to add - do you think this would help you in your setup?.
However, it would be more "kubernetes-idiomatic" to spread those processes producing different log files into independent pods.

[jvassev]

@cw-sakamoto In my previous comment I assumed you have two log files produced by the same container in a single pod.
If the logs files are produced by different containers then you can solve your case even with the current version by using the _container meta-label:

<match $labels(msg=hello, _container=cont1)>
  @type test123
</match>

<match $labels(msg=hello, _container=cont2)>
  @type test456
</match>

[cw-sakamoto]

@jvassev Thanks for your comments.

the same container in a single pod.

You are right. I assume the same container.

It can be solved by #26 (comment) , however I thought that it is difficult to distinguish from label of k8s itself by this writing style.

In general kube-fluentd-operator tries to step away from the fluentd tag. Instead, it promotes Kubernetes labels.

Under this basic idea, I think that my proposal(add fluentd tag) is difficult, so think a little more...🤔

[cw-sakamoto]

This is just idea...add $labels meta_label _file(or _path) like _container

$labels(msg=hello, _file=/var/log/hello.log)

or

$labels(msg=hello, _path=/var/log/hello.log)

What do you think this idea?

[jvassev]

I thinks it's better to use add_labels because labels values have strict format validation. A path can contain =, ,, which will confuse the $labels macro

[jvassev]

@cw-sakamoto
I've implemented the add_labels feature described earlier. You can give it a try with this image jvassev/kube-fluentd-operator:v1.6.0-17-gbdc4593. It's built from bdc4593.

Just put add_labels sub-component=compXXX to your mounted_files and then you can distinguish between different log files from the same container.

[jvassev]

The add_labels feature is now merged.