Implement tag-rewrite feature

[dawidmalina]

Hi

I'm evaluating fluentd-bit and it looks very promissing. I know that record_modifier was just merged and i would like to convert my old config to fluentd-bit way. Is it possible to convert this fluentd config part to fluentd-bit?

<filter kubernetes.**>
  @type record_modifier
  <record>
    enable_ruby true
    tag ${ record["logger_name"] ? "json" : "flat" }.kubernetes.${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["pod_name"]}-${record["kubernetes"]["container_name"]}
  </record>
</filter>
<match kubernetes.**>
  @type rewrite_tag_filter
  rewriterule1 tag (.+)       goto.$1
</match>

[edsiper]

@dawidmalina the tag rewrite plugin was just sent in a PR but have not been reviewed yet.

The type of string composing in a configuration is not supported yet, likely to be implemented for 0.13 version (we are releasing 0.11.9 today and 0.12 ~next week)

[edsiper]

Renaming this topic to set it as an enhancement request

[mpeltonen]

Fluentd rewrite_tag_filter functionality would be important for us too, to be able to completely replace fluentd as a forwarder. One usecase is multiple docker containers sending to single forward input with fluentd logging driver and containers can have different needs for parsing formats and may also have different stdout and stderr formats.

For example nginx container needs to separate stderr and stdout for matching parsers for different formats:

<match docker.nginx>
  @type rewrite_tag_filter
  remove_tag_prefix docker
  rewriterule1 source stdout ${tag}.access
  rewriterule2 source stderr ${tag}.error
</match>

[dawidmalina]

@edsiper any news (plans) for this improvement?

[epcim]

I am looking for a possibility to rewrite tag while using another field. for example:

rewriterule1 tag  ${tag}.${component}

[edsiper]

No ETA yet for this feature.

[dawidmalina]

@edsiper any news about this feature?

[dawidmalina]

@edsiper do you think this can be implemented for 0.15 ? I think there are a lot of users looking for this change.

Maybe you could put here some guidelines how this can be implemented and someone could contribute.

[stalkerg]

looks like it will complicate #291
but I think the duplicate record with the different tag will be enough.

[stalkerg]

yes, after checking all sources, much easier way it makes some meta plugin who have one input and many outputs with different tags (after you can be filtering each tag).
Right now we have "static" routing... in the source, it looks like "in_" plugin with filters and output handler we have no something like normal routing in the source at all.

[michaelharrer]

This would be a great feature.
In our case we want to split the output into two different indices, depending on a specific value in a field. Right now it just seems not possible with Fluent-Bit?