Example for rewrite-tag getting stuck locally

[jmdangelo]

Bug Report

Describe the bug
I've downloaded fluent-bit through homebrew on a mac (i've also compiled the binaries and tried that) and tried replicating the results seen in this docs page rewrite-tag filter for how to use rewrite-tag. When i run it with the config given at the bottom of the page, i don't get any output.

fluentbit fluent-bit -c example.conf
Fluent Bit v1.5.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/09/23 17:33:53] [ info] [engine] started (pid=22553)
[2020/09/23 17:33:53] [ info] [storage] version=1.0.5, initializing...
[2020/09/23 17:33:53] [ info] [storage] in-memory
[2020/09/23 17:33:53] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/09/23 17:33:53] [ info] [sp] stream processor started
^C[engine] caught signal (SIGINT)
[2020/09/23 17:34:07] [ info] [input] pausing re_emitted

if i change

Rule          $tool ^(fluent)$  from.$TAG.new.$tool.$sub['s1']['s2'].out false

to

Rule          $tool ^(fluent)$  from.$TAG.new.$tool.$sub['s1']['s2'].out true

and tell output to match * I begin to get the the original log lines come through

fluentbit fluent-bit -c example.conf
Fluent Bit v1.5.6
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/09/23 17:36:12] [ info] [engine] started (pid=22942)
[2020/09/23 17:36:12] [ info] [storage] version=1.0.5, initializing...
[2020/09/23 17:36:12] [ info] [storage] in-memory
[2020/09/23 17:36:12] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/09/23 17:36:12] [ info] [sp] stream processor started
[0] test_tag: [1600907773.972616000, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[0] test_tag: [1600907774.973662000, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[0] test_tag: [1600907775.975126000, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
^C[engine] caught signal (SIGINT)
[2020/09/23 17:36:17] [ info] [input] pausing re_emitted

It seems to me that it is getting to the filter step, but failing in some way and not outputting the resulting messages with the new tag, if I am just missing something completely that would be great.
To Reproduce

• Steps to reproduce the problem:
  1 download fluent-bit using homebrew
  2 run fluent-bit using this configuration file

[SERVICE]
    Flush     1
    Log_Level info

[INPUT]
    NAME   dummy
    Dummy  {"tool": "fluent", "sub": {"s1": {"s2": "bit"}}}
    Tag    test_tag

[FILTER]
    Name          rewrite_tag
    Match         test_tag
    Rule          $tool ^(fluent)$  from.$TAG.new.$tool.$sub['s1']['s2'].out false
    Emitter_Name  re_emitted

[OUTPUT]
    Name   stdout
    Match  from.*

check if there is any output with the new tag

Expected behavior
I would expect the original tag test_tag will be rewritten as from.test_tag.new.fluent.bit.out:

Screenshots

Your Environment

• Version used: v1.5.6
• Configuration: listed above
• Environment name and version (e.g. Kubernetes? What version?): local on mac osx version 10.14.6
• Server type and version: local
• Operating System and version: mac osx version 10.14.6
• Filters and plugins: listed in configuration file

Additional context
I was attempting to use fluentbit with firelens to send log messages matching a certain text in the log to a firehose delivery stream then to an s3 bucket. I was having trouble getting the [filter] portion to work and it would just catch all logs that match the regex pattern and leave them hanging. I tried to do some local development to learn more about filters and how to use them, but couldn't get this example to work

[edsiper]

I tried your config and it works here:

$ ../../build/bin/fluent-bit -c fluent-bit.conf 
Fluent Bit v1.6.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/09/23 19:06:45] [ info] [engine] started (pid=2011850)
[2020/09/23 19:06:45] [ info] [storage] version=1.0.4, initializing...
[2020/09/23 19:06:45] [ info] [storage] in-memory
[2020/09/23 19:06:45] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/09/23 19:06:45] [ info] [sp] stream processor started
[0] from.test_tag.new.fluent.bit.out: [1600909606.044451276, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[0] from.test_tag.new.fluent.bit.out: [1600909607.044405473, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
[0] from.test_tag.new.fluent.bit.out: [1600909608.045690552, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}]
^C[engine] caught signal (SIGINT)
[2020/09/23 19:06:49] [ info] [input] pausing re_emitted

[jmdangelo]

Used the official aws-for-fluent-bit docker image to test this morning, seems like it only doesn't work on local installation

[edsiper]

which Fluent Bit version uses aws-for-fluent-bit docker ?

[jmdangelo]

which Fluent Bit version uses aws-for-fluent-bit docker ?

it was fluent-bit 1.5.6, same as my homebrew installation

[AnujAroshA]

I'm getting the same issue with fluent-bit 1.6.0 & 1.6.1
Tested locally on macOS Catalina (Version 10.15.7)

[AnujAroshA]

Can confirm this issue is related to macOS. Because I tried with an Ubuntu VM (20.04.1) and this works. @edsiper glad we can get a fix for macOS as well.

[pavel-khritonenko]

Can confirm that it still doesn't work in OSX (fluent-bit version 1.7.2)

[interfan7]

Same here, it doesn't work on fluent-bit 1.7.2 over newest OSX (some big-sur).

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.