fluent-bit does not repsect the DB path in the input section

[yiwenshao]

Bug Report

Describe the bug
I am using fluent-bit 1.14 with the following configuration:

[INPUT]
    Name tail
    storage.type  filesystem
    Path /data/log1
    DB /fluentbit-pos/pods
    Tag rule1.*

[INPUT]
    Name tail
    storage.type  filesystem
    Path /data/log2
    DB /fluentbit-pos/pods
    Tag rule3.*

[INPUT]
    Name tail
    storage.type  filesystem
    Path /data/log3
    DB /fluentbit-pos/pods2
    Tag rule2.*

[OUTPUT]
    Name  stdout
    Match *

I do find log in the output. However, under directory /fluentbit-pos, only one pos file pods is generated. Where I inspect the content using sqlite, I find that /fluentbit-pos/pos contains offset info of /data/log1,/data/log2, and /data/log3.

Expected behavior
/fluentbit-pos/pods should only contain offset info of /data/log1 and /data/log2. There should be another file /fluentbit-pos/pod2 which tracks /data/log3.

---

Also, I try to change my configuration to

[INPUT]
    Name tail
    storage.type  filesystem
    Path /data/log1
    DB /fluentbit-pos/pods
    Tag rule1.*

[INPUT]
    Name tail
    storage.type  filesystem
    Path /data/log2
    DB /fluentbit-pos/pods3
    Tag rule3.*

[INPUT]
    Name tail
    storage.type  filesystem
    Path /data/log3
    DB /fluentbit-pos/pods2
    Tag rule2.*

[OUTPUT]
    Name  stdout
    Match *

This time, fluentbit work as expectd. three files are generated under /fluentbit-pos. /fluentbit-pos/pods, /fluentbit-pos/pods3, /fluentbit-pos/pods2 tracks /data/log1, /data/log2, /data/log3 respectively.

[edsiper]

We don't use a pos file as Fluentd does, we use SQLite as a backend database. If you set the same database path, it will be shared. Just set different database files for different sections if that is desired.

[yiwenshao]

Thanks for your reply.
The issue is that I set different database paths, but the database file is still shared.
The first configuration above contains three input sections, the first and second section share database path /fluentbit-pos/pods and the third input section has database path /fluentbit-pos/pods2, so I expect two database files to be created. However, only one database file /fluentbit-pos/pods is created, and this file tracks /data/log1, /data/log2, and /data/log3. This is strange because I expect /fluentbit-pos/pods to trace only /data/log1, and /data/log2. There should be another database file /fluentbit-pos/pods2, which tracks /data/log3.

In the second configuration I specified three database files /fluentbit-pos/pods /fluentbit-pos/pods2 and /fluentbit-pos/pods3.

This time, three database files are generated, and fluent-bit work as expected.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.