From c5a89f9fb29c735285970c05af1303b1020aa960 Mon Sep 17 00:00:00 2001 From: Patrick Stephens Date: Tue, 23 Sep 2025 12:23:21 +0100 Subject: [PATCH 1/3] tail: ensure db file is unique Signed-off-by: Patrick Stephens --- pipeline/inputs/tail.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 0a137fb6e..c3bcaf8be 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -22,7 +22,7 @@ The plugin supports the following configuration parameters: | `ignore_older` | Ignores files older than `ignore_older`. Supports `m`, `h`, `d` (minutes, hours, days) syntax. | Read all. | | `skip_long_lines` | When a monitored file reaches its buffer capacity due to a very long line (`buffer_max_size`), the default behavior is to stop monitoring that file. `skip_long_lines` alter that behavior and instruct Fluent Bit to skip long lines and continue processing other lines that fit into the buffer size. | `off` | | `skip_empty_lines` | Skips empty lines in the log file from any further processing or output. | `off` | -| `db` | Specify the database file to keep track of monitored files and offsets. | _none_ | +| `db` | Specify the database file to keep track of monitored files and offsets. Recommended to be unique per plugin. | _none_ | | `db.sync` | Set a default synchronization (I/O) method. This flag affects how the internal SQLite engine do synchronization to disk, for more details about each option see [the SQLite documentation](https://www.sqlite.org/pragma.html#pragma_synchronous). Most scenarios will be fine with `normal` mode. If you need full synchronization after every write operation set `full` mode. `full` has a high I/O performance cost. Values: `extra`, `full`, `normal`, `off`. | `normal` | | `db.locking` | Specify that the database will be accessed only by Fluent Bit. Enabling this feature helps increase performance when accessing the database but restricts externals tool from querying the content. | `false` | | `db.journal_mode` | Sets the journal mode for databases (`wal`). Enabling `wal` provides higher performance. `wal` isn't compatible with shared network file systems. | `wal` | @@ -75,6 +75,8 @@ If no database file is present, positioning behavior depends on the value of `re - When `read_from_head` is `true`, the plugin reads from the beginning of the file. - When `read_from_head` is `false`, the plugin starts monitoring from the end of the file (classic "tail" behavior). This means that only new content written after Fluent Bit starts will be monitored. +The database file essentially stores `inode=offset` so it is recommended to be unique per instance of the plugin, e.g. if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state. + ## Monitor a large number of files To monitor a large number of files, you can increase the `inotify` settings in your Linux environment by modifying the following `sysctl` parameters: From 18f5ca7e469f0970e4c24c62c3a105165988443b Mon Sep 17 00:00:00 2001 From: Patrick Stephens Date: Tue, 23 Sep 2025 13:05:43 +0100 Subject: [PATCH 2/3] fix: resolve review comments Signed-off-by: Patrick Stephens --- pipeline/inputs/tail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index c3bcaf8be..9978c5572 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -75,7 +75,7 @@ If no database file is present, positioning behavior depends on the value of `re - When `read_from_head` is `true`, the plugin reads from the beginning of the file. - When `read_from_head` is `false`, the plugin starts monitoring from the end of the file (classic "tail" behavior). This means that only new content written after Fluent Bit starts will be monitored. -The database file essentially stores `inode=offset` so it is recommended to be unique per instance of the plugin, e.g. if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state. +The database file essentially stores `inode=offset` so it is recommended to be unique per instance of the plugin, for example if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state. ## Monitor a large number of files From 75ce3523a87ac5353b3c899b58796e3bcaedd853 Mon Sep 17 00:00:00 2001 From: Patrick Stephens Date: Tue, 23 Sep 2025 13:18:18 +0100 Subject: [PATCH 3/3] fix: resolve review comments Signed-off-by: Patrick Stephens --- pipeline/inputs/tail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 9978c5572..9d56320b7 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -75,7 +75,7 @@ If no database file is present, positioning behavior depends on the value of `re - When `read_from_head` is `true`, the plugin reads from the beginning of the file. - When `read_from_head` is `false`, the plugin starts monitoring from the end of the file (classic "tail" behavior). This means that only new content written after Fluent Bit starts will be monitored. -The database file essentially stores `inode=offset` so it is recommended to be unique per instance of the plugin, for example if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state. +The database file essentially stores `inode=offset` so it should be unique per instance of the plugin, for example if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state. ## Monitor a large number of files