Updated blob input plugin documentation with examples. Fixes #2185. #2186
Conversation
Signed-off-by: Eric D. Schabell <eric@schabell.org>
WalkthroughExpanded and clarified the Blob input plugin documentation: detailed behavior, all configuration parameters, operational notes (database/WAL, storage), use cases, examples (YAML and fluent-bit.conf), and command-line/config usage. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Scanner as Directory Scanner
participant DB as State DB (WAL)
participant Reader as File Reader
participant Parser as Record Creator
participant Forwarder as Fluent Bit Forwarder
Note over Scanner,DB: Periodic scan (scan_refresh_interval)
Scanner->>DB: query processed state
DB-->>Scanner: list unprocessed files
Scanner->>Reader: open & read file
Reader->>Parser: produce record(s) (binary payload)
Parser->>DB: persist processing state (WAL entry)
Parser->>Forwarder: forward record with `tag`
Forwarder-->>Scanner: acknowledge (implicit)
Note over DB: Optional upload_* actions or file post-processing occur after forwarding
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes
Suggested labels
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (5 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
pipeline/inputs/blob.md (1)
16-16: Reduce repetitive sentence structure inmem_buf_limitdescription.Three consecutive sentences begin with "If", which impacts readability. Restructure to improve flow.
-| `mem_buf_limit` | Set a memory buffer limit for the input plugin instance in bytes. If the limit is reached, the plugin will pause until the buffer is drained. If set to `0`, the buffer limit is disabled. If the plugin has enabled filesystem buffering, this limit won't apply. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `0` | +| `mem_buf_limit` | Set a memory buffer limit for the input plugin instance in bytes. When the limit is reached, the plugin will pause until the buffer is drained. Set to `0` to disable the buffer limit. Note: filesystem buffering may override this limit. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `0` |
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
pipeline/inputs/blob.md(2 hunks)
🧰 Additional context used
🪛 GitHub Actions: Lint PRs
pipeline/inputs/blob.md
[warning] 3-3: FluentBit.Simplicity: Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
[warning] 11-11: FluentBit.Simplicity: Avoid words like "Useful" that imply ease of use, because the user may find this action difficult.
[warning] 18-18: FluentBit.Simplicity: Avoid words like "Useful" that imply ease of use, because the user may find this action difficult.
🪛 GitHub Check: runner / vale
pipeline/inputs/blob.md
[warning] 61-61:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/inputs/blob.md", "range": {"start": {"line": 61, "column": 26}}}, "severity": "WARNING"}
[failure] 19-19:
[vale] reported by reviewdog 🐶
[FluentBit.Latin] Use 'for example' instead of 'e.g.,'.
Raw Output:
{"message": "[FluentBit.Latin] Use 'for example' instead of 'e.g.,'.", "location": {"path": "pipeline/inputs/blob.md", "range": {"start": {"line": 19, "column": 293}}}, "severity": "ERROR"}
[warning] 18-18:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "Useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "Useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/inputs/blob.md", "range": {"start": {"line": 18, "column": 171}}}, "severity": "WARNING"}
[warning] 11-11:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "Useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "Useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/inputs/blob.md", "range": {"start": {"line": 11, "column": 104}}}, "severity": "WARNING"}
[warning] 3-3:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/inputs/blob.md", "range": {"start": {"line": 3, "column": 228}}}, "severity": "WARNING"}
🪛 LanguageTool
pipeline/inputs/blob.md
[style] ~16-~16: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...t to 0, the buffer limit is disabled. If the plugin has enabled filesystem buffe...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🔇 Additional comments (1)
pipeline/inputs/blob.md (1)
123-250: Comprehensive examples with realistic configurations.The three example scenarios (database tracking, file exclusion with filesystem storage, and post-processing file actions) effectively demonstrate practical use cases. Concrete paths and properly formatted YAML/conf pairs provide clear implementation guidance.
Signed-off-by: Eric D. Schabell <eric@schabell.org>
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (4)
pipeline/inputs/blob.md (4)
3-3: Replace subjective language "useful" with objective phrasing.This word was flagged in the previous review and should be replaced per the style guide to avoid implying ease of use.
-The _Blob_ input plugin monitors a directory and processes binary (blob) files. It scans the specified path at regular intervals, reads binary files, and forwards them as records through the Fluent Bit pipeline. This plugin is useful for processing binary log files, artifacts, or any binary data that needs to be collected and forwarded to outputs. +The _Blob_ input plugin monitors a directory and processes binary (blob) files. It scans the specified path at regular intervals, reads binary files, and forwards them as records through the Fluent Bit pipeline. This plugin applies to processing binary log files, artifacts, or any binary data that needs to be collected and forwarded to outputs.
11-11: Replace "Useful" with objective language in alias parameter description.This word was flagged in the previous review and violates the style guide.
-| `alias` | Sets an alias for multiple instances of the same input plugin. Useful when you need to run multiple blob input instances with different configurations. | _none_ | +| `alias` | Sets an alias for multiple instances of the same input plugin. When running multiple blob input instances with different configurations, each can be assigned a unique alias for identification. | _none_ |
18-18: Revise routable parameter description for objective language.The current wording "Use this for testing" is imperative rather than objective. Align with the previous review suggestion for clarity.
-| `routable` | If `true`, the data generated by the plugin can be forwarded to other plugins or outputs. If `false`, the data will be discarded. Use this for testing or when you want to process data but not forward it. | `true` | +| `routable` | If `true`, the data generated by the plugin can be forwarded to other plugins or outputs. If `false`, the data will be discarded. Set to `false` for testing scenarios or when processing data without forwarding to outputs. | `true` |
59-66: Revise "Use cases" section heading for style guide compliance.The current phrasing "The Blob input plugin common use cases are:" is grammatically awkward. The previous review suggested more objective phrasing.
-The Blob input plugin common use cases are: +The Blob input plugin applies to the following scenarios:
🧹 Nitpick comments (1)
pipeline/inputs/blob.md (1)
16-16: Reduce repetitive sentence structure; three consecutive sentences begin with "If".While the content is accurate, consider rewording to improve readability and reduce the repetitive structure flagged by LanguageTool.
-| `mem_buf_limit` | Set a memory buffer limit for the input plugin instance in bytes. If the limit is reached, the plugin will pause until the buffer is drained. If set to `0`, the buffer limit is disabled. If the plugin has enabled filesystem buffering, this limit won't apply. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `0` | +| `mem_buf_limit` | Set a memory buffer limit for the input plugin instance in bytes. When the limit is reached, the plugin pauses until the buffer is drained; set to `0` to disable. Filesystem buffering, if enabled, will not be subject to this limit. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `0` |
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
pipeline/inputs/blob.md(2 hunks)
🧰 Additional context used
🪛 GitHub Check: runner / vale
pipeline/inputs/blob.md
[warning] 11-11:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "Useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "Useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/inputs/blob.md", "range": {"start": {"line": 11, "column": 104}}}, "severity": "WARNING"}
[warning] 3-3:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/inputs/blob.md", "range": {"start": {"line": 3, "column": 228}}}, "severity": "WARNING"}
🪛 LanguageTool
pipeline/inputs/blob.md
[style] ~16-~16: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...t to 0, the buffer limit is disabled. If the plugin has enabled filesystem buffe...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🔇 Additional comments (1)
pipeline/inputs/blob.md (1)
33-66: Excellent expansion of documentation with new sections and examples.The additions of "How it works", "Database file", and "Use cases" sections significantly improve clarity. The examples in both YAML and fluent-bit.conf formats are well-structured, practical, and demonstrate realistic configurations. The database file explanation with SQLite3/WAL details is particularly helpful for users operating Fluent Bit in production.
|
Ai reviews passed so merging. |
1 participant
Updated blob input plugin documentation with examples. Fixes #2185.
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.