Documented new 4.2 feature Dead Letter Queue (DLQ). Fixes #2207. #2208
Conversation
…d storage docs. Applies to #2207. Signed-off-by: Eric D. Schabell <eric@schabell.org>
…Applies to #2207. Signed-off-by: Eric D. Schabell <eric@schabell.org>
Signed-off-by: Eric D. Schabell <eric@schabell.org>
…e options and linked to DLQ documentation in service section docs. Applies to #2207. Signed-off-by: Eric D. Schabell <eric@schabell.org>
WalkthroughAdds documentation for a Dead Letter Queue (DLQ) across administration guides: introduces DLQ behavior, storage configuration keys and examples, DLQ file naming/location, troubleshooting guidance, lifecycle/cleanup notes, and cross-references to buffering/storage and retry docs. Changes
Sequence Diagram(s)(omitted — changes are documentation-only and do not modify runtime control flow) Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (5 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (15)
administration/troubleshooting.md (6)
5-5: Use sentence-style capitalization for the TOC entry.Line 5 should follow FluentBit style guidelines for headings.
- - [Dead letter queue: preserve failed chunks](#dead-letter-queue) + - [Dead letter queue: preserve failed chunks](#dead-letter-queue)Note: If this is already correct, verify the exact style rule applied.
9-9: Use sentence-style capitalization for the heading.Per FluentBit.Headings rule, headings should use sentence-style capitalization (capitalize only the first letter and proper nouns).
- ## Dead Letter Queue + ## Dead letter queue
11-11: Replace "useful" with more neutral wording.Per FluentBit.Simplicity guidelines, avoid "useful" as it implies ease of use. Consider a more descriptive alternative.
- The Dead Letter Queue (DLQ) feature preserves chunks that fail to be delivered to output destinations. This is useful for troubleshooting delivery failures without losing data. + The Dead Letter Queue (DLQ) feature preserves chunks that fail to be delivered to output destinations. This enables troubleshooting delivery failures without losing data.
13-13: Use sentence-style capitalization for the subheading.Per FluentBit.Headings rule, this H3 should use sentence-style capitalization.
- ### Enable DLQ + ### Enable DLQ(Note: Verify if "DLQ" as an acronym at sentence start requires capitalization; this may already be correct.)
49-49: Use sentence-style capitalization for the subheading.Per FluentBit.Headings rule, this H3 should use sentence-style capitalization.
- ### Examine DLQ files + ### Examine DLQ files
55-55: Use sentence-style capitalization for the subheading.Per FluentBit.Headings rule, this H3 should use sentence-style capitalization.
- ### DLQ management + ### DLQ managementadministration/buffering-and-storage.md (8)
154-154: Use sentence-style capitalization for the heading.Per FluentBit.Headings rule, headings should use sentence-style capitalization.
- ### Dead Letter Queue (DLQ) + ### Dead letter queue (DLQ)
158-158: Use sentence-style capitalization for the subheading.Per FluentBit.Headings rule, this H4 should use sentence-style capitalization.
- #### When DLQ is triggered + #### When DLQ is triggered
174-174: Use sentence-style capitalization for the subheading.Per FluentBit.Headings rule, this H4 should use sentence-style capitalization.
- #### DLQ file location and format + #### DLQ file location and format
195-195: Rephrase to avoid possessive form of "Fluent Bit".Per FluentBit.Possessives rule, rewrite "Bit's" to avoid the possessive 's.
- The file contains the original chunk data in Fluent Bit's internal format, preserving all records and metadata. + The file contains the original chunk data in the internal format of Fluent Bit, preserving all records and metadata.
197-197: Use sentence-style capitalization for the subheading.Per FluentBit.Headings rule, this H4 should use sentence-style capitalization.
- #### Troubleshooting with DLQ + #### Troubleshooting with DLQ
199-199: Replace "useful" with more neutral wording.Per FluentBit.Simplicity guidelines, avoid "useful" as it implies ease of use. Consider a more descriptive alternative.
- The DLQ feature is particularly useful for: + The DLQ feature enables the following capabilities:
226-227: Add nonbreaking space between number and unit in configuration examples.Per FluentBit.Units rule, update "5M" to "5 M" (nonbreaking space) for consistency.
storage.backlog.mem_limit: 5M - storage.keep.rejected: on - storage.rejected.path: rejected + storage.keep.rejected: on + storage.rejected.path: rejectedNote: Verify if all "5M" instances should use nonbreaking space. Check line 224 in the first service example as well.
Also applies to: 242-243
249-249: Add nonbreaking space between number and unit.Per FluentBit.Units rule, "5MB" should be "5 M" or "5 MB" with proper spacing.
- This configuration sets an optional buffering mechanism where the route to the data is `/var/log/flb-storage/`. It uses `normal` synchronization mode, without running a checksum and up to a maximum of 5MB of memory when processing backlog data. Additionally, the dead letter queue is enabled, and rejected chunks are stored in `/var/log/flb-storage/rejected/`. + This configuration sets an optional buffering mechanism where the route to the data is `/var/log/flb-storage/`. It uses `normal` synchronization mode, without running a checksum and up to a maximum of 5 MB of memory when processing backlog data. Additionally, the dead letter queue is enabled, and rejected chunks are stored in `/var/log/flb-storage/rejected/`.administration/configuring-fluent-bit/yaml/service-section.md (1)
43-43: Spell out DLQ acronym on first mention.Per FluentBit.Acronyms rule, unfamiliar acronyms should be spelled out. This is the first mention of DLQ in this file.
- | `storage.keep.rejected` | Enable the dead letter queue (DLQ) to preserve chunks that fail to be delivered. Accepted values: `off` or `on`. | `off` | + | `storage.keep.rejected` | Enable the Dead Letter Queue (DLQ) feature to preserve chunks that fail to be delivered. Accepted values: `off` or `on`. | `off` |
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
administration/buffering-and-storage.md(3 hunks)administration/configuring-fluent-bit/yaml/service-section.md(1 hunks)administration/scheduling-and-retries.md(1 hunks)administration/troubleshooting.md(1 hunks)
🧰 Additional context used
🪛 GitHub Actions: Lint PRs
administration/troubleshooting.md
[info] 9-9: [FluentBit.Headings] 'Dead Letter Queue' should use sentence-style capitalization.
[info] 13-13: [FluentBit.Headings] 'Enable DLQ' should use sentence-style capitalization.
[info] 49-49: [FluentBit.Headings] 'Examine DLQ files' should use sentence-style capitalization.
[info] 55-55: [FluentBit.Headings] 'DLQ management' should use sentence-style capitalization.
[warning] 199-199: [FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
administration/configuring-fluent-bit/yaml/service-section.md
[info] 43-43: [FluentBit.Acronyms] Spell out 'DLQ', if it's unfamiliar to the audience.
administration/buffering-and-storage.md
[info] 154-154: [FluentBit.Headings] 'Dead Letter Queue (DLQ)' should use sentence-style capitalization.
[info] 158-158: [FluentBit.Headings] 'When DLQ is triggered' should use sentence-style capitalization.
[info] 174-174: [FluentBit.Headings] 'DLQ file location and format' should use sentence-style capitalization.
[warning] 195-195: [FluentBit.Possessives] Rewrite 'Bit's' to not use 's.
[info] 197-197: [FluentBit.Headings] 'Troubleshooting with DLQ' should use sentence-style capitalization.
[warning] 199-199: [FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
[info] 249-249: [FluentBit.Units] Put a nonbreaking space between the number and the unit in '5MB'.
🪛 GitHub Check: runner / vale
administration/troubleshooting.md
[warning] 11-11:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "administration/troubleshooting.md", "range": {"start": {"line": 11, "column": 112}}}, "severity": "WARNING"}
[warning] 9-9:
[vale] reported by reviewdog 🐶
[FluentBit.Headings] 'Dead Letter Queue' should use sentence-style capitalization.
Raw Output:
{"message": "[FluentBit.Headings] 'Dead Letter Queue' should use sentence-style capitalization.", "location": {"path": "administration/troubleshooting.md", "range": {"start": {"line": 9, "column": 4}}}, "severity": "INFO"}
administration/configuring-fluent-bit/yaml/service-section.md
[warning] 43-43:
[vale] reported by reviewdog 🐶
[FluentBit.Acronyms] Spell out 'DLQ', if it's unfamiliar to the audience.
Raw Output:
{"message": "[FluentBit.Acronyms] Spell out 'DLQ', if it's unfamiliar to the audience.", "location": {"path": "administration/configuring-fluent-bit/yaml/service-section.md", "range": {"start": {"line": 43, "column": 59}}}, "severity": "INFO"}
administration/buffering-and-storage.md
[warning] 199-199:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 199, "column": 33}}}, "severity": "WARNING"}
[warning] 197-197:
[vale] reported by reviewdog 🐶
[FluentBit.Headings] 'Troubleshooting with DLQ' should use sentence-style capitalization.
Raw Output:
{"message": "[FluentBit.Headings] 'Troubleshooting with DLQ' should use sentence-style capitalization.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 197, "column": 6}}}, "severity": "INFO"}
[warning] 195-195:
[vale] reported by reviewdog 🐶
[FluentBit.Possessives] Rewrite 'Bit's' to not use 's.
Raw Output:
{"message": "[FluentBit.Possessives] Rewrite 'Bit's' to not use 's.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 195, "column": 53}}}, "severity": "WARNING"}
[warning] 174-174:
[vale] reported by reviewdog 🐶
[FluentBit.Headings] 'DLQ file location and format' should use sentence-style capitalization.
Raw Output:
{"message": "[FluentBit.Headings] 'DLQ file location and format' should use sentence-style capitalization.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 174, "column": 6}}}, "severity": "INFO"}
[warning] 158-158:
[vale] reported by reviewdog 🐶
[FluentBit.Headings] 'When DLQ is triggered' should use sentence-style capitalization.
Raw Output:
{"message": "[FluentBit.Headings] 'When DLQ is triggered' should use sentence-style capitalization.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 158, "column": 6}}}, "severity": "INFO"}
[warning] 154-154:
[vale] reported by reviewdog 🐶
[FluentBit.Headings] 'Dead Letter Queue (DLQ)' should use sentence-style capitalization.
Raw Output:
{"message": "[FluentBit.Headings] 'Dead Letter Queue (DLQ)' should use sentence-style capitalization.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 154, "column": 5}}}, "severity": "INFO"}
[warning] 249-249:
[vale] reported by reviewdog 🐶
[FluentBit.Units] Put a nonbreaking space between the number and the unit in '5MB'.
Raw Output:
{"message": "[FluentBit.Units] Put a nonbreaking space between the number and the unit in '5MB'.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 249, "column": 202}}}, "severity": "INFO"}
🪛 LanguageTool
administration/buffering-and-storage.md
[style] ~203-~203: Consider using a different verb for a more formal wording.
Context: ...play or transform rejected chunks after fixing the underlying issue. - Debugging: ...
(FIX_RESOLVE)
🔇 Additional comments (1)
administration/scheduling-and-retries.md (1)
98-100: LGTM!The hint box placement and DLQ reference are well-positioned after the
Retry_Limittable to guide users on data preservation options. The link to the full DLQ documentation is appropriate.
Signed-off-by: Eric D. Schabell <eric@schabell.org>
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
administration/troubleshooting.md(1 hunks)
🧰 Additional context used
🪛 GitHub Check: runner / vale
administration/troubleshooting.md
[warning] 11-11:
[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.
Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.", "location": {"path": "administration/troubleshooting.md", "range": {"start": {"line": 11, "column": 112}}}, "severity": "WARNING"}
🔇 Additional comments (5)
administration/troubleshooting.md (5)
5-7: Well-organized navigation structure.The DLQ entry at the top of the TOC provides good discoverability for users troubleshooting delivery issues.
13-38: Configuration examples are clear and well-formatted.Both YAML and fluent-bit.conf examples correctly demonstrate the three required storage keys with consistent values. Good practice to show both configuration formats.
40-48: Failure trigger conditions are clearly defined.The bullet points enumerate the specific scenarios that cause chunks to be routed to the DLQ. This helps users understand when DLQ preservation occurs.
49-54: Filename breakdown example is helpful for troubleshooting.The concrete example (
kube_var_log_containers_test_400_http_0x7f8b4c.flb) with explanation of components (tag, status code, plugin name) makes it easy for users to correlate files to specific failures.
55-61: Warning and cross-reference appropriately address lifecycle concerns.The warning box correctly highlights the need for manual cleanup and disk monitoring. The link to buffering-and-storage documentation provides a natural path for users seeking deeper configuration details.
Signed-off-by: Eric D. Schabell <eric@schabell.org>
Signed-off-by: Eric D. Schabell <eric@schabell.org>
Signed-off-by: Eric D. Schabell <eric@schabell.org>
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
administration/buffering-and-storage.md (1)
203-203: Consider a more formal verb choice for the bullet heading.LanguageTool suggests using a different verb than "Debugging" for more formal wording. Consider changing "Debugging: Analyze..." to "Analysis: Analyze..." or similar phrasing to improve consistency with formal documentation tone.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
administration/buffering-and-storage.md(3 hunks)administration/configuring-fluent-bit/yaml/service-section.md(1 hunks)administration/troubleshooting.md(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- administration/troubleshooting.md
🧰 Additional context used
🪛 GitHub Check: runner / vale
administration/buffering-and-storage.md
[warning] 154-154:
[vale] reported by reviewdog 🐶
[FluentBit.Acronyms] Spell out 'DLQ', if it's unfamiliar to the audience.
Raw Output:
{"message": "[FluentBit.Acronyms] Spell out 'DLQ', if it's unfamiliar to the audience.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 154, "column": 24}}}, "severity": "INFO"}
[warning] 154-154:
[vale] reported by reviewdog 🐶
[FluentBit.Headings] 'Dead letter queue (DLQ)' should use sentence-style capitalization.
Raw Output:
{"message": "[FluentBit.Headings] 'Dead letter queue (DLQ)' should use sentence-style capitalization.", "location": {"path": "administration/buffering-and-storage.md", "range": {"start": {"line": 154, "column": 5}}}, "severity": "INFO"}
🪛 LanguageTool
administration/buffering-and-storage.md
[style] ~203-~203: Consider using a different verb for a more formal wording.
Context: ...play or transform rejected chunks after fixing the underlying issue. - Debugging: ...
(FIX_RESOLVE)
🔇 Additional comments (5)
administration/configuring-fluent-bit/yaml/service-section.md (1)
29-48: Comprehensive storage configuration table with proper cross-linking.The new Storage configuration subsection is well-structured with clear key-value descriptions, accurate defaults, and good coverage of DLQ-related settings (
storage.keep.rejected,storage.rejected.path). The cross-references to scheduling-and-retries and buffering-and-storage documentation support the PR's goal of interconnected documentation.Please verify that the cross-reference links at lines 46 and 48 (to
scheduling-and-retries.md#Scheduling-and-Retriesandbuffering-and-storage.md) resolve correctly and use the correct anchor syntax for your documentation platform.administration/buffering-and-storage.md (4)
154-154: Align heading capitalization and acronym introduction with project style guidelines.Vale flagged two style concerns: (1) the heading "Dead letter queue (DLQ)" should use sentence-style capitalization (e.g., "Dead letter queue (dlq)"), and (2) the acronym should be spelled out on first mention if unfamiliar to the audience. Check if these align with your project's documentation standards.
154-210: Dead Letter Queue section is comprehensive and well-structured.The new section provides clear trigger scenarios, explicit requirements, file location/format with concrete examples, and appropriate troubleshooting guidance. The warning about manual cleanup and disk usage monitoring in the info box is valuable for operators.
226-227: Service examples correctly integrate DLQ configuration.The added configuration keys (
storage.keep.rejectedandstorage.rejected.path) are properly formatted in both YAML and CONF syntax and align with the configuration table in the Service section.Also applies to: 242-243
249-249: Updated service description accurately reflects DLQ enablement.The description now mentions that "the dead letter queue is enabled, and rejected chunks are stored in
/var/log/flb-storage/rejected/," which correctly maps the configuration to its behavior.
|
AI review passed, merging. |
1 participant
Updated administration section with the following to fix #2207:
buffering-and-storage.mdscheduling-and-retries.mdtroubleshooting.mdservice-section.mdSummary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.