docs: FluentD buffer size configuration

SumoLogic · Apr 12, 2022 · e61c3bf · e61c3bf
1 parent 25f66a9
commit e61c3bf
Show file tree

Hide file tree

Showing 3 changed files with 39 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,19 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+### Changed
+
+### Fixed
+
+- docs: FluentD buffer size configuration [#2232][#2232]
+
+[#2232]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2232
+[Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v2.6.0...main
+
 ## [v2.7.0][v2.7.0]
 
 ### Released 2022-04-11

diff --git a/deploy/docs/Best_Practices.md b/deploy/docs/Best_Practices.md
@@ -393,6 +393,27 @@ Should you have any connectivity problems, depending on the buffer size your set
 be able to survive for a given amount of time without a data loss, delivering the data
 later when everything is operational again.
 
+The FluentD buffer size is controlled by two major parameters - the size of the persistent volume
+in Kubernetes, and the maximum size of the buffer on disk. Both need to be adjusted if you want
+to buffer more (or less) data.
+
+For example:
+
+```yaml
+fluentd:
+  ## Persist data to a persistent volume; When enabled, fluentd uses the file buffer instead of memory buffer.
+  persistence:
+    ## After changing this value please follow steps described in:
+    ## https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/main/deploy/docs/FluentdPersistence.md
+    enabled: true
+    size: 20Gi
+  buffer:
+    totalLimitSize: "20G"
+```
+
+The `fluentd.buffer` section contains other settings for FluentD buffering. Only change those if you know
+what you're doing and have studied the relevant documentation carefully.
+
 To calculate this time you need to know how much data you send. For the calculations below
 we made an assumption that a single metric data point is around 1 kilobyte in size, including
 metadata. This assumption is based on the average data we ingest. By default, for file based

diff --git a/deploy/docs/FluentdPersistence.md b/deploy/docs/FluentdPersistence.md
@@ -36,8 +36,13 @@ fluentd:
     # annotations: {}
     accessMode: ReadWriteOnce
     size: 10Gi
+  buffer:
+    totalLimitSize: "10G"
 ```
 
+Keep in mind that you need to adjust `fluentd.buffer.totalLimitSize` in order for FluentD to actually queue
+more data - increasing volume size alone is not enough.
+
 Use one of following two strategies to prepare existing collection for enabling Fluentd persistence:
 
 - ### Enabling Fluentd persistence by recreating Fluentd StatefulSet