diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e9adaa4db..2222db7609 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,19 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +### Changed + +### Fixed + +- docs: FluentD buffer size configuration [#2232][#2232] + +[#2232]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2232 +[Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v2.6.0...main + ## [v2.7.0][v2.7.0] ### Released 2022-04-11 diff --git a/deploy/docs/Best_Practices.md b/deploy/docs/Best_Practices.md index 882bbd5d5c..8b97732ef0 100644 --- a/deploy/docs/Best_Practices.md +++ b/deploy/docs/Best_Practices.md @@ -393,6 +393,27 @@ Should you have any connectivity problems, depending on the buffer size your set be able to survive for a given amount of time without a data loss, delivering the data later when everything is operational again. +The FluentD buffer size is controlled by two major parameters - the size of the persistent volume +in Kubernetes, and the maximum size of the buffer on disk. Both need to be adjusted if you want +to buffer more (or less) data. + +For example: + +```yaml +fluentd: + ## Persist data to a persistent volume; When enabled, fluentd uses the file buffer instead of memory buffer. + persistence: + ## After changing this value please follow steps described in: + ## https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/main/deploy/docs/FluentdPersistence.md + enabled: true + size: 20Gi + buffer: + totalLimitSize: "20G" +``` + +The `fluentd.buffer` section contains other settings for FluentD buffering. Only change those if you know +what you're doing and have studied the relevant documentation carefully. + To calculate this time you need to know how much data you send. For the calculations below we made an assumption that a single metric data point is around 1 kilobyte in size, including metadata. This assumption is based on the average data we ingest. By default, for file based diff --git a/deploy/docs/FluentdPersistence.md b/deploy/docs/FluentdPersistence.md index ae80b9a1b8..b9680abb1e 100644 --- a/deploy/docs/FluentdPersistence.md +++ b/deploy/docs/FluentdPersistence.md @@ -36,8 +36,13 @@ fluentd: # annotations: {} accessMode: ReadWriteOnce size: 10Gi + buffer: + totalLimitSize: "10G" ``` +Keep in mind that you need to adjust `fluentd.buffer.totalLimitSize` in order for FluentD to actually queue +more data - increasing volume size alone is not enough. + Use one of following two strategies to prepare existing collection for enabling Fluentd persistence: - ### Enabling Fluentd persistence by recreating Fluentd StatefulSet