diff --git a/deploy/docs/Best_Practices.md b/deploy/docs/Best_Practices.md index f48c82c915..725bd1314c 100644 --- a/deploy/docs/Best_Practices.md +++ b/deploy/docs/Best_Practices.md @@ -146,10 +146,13 @@ key as follows: fluentd: ## Persist data to a persistent volume; When enabled, fluentd uses the file buffer instead of memory buffer. persistence: - ## After setting the value to true, run the helm upgrade command with the --force flag. + ## After changing this value please follow steps described in: + ## https://github.com/SumoLogic/sumologic-kubernetes-collection/tree/main/deploy/docs/FluentdPersistence.md enabled: true ``` +After changing Fluentd persistence setting (enable or disable) follow steps described in [Fluentd Persistence](FluentdPersistence.md). + Additional buffering and flushing parameters can be added in the `extraConf`, in the `fluentd` buffer section. diff --git a/deploy/docs/FluentdPersistence.md b/deploy/docs/FluentdPersistence.md new file mode 100644 index 0000000000..37d0cb1035 --- /dev/null +++ b/deploy/docs/FluentdPersistence.md @@ -0,0 +1,292 @@ +# Fluentd persistence + +Starting with `v2.0.0` we're using file-based buffer for Fluentd instead of less +reliable in-memory buffer. + +The buffer configuration can be set in the `values.yaml` file under the `fluentd` +key as follows: + +```yaml +fluentd: + persistence: + enabled: true +``` + +When setting for Fluentd persistence is changed (enable or disable) it is required to recreate or delete existing Fluentd StatefulSet, +as it is not possible to change it and add/remove `volumeClaimTemplate`. + +**Note:** The below commands are using `yq` in version `3.4.0` <= `x` < `4.0.0`. + +## Enabling Fluentd persistence + +To enable Fluentd persistence in existing collection modify `values.yaml` file under the `fluentd` +key as follows: + +```yaml +fluentd: + persistence: + enabled: true + ## If defined, storageClassName: + ## If set to "-", storageClassName: "", which disables dynamic provisioning + ## If undefined (the default) or set to null, no storageClassName spec is + ## set, choosing the default provisioner. (gp2 on AWS, standard on + ## GKE, Azure & OpenStack) + ## + # storageClass: "-" + # annotations: {} + accessMode: ReadWriteOnce + size: 10Gi +``` + +Use one of following two strategies to prepare existing collection for enabling Fluentd persistence: + +- ### Enabling Fluentd persistence by recreating Fluentd StatefulSet + + Recreating Fluentd StatefulSet with new `volumeClaimTemplate` may cause that logs and metrics + will not be available in the time of recreation. It usually shouldn't take more than several seconds. + + To recreate Fluentd StatefulSets with new `volumeClaimTemplate` one can run + the following commands for all Fluentd StatefulSets: + + ```bash + VOLUME_CLAIM_TEMPLATE=$(cat <<-"EOF" + metadata: + name: buffer + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + EOF + ) && \ + BUFFER_VOLUME=$(cat <<-"EOF" + mountPath: /fluentd/buffer + name: buffer + EOF + )&& \ + kubectl --namespace get statefulset -sumologic-fluentd-logs --output yaml | \ + yq w - "spec.volumeClaimTemplates[+]" --from <(echo "${VOLUME_CLAIM_TEMPLATE}") | \ + yq w - "spec.template.spec.containers[0].volumeMounts[+]" --from <(echo "${BUFFER_VOLUME}") | \ + kubectl apply --namespace --force --filename - + ``` + + ```bash + VOLUME_CLAIM_TEMPLATE=$(cat <<-"EOF" + metadata: + name: buffer + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + EOF + ) && \ + BUFFER_VOLUME=$(cat <<-"EOF" + mountPath: /fluentd/buffer + name: buffer + EOF + )&& \ + kubectl --namespace get statefulset -sumologic-fluentd-metrics --output yaml | \ + yq w - "spec.volumeClaimTemplates[+]" --from <(echo "${VOLUME_CLAIM_TEMPLATE}") | \ + yq w - "spec.template.spec.containers[0].volumeMounts[+]" --from <(echo "${BUFFER_VOLUME}") | \ + kubectl apply --namespace --force --filename - + ``` + + ```bash + VOLUME_CLAIM_TEMPLATE=$(cat <<-"EOF" + metadata: + name: buffer + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + EOF + ) && \ + BUFFER_VOLUME=$(cat <<-"EOF" + mountPath: /fluentd/buffer + name: buffer + EOF + )&& \ + kubectl --namespace get statefulset -sumologic-fluentd-events --output yaml | \ + yq w - "spec.volumeClaimTemplates[+]" --from <(echo "${VOLUME_CLAIM_TEMPLATE}") | \ + yq w - "spec.template.spec.containers[0].volumeMounts[+]" --from <(echo "${BUFFER_VOLUME}") | \ + kubectl apply --namespace --force --filename - + ``` + + Remember to adjust `volumeClaimTemplate` (`VOLUME_CLAIM_TEMPLATE` variable in command above) + which will be added to `volumeClaimTemplates` in StatefulSet `spec` according to your needs, + for details please check `PersistentVolumeClaim` in Kubernetes API specification. + + **Notice** When StatefulSets managed by helm are modified by commands specified above, + one might expect a warning similar to the one below: + `Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply` + +- ### Enabling Fluentd persistence by preparing temporary instances of Fluentd and removing earlier created + + To create temporary instances of Fluentd StatefulSets and avoid of break in logs and metrics: + + ```bash + kubectl get statefulset --namespace -sumologic-fluentd-logs --output yaml | \ + yq w - "metadata.name" tmp--sumologic-fluentd-logs | \ + yq w - "metadata.labels[heritage]" "tmp" | \ + yq w - "spec.template.metadata.labels[heritage]" "tmp" | \ + yq w - "spec.selector.matchLabels[heritage]" "tmp" | \ + kubectl create --filename - + ``` + + ```bash + kubectl get statefulset --namespace -sumologic-fluentd-metrics --output yaml | \ + yq w - "metadata.name" tmp--sumologic-fluentd-metrics | \ + yq w - "metadata.labels[heritage]" "tmp" | \ + yq w - "spec.template.metadata.labels[heritage]" "tmp" | \ + yq w - "spec.selector.matchLabels[heritage]" "tmp" | \ + kubectl create --filename - + ``` + + ```bash + kubectl get statefulset --namespace -sumologic-fluentd-events --output yaml | \ + yq w - "metadata.name" tmp--sumologic-fluentd-events | \ + yq w - "metadata.labels[heritage]" "tmp" | \ + yq w - "spec.template.metadata.labels[heritage]" "tmp" | \ + yq w - "spec.selector.matchLabels[heritage]" "tmp" | \ + kubectl create --filename - + ``` + + Delete old instances of Fluentd StatefulSets: + + ```bash + kubectl wait --for=condition=ready pod \ + --namespace \ + --selector "release==,heritage=tmp" && \ + kubectl delete statefulset --namespace -sumologic-fluentd-events && \ + kubectl delete statefulset --namespace -sumologic-fluentd-logs && \ + kubectl delete statefulset --namespace -sumologic-fluentd-metrics + ``` + + **Notice:** After collection upgrade is done, in order to remove the temporary Fluentd + StatefulSets run the following command: + + ```bash + kubectl wait --for=condition=ready pod \ + --namespace \ + --selector "release==,heritage=Helm" && \ + kubectl delete statefulset \ + --namespace \ + --selector "release==,heritage=tmp" + ``` + +Upgrade collection with Fluentd persistence enabled, e.g. + +```bash +helm upgrade sumologic/sumologic --version= -f +``` + +## Disabling Fluentd persistence + +To disable Fluentd persistence in existing collection modify `values.yaml` file under the `fluentd` +key as follows: + +```yaml +fluentd: + persistence: + enabled: false +``` + +Use one of following two strategies to prepare existing collection for disabling Fluentd persistence: + +- ### Disabling Fluentd persistence by recreating Fluentd StatefulSet + + Recreating Fluentd StatefulSet without `volumeClaimTemplate` may cause that logs and metrics + will not be available in the time of recreation. It usually shouldn't take more than several seconds. + + To recreate Fluentd StatefulSets without `volumeClaimTemplate` one can run + the following commands for all Fluentd StatefulSets: + + ```bash + kubectl --namespace get statefulset -sumologic-fluentd-logs --output yaml | \ + yq d - "spec.template.spec.containers[*].volumeMounts(name==buffer)" | \ + yq d - "spec.volumeClaimTemplates(metadata.name==buffer)" | \ + kubectl apply --namespace --force --filename - + ``` + + ```bash + kubectl --namespace get statefulset -sumologic-fluentd-metrics --output yaml | \ + yq d - "spec.template.spec.containers[*].volumeMounts(name==buffer)" | \ + yq d - "spec.volumeClaimTemplates(metadata.name==buffer)" | \ + kubectl apply --namespace --force --filename - + ``` + + ```bash + kubectl --namespace get statefulset -sumologic-fluentd-events --output yaml | \ + yq d - "spec.template.spec.containers[*].volumeMounts(name==buffer)" | \ + yq d - "spec.volumeClaimTemplates(metadata.name==buffer)" | \ + kubectl apply --namespace --force --filename - + ``` + + **Notice** When StatefulSets managed by helm are modified by commands specified above, + one might expect a warning similar to the one below: + `Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply` + +- ### Disabling Fluentd persistence by preparing temporary instances of Fluentd and removing earlier created + + To create temporary instances of Fluentd StatefulSets and avoid of break in logs and metrics: + + ```bash + kubectl get statefulset --namespace -sumologic-fluentd-logs --output yaml | \ + yq w - "metadata.name" tmp--sumologic-fluentd-logs | \ + yq w - "metadata.labels[heritage]" "tmp" | \ + yq w - "spec.template.metadata.labels[heritage]" "tmp" | \ + yq w - "spec.selector.matchLabels[heritage]" "tmp" | \ + kubectl create --filename - + ``` + + ```bash + kubectl get statefulset --namespace -sumologic-fluentd-metrics --output yaml | \ + yq w - "metadata.name" tmp--sumologic-fluentd-metrics | \ + yq w - "metadata.labels[heritage]" "tmp" | \ + yq w - "spec.template.metadata.labels[heritage]" "tmp" | \ + yq w - "spec.selector.matchLabels[heritage]" "tmp" | \ + kubectl create --filename - + ``` + + ```bash + kubectl get statefulset --namespace -sumologic-fluentd-events --output yaml | \ + yq w - "metadata.name" tmp--sumologic-fluentd-events | \ + yq w - "metadata.labels[heritage]" "tmp" | \ + yq w - "spec.template.metadata.labels[heritage]" "tmp" | \ + yq w - "spec.selector.matchLabels[heritage]" "tmp" | \ + kubectl create --filename - + ``` + + Delete old instances of Fluentd StatefulSets: + + ```bash + kubectl wait --for=condition=ready pod \ + --namespace \ + --selector "release==,heritage=tmp" && \ + kubectl delete statefulset --namespace -sumologic-fluentd-events && \ + kubectl delete statefulset --namespace -sumologic-fluentd-logs && \ + kubectl delete statefulset --namespace -sumologic-fluentd-metrics + ``` + + **Notice:** After collection upgrade is done, in order to remove the temporary Fluentd + StatefulSets run the following command: + + ```bash + kubectl wait --for=condition=ready pod \ + --namespace \ + --selector "release==,heritage=Helm" && \ + kubectl delete statefulset \ + --namespace \ + --selector "release==,heritage=tmp" + ``` + +Upgrade collection with Fluentd persistence disabled, e.g. + +```bash +helm upgrade sumologic/sumologic --version= -f +``` diff --git a/deploy/docs/v2_migration_doc.md b/deploy/docs/v2_migration_doc.md index 55e1a3394f..c3ca037e83 100644 --- a/deploy/docs/v2_migration_doc.md +++ b/deploy/docs/v2_migration_doc.md @@ -93,6 +93,8 @@ as well as the exact steps for migration. app.kubernetes.io/instance: ``` +- Persistence for Fluentd is enabled by default. + ## How to upgrade ### Requirements @@ -260,7 +262,32 @@ One of the following two strategies can be used: --selector "app=fluent-bit,release=,heritage=tmp" ``` -### 4. Run upgrade script +### 4. Configure Fluentd persistence + +Starting with `v2.0.0` we're using file-based buffer for Fluentd instead of less +reliable in-memory buffer (`fluentd.persistence.enabled=true`). + +When Fluentd persistence is enabled in collection to upgrade no action is required. + +When Fluentd persistence is disabled (default setting in `1.3.5` release) +it is required either go through persistence enabling procedure before upgrade (recommended) +or preserve existing setting and modify default setting for Fluentd persistence in `2.0.0` release. + +**In order to enable persistence in existing collection** please follow one of persistence enabling procedures described in +[Enabling Fluentd Persistence](FluentdPersistence.md#enabling-fluentd-persistence) guide before upgrade. + +If Fluentd persistence is disabled in existing collection and it is desired to preserve this setting, +modify defaults and disable persistence either by adding `--set fluentd.persistence.enabled=false` to `helm upgrade` command or +in the `values.yaml` file under the `fluentd` +key as follows: + +```yaml +fluentd: + persistence: + enabled: false +``` + +### 5. Run upgrade script For Helm users, the only breaking changes are the renamed config parameters. For users who use a `values.yaml` file, we provide a script that users can run diff --git a/deploy/helm/sumologic/README.md b/deploy/helm/sumologic/README.md index 50a62a9283..7dd64d6f8a 100644 --- a/deploy/helm/sumologic/README.md +++ b/deploy/helm/sumologic/README.md @@ -53,7 +53,7 @@ Parameter | Description | Default `fluentd.podLabels` | Additional labels for all fluentd pods | `{}` `fluentd.podAnnotations` | Additional annotations for all fluentd pods | `{}` `fluentd.podSecurityPolicy.create` | If true, create & use `podSecurityPolicy` for fluentd resources | `false` -`fluentd.persistence.enabled` | Persist data to a persistent volume; When enabled, fluentd uses the file buffer instead of memory buffer. After setting the value to true, run the helm upgrade command with the --force flag. | `false` +`fluentd.persistence.enabled` | Persist data to a persistent volume; When enabled, fluentd uses the file buffer instead of memory buffer. After changing this value follow steps described in [Fluentd Persistence](FluentdPersistence.md).| `true` `fluentd.persistence.storageClass` | If defined, storageClassName: . If set to "-", storageClassName: "", which disables dynamic provisioning. If undefined (the default) or set to null, no storageClassName spec is set, choosing the default provisioner. (gp2 on AWS, standard on GKE, Azure & OpenStack) | `Nil` `fluentd.persistence.annotations` | Annotations for the persistence. | `Nil` `fluentd.persistence.accessMode` | The accessMode for persistence. | `ReadWriteOnce` diff --git a/deploy/helm/sumologic/values.yaml b/deploy/helm/sumologic/values.yaml index a4cd07adb2..31248c8284 100644 --- a/deploy/helm/sumologic/values.yaml +++ b/deploy/helm/sumologic/values.yaml @@ -233,7 +233,8 @@ fluentd: ## Persist data to a persistent volume; When enabled, fluentd uses the file buffer instead of memory buffer. persistence: - ## After setting the value to true, run the helm upgrade command with the --force flag. + ## After changing this value please follow steps described in: + ## https://github.com/SumoLogic/sumologic-kubernetes-collection/tree/main/deploy/docs/FluentdPersistence.md enabled: true ## If defined, storageClassName: