Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update information about changing Fluentd persistence #1294

Merged
merged 1 commit into from Jan 12, 2021
Merged

Update information about changing Fluentd persistence #1294

merged 1 commit into from Jan 12, 2021

Conversation

kkujawa-sumo
Copy link
Contributor

@kkujawa-sumo kkujawa-sumo commented Jan 7, 2021 •
edited

Description

Related #1214
Some of changes because of differences between helm2 and helm3 related to --force option, for details see : helm/helm#7082 (comment), helm/helm#7431

Testing performed
  • Redeploy fluentd and fluentd-events pods
  • Confirm events, logs, and metrics are coming in

@kkujawa-sumo kkujawa-sumo changed the title Update information about changing Fleuntd persistance [WIP] Update information about changing Fleuntd persistance Jan 7, 2021
@kkujawa-sumo kkujawa-sumo force-pushed the kk-fluentd-intruction branch 6 times, most recently from bd1f409 to fb0e2aa Compare January 7, 2021 13:57
@kkujawa-sumo kkujawa-sumo changed the title [WIP] Update information about changing Fleuntd persistance [WIP] Update information about changing Fleuntd persistence Jan 7, 2021
@kkujawa-sumo kkujawa-sumo force-pushed the kk-fluentd-intruction branch 3 times, most recently from e19bc92 to 418415d Compare January 7, 2021 15:34
@kkujawa-sumo kkujawa-sumo marked this pull request as ready for review January 7, 2021 16:08
@kkujawa-sumo kkujawa-sumo requested a review from a team January 7, 2021 16:08
@kkujawa-sumo kkujawa-sumo changed the title [WIP] Update information about changing Fleuntd persistence Update information about changing Fleuntd persistence Jan 7, 2021
@kkujawa-sumo kkujawa-sumo force-pushed the kk-fluentd-intruction branch 7 times, most recently from 155ac84 to 804c208 Compare January 8, 2021 13:54
@perk-sumo perk-sumo added this to the v2.0 milestone Jan 8, 2021
@kkujawa-sumo
Copy link
Contributor Author

Related #1214

perk-sumo
perk-sumo reviewed Jan 11, 2021
View reviewed changes
deploy/docs/v2_migration_doc.md Outdated
### 4. Configure Fluentd persistence

Starting with `v2.0.0` we're using file-based buffer for Fluentd instead of less
reliable in-memory buffer (`fluentd.persistence.enabled=true`).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
reliable in-memory buffer (`fluentd.persistence.enabled=true`).
reliable in-memory buffer (`fluentd.persistence.enabled=true`) by default.

sumo-drosiek
sumo-drosiek reviewed Jan 11, 2021
View reviewed changes
deploy/docs/FluentdPersistence.md Outdated
```

When setting for Fluentd persistence is changed (enable or disable) it is required to recreate or delete existing Fluentd StatefulSet,
as it is not possible to change it and add/remove `volumeClaimTemplate`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
as it is not possible to change it and add/remove `volumeClaimTemplate`.
as it is not possible to add/remove `volumeClaimTemplate` for it.

sumo-drosiek
sumo-drosiek reviewed Jan 11, 2021
View reviewed changes
deploy/docs/FluentdPersistence.md Outdated

## Enabling Fluentd persistence

To enable Fluentd persistence in existing collection modify `values.yaml` file under the `fluentd`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To enable Fluentd persistence in existing collection modify `values.yaml` file under the `fluentd`
To enable Fluentd persistence in existing collection modify `values.yaml` file under the `fluentd`

sumo-drosiek
sumo-drosiek reviewed Jan 11, 2021
View reviewed changes
deploy/docs/FluentdPersistence.md Outdated
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## set, choosing the default provisioner. (gp2 on AWS, standard on
## set, choosing the default provisioner (gp2 on AWS, standard on

sumo-drosiek
sumo-drosiek reviewed Jan 11, 2021
View reviewed changes
deploy/docs/FluentdPersistence.md Outdated

## Disabling Fluentd persistence

To disable Fluentd persistence in existing collection modify `values.yaml` file under the `fluentd`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To disable Fluentd persistence in existing collection modify `values.yaml` file under the `fluentd`
To disable Fluentd persistence in existing collection modify `values.yaml` file under the `fluentd`

sumo-drosiek
sumo-drosiek previously approved these changes Jan 11, 2021
View reviewed changes
perk-sumo
perk-sumo reviewed Jan 11, 2021
View reviewed changes
Copy link
Contributor

@perk-sumo perk-sumo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of things to change.

deploy/docs/v2_migration_doc.md Outdated
Starting with `v2.0.0` we're using file-based buffer for Fluentd instead of less
reliable in-memory buffer (`fluentd.persistence.enabled=true`).

When Fluentd persistence is enabled in collection to upgrade no action is required.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When Fluentd persistence is enabled in collection to upgrade no action is required.
When Fluentd persistence is enabled then no action is required in order to upgrade.

deploy/docs/v2_migration_doc.md Outdated
When Fluentd persistence is enabled in collection to upgrade no action is required.

When Fluentd persistence is disabled (default setting in `1.3.5` release)
it is required either go through persistence enabling procedure before upgrade (recommended)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
it is required either go through persistence enabling procedure before upgrade (recommended)
it is required to either go through persistence enabling procedure before upgrade (recommended)

deploy/docs/v2_migration_doc.md Outdated
**In order to enable persistence in existing collection** please follow one of persistence enabling procedures described in
[Enabling Fluentd Persistence](FluentdPersistence.md#enabling-fluentd-persistence) guide before upgrade.

If Fluentd persistence is disabled in existing collection and it is desired to preserve this setting,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If Fluentd persistence is disabled in existing collection and it is desired to preserve this setting,
If Fluentd persistence is disabled and it is desired to preserve this setting,

deploy/docs/FluentdPersistence.md Outdated
# Fluentd persistence

Starting with `v2.0.0` we're using file-based buffer for Fluentd instead of less
reliable in-memory buffer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
reliable in-memory buffer.
reliable in-memory buffer by default.

deploy/docs/FluentdPersistence.md Outdated
enabled: true
```

When setting for Fluentd persistence is changed (enable or disable) it is required to recreate or delete existing Fluentd StatefulSet,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When setting for Fluentd persistence is changed (enable or disable) it is required to recreate or delete existing Fluentd StatefulSet,
When the Fluentd persistence setting is to be changed (enabled or disabled) it is required to recreate or delete existing Fluentd StatefulSet,

deploy/docs/FluentdPersistence.md Outdated
Comment on lines 45 to 46
Recreating Fluentd StatefulSet with new `volumeClaimTemplate` may cause that logs and metrics
will not be available in the time of recreation. It usually shouldn't take more than several seconds.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Recreating Fluentd StatefulSet with new `volumeClaimTemplate` may cause that logs and metrics
will not be available in the time of recreation. It usually shouldn't take more than several seconds.
In a heavy used clusters with high load of logs and metrics it might be possible that
recreating Fluentd StatefulSet with new `volumeClaimTemplate` may cause logs and metrics
being unavailable for the time of recreation. It usually shouldn't take more than several seconds.

deploy/docs/FluentdPersistence.md Outdated

- ### Enabling Fluentd persistence by preparing temporary instances of Fluentd and removing earlier created

To create temporary instances of Fluentd StatefulSets and avoid of break in logs and metrics:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To create temporary instances of Fluentd StatefulSets and avoid of break in logs and metrics:
To create a temporary instances of Fluentd StatefulSets and avoid a loss of logs or metrics:

deploy/docs/FluentdPersistence.md Outdated
kubectl delete statefulset --namespace <NAMESPACE-NAME> <RELEASE-NAME>-sumologic-fluentd-metrics
```

**Notice:** After collection upgrade is done, in order to remove the temporary Fluentd
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Notice:** After collection upgrade is done, in order to remove the temporary Fluentd
**Notice:** After the Helm chart upgrade is done, in order to remove the temporary Fluentd

deploy/docs/FluentdPersistence.md Outdated
--selector "release==<RELEASE-NAME>,heritage=tmp"
```

Upgrade collection with Fluentd persistence enabled, e.g.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be a little bit before the deletion command?

deploy/docs/FluentdPersistence.md Outdated
--selector "release==<RELEASE-NAME>,heritage=tmp"
```

Upgrade collection with Fluentd persistence disabled, e.g.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this should be up a bit IMO.

@kkujawa-sumo kkujawa-sumo force-pushed the kk-fluentd-intruction branch 3 times, most recently from ab6b436 to c03acd1 Compare January 11, 2021 11:24
@kkujawa-sumo
Copy link
Contributor Author

@perk-sumo @pmalek-sumo it is ready for next review.

pmalek-sumo
pmalek-sumo reviewed Jan 11, 2021
View reviewed changes
deploy/docs/v2_migration_doc.md Outdated
Comment on lines 279 to 282
If Fluentd persistence is disabled and it is desired to preserve this setting,
modify defaults and disable persistence either by adding `--set fluentd.persistence.enabled=false` to `helm upgrade` command or
in the `values.yaml` file under the `fluentd`
key as follows:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If Fluentd persistence is disabled and it is desired to preserve this setting,
modify defaults and disable persistence either by adding `--set fluentd.persistence.enabled=false` to `helm upgrade` command or
in the `values.yaml` file under the `fluentd`
key as follows:
If Fluentd persistence is disabled and it is desired to preserve this setting,
modify defaults and disable persistence either by adding `--set fluentd.persistence.enabled=false`
to `helm upgrade` command or in the `values.yaml` file under the `fluentd` key as follows:

@kkujawa-sumo kkujawa-sumo force-pushed the kk-fluentd-intruction branch 3 times, most recently from 2989678 to 714867d Compare January 12, 2021 16:46
perk-sumo
perk-sumo approved these changes Jan 12, 2021
View reviewed changes
Copy link
Contributor

@perk-sumo perk-sumo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@perk-sumo
Copy link
Contributor

Please squash the commits before merge :)

@perk-sumo perk-sumo added the documentation documentation label Jan 12, 2021
@perk-sumo
Copy link
Contributor

We should backport it to v1.3 when it's ready.

@kkujawa-sumo @perk-sumo
Update information about changing Fluentd persistence
1326103 
Use variables to let users to less substitutions

Add info to fill variables with proper values. Also move this info before command is run.

Add command to remove remaining pvc

Add envs to pvc deletion commands

Signed-off-by: Katarzyna Kujawa <kkujawa@sumologic.com>
Co-authored-by: Marcin 'Perk' Stożek <perk@sumologic.com>
@kkujawa-sumo kkujawa-sumo merged commit 8b9195c into main Jan 12, 2021
@kkujawa-sumo kkujawa-sumo deleted the kk-fluentd-intruction branch January 12, 2021 17:06
@kkujawa-sumo kkujawa-sumo mentioned this pull request Jan 13, 2021
Closed
@perk-sumo perk-sumo mentioned this pull request Jan 13, 2021
Closed
@kkujawa-sumo kkujawa-sumo mentioned this pull request Jan 13, 2021
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pmalek-sumo pmalek-sumo pmalek-sumo left review comments

@perk-sumo perk-sumo perk-sumo approved these changes

@sumo-drosiek sumo-drosiek sumo-drosiek left review comments

@andrzej-stencel andrzej-stencel Awaiting requested review from andrzej-stencel

@frankreno frankreno Awaiting requested review from frankreno

Assignees
No one assigned
Labels
documentation documentation
Projects
None yet
Milestone
v2.0
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@kkujawa-sumo @perk-sumo @sumo-drosiek @pmalek-sumo