Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle #9477

embik · 2022-03-30T10:22:18Z

What does this PR do / Why do we need it:
Yesterday, the etcd team announced that they do not recommend usage of etcd 3.5 in production anymore due to data consistency issues that can occur when etcd is terminated under high load. They expect that an upcoming etcd 3.5.3 release will fix this problem, but no ETA has been communicated so far.

This PR, as an initial mitigation, enables corruption checks for etcd 3.5 StatefulSets. This was initially added in #2460 as a Seed level feature flag and is available both in plain etcd and etcd-launcher. We're setting it to always true for the affected etcd version.

Corruption checks will not repair corrupted nodes, but will prevent them from starting, and therefore hopefully prevent data inconsistency in the live cluster. Corrupted members need to be manually removed and replaced if running plain etcd. etcd-launcher can be forced to rebuild the member by deleting the matching PV and letting the automatic PV recovery kick in.

In addition, this PR bumps the periodic corruption checks up to 4 hours from 10 minutes. The reason behind that is this comment on the etcd data consistency issue. Since the checks produce some overhead, running them every 10 minutes seems rather excessive.

Does this PR close any issues?:
Fixes #

Special notes for your reviewer:

Documentation:

Does this PR introduce a user-facing change?:

For user clusters that use etcd 3.5 (Kubernetes 1.22 clusters), etcd corruption checks are turned on to detect [etcd data consistency issues](https://github.com/etcd-io/etcd/issues/13766). Checks run at etcd startup and every 4 hours

Signed-off-by: Marvin Beckers marvin@kubermatic.com

Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

embik · 2022-03-30T10:23:02Z

/cherry-pick release/v2.20

kubermatic-bot · 2022-03-30T10:23:03Z

@embik: once the present PR merges, I will cherry-pick it on top of release/v2.20 in a new PR and assign it to you.

In response to this:

/cherry-pick release/v2.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

embik · 2022-03-30T10:23:10Z

/cherry-pick release/v2.19

kubermatic-bot · 2022-03-30T10:23:11Z

@embik: once the present PR merges, I will cherry-pick it on top of release/v2.19 in a new PR and assign it to you.

In response to this:

/cherry-pick release/v2.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

embik · 2022-03-30T10:23:14Z

/cherry-pick release/v2.18

kubermatic-bot · 2022-03-30T10:23:16Z

@embik: once the present PR merges, I will cherry-pick it on top of release/v2.18 in a new PR and assign it to you.

In response to this:

/cherry-pick release/v2.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

xmudrii

/lgtm
/approve

pkg/resources/etcd/statefulset.go

kubermatic-bot · 2022-03-30T10:30:15Z

LGTM label has been added.

Git tree hash: f46250ab1578a98127672ce384743d419548f3aa

kubermatic-bot · 2022-03-30T10:30:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: embik, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [embik,xmudrii]
~~pkg/resources/OWNERS~~ [embik,xmudrii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

kubermatic-bot · 2022-03-30T11:04:39Z

LGTM label has been added.

Git tree hash: 5068b51ce57ae30a548a2391c3c16ff51ddaa484

kubermatic-bot · 2022-03-30T11:45:33Z

@embik: #9477 failed to apply on top of branch "release/v2.20":

Applying: Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle
Applying: Use Check instead of Validate
Applying: Update fixtures
Using index info to reconstruct a base tree...
A	pkg/resources/test/fixtures/statefulset-aws-1.23.5-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-azure-1.23.5-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-bringyourown-1.23.5-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-digitalocean-1.23.5-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml
A	pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml
A	pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml left in tree.
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml left in tree.
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml left in tree.
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml left in tree.
Auto-merging pkg/resources/test/fixtures/statefulset-azure-1.21.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-azure-1.20.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.21.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.20.0-etcd.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0003 Update fixtures
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release/v2.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubermatic-bot · 2022-03-30T11:47:21Z

@embik: #9477 failed to apply on top of branch "release/v2.19":

Applying: Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle
Using index info to reconstruct a base tree...
M	cmd/etcd-launcher/main.go
M	pkg/resources/etcd/statefulset.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/resources/etcd/statefulset.go
Auto-merging cmd/etcd-launcher/main.go
Applying: Use Check instead of Validate
Applying: Update fixtures
Using index info to reconstruct a base tree...
M	pkg/resources/test/fixtures/statefulset-aws-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-aws-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-azure-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-azure-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-bringyourown-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-bringyourown-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-digitalocean-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-digitalocean-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-openstack-1.22.1-etcd-externalCloudProvider.yaml
M	pkg/resources/test/fixtures/statefulset-openstack-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml
A	pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-vsphere-1.22.1-etcd-externalCloudProvider.yaml
M	pkg/resources/test/fixtures/statefulset-vsphere-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml
A	pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml left in tree.
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml left in tree.
Auto-merging pkg/resources/test/fixtures/statefulset-vsphere-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-vsphere-1.22.1-etcd-externalCloudProvider.yaml
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml left in tree.
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml left in tree.
Auto-merging pkg/resources/test/fixtures/statefulset-openstack-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-openstack-1.22.1-etcd-externalCloudProvider.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-digitalocean-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-bringyourown-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-azure-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-azure-1.21.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-azure-1.20.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.21.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.20.0-etcd.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0003 Update fixtures
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release/v2.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubermatic-bot · 2022-03-30T11:47:59Z

@embik: #9477 failed to apply on top of branch "release/v2.18":

Applying: Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle
Using index info to reconstruct a base tree...
M	cmd/etcd-launcher/main.go
M	pkg/resources/etcd/statefulset.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/resources/etcd/statefulset.go
Auto-merging cmd/etcd-launcher/main.go
Applying: Use Check instead of Validate
Applying: Update fixtures
Using index info to reconstruct a base tree...
M	pkg/resources/test/fixtures/statefulset-aws-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-aws-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-azure-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-azure-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-bringyourown-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-bringyourown-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-digitalocean-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-digitalocean-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-openstack-1.22.1-etcd-externalCloudProvider.yaml
M	pkg/resources/test/fixtures/statefulset-openstack-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml
A	pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml
M	pkg/resources/test/fixtures/statefulset-vsphere-1.22.1-etcd-externalCloudProvider.yaml
M	pkg/resources/test/fixtures/statefulset-vsphere-1.22.1-etcd.yaml
A	pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml
A	pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd.yaml left in tree.
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-vsphere-1.23.5-etcd-externalCloudProvider.yaml left in tree.
Auto-merging pkg/resources/test/fixtures/statefulset-vsphere-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-vsphere-1.22.1-etcd-externalCloudProvider.yaml
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd.yaml left in tree.
CONFLICT (modify/delete): pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml deleted in HEAD and modified in Update fixtures. Version Update fixtures of pkg/resources/test/fixtures/statefulset-openstack-1.23.5-etcd-externalCloudProvider.yaml left in tree.
Auto-merging pkg/resources/test/fixtures/statefulset-openstack-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-openstack-1.22.1-etcd-externalCloudProvider.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-digitalocean-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-bringyourown-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-azure-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-azure-1.19.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.22.1-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.21.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.20.0-etcd.yaml
Auto-merging pkg/resources/test/fixtures/statefulset-aws-1.19.0-etcd.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0003 Update fixtures
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release/v2.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…ubermatic#9477) * Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Use Check instead of Validate Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Update fixtures Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Update testdata for etcd command with corruption flags Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

…se to 4h cycle (#9480) * Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle (#9477) * Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Use Check instead of Validate Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Update fixtures Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Update testdata for etcd command with corruption flags Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Regenerate fixtures Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

…9477) (#9481) * Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Use Check instead of Validate Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Update fixtures Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Update testdata for etcd command with corruption flags Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

…9477) (#9482) * Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Use Check instead of Validate Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Update fixtures Signed-off-by: Marvin Beckers <marvin@kubermatic.com> * Update testdata for etcd command with corruption flags Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle

2f4e03d

Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

xmudrii approved these changes Mar 30, 2022

View reviewed changes

pkg/resources/etcd/statefulset.go Outdated Show resolved Hide resolved

kubermatic-bot assigned xmudrii Mar 30, 2022

kubermatic-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 30, 2022

embik added 2 commits March 30, 2022 12:37

Use Check instead of Validate

fd7a55f

Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

Update fixtures

91985ae

Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

kubermatic-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 30, 2022

Update testdata for etcd command with corruption flags

97b56f7

Signed-off-by: Marvin Beckers <marvin@kubermatic.com>

xmudrii approved these changes Mar 30, 2022

View reviewed changes

kubermatic-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 30, 2022

kubermatic-bot merged commit dfa4ec7 into kubermatic:master Mar 30, 2022

embik mentioned this pull request Mar 30, 2022

[release/v2.20] Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle #9480

Merged

embik mentioned this pull request Mar 30, 2022

[release/v2.19] Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle #9481

Merged

embik mentioned this pull request Mar 30, 2022

[release/v2.18] Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle #9482

Merged

embik deleted the etcd-corruption-checks branch March 31, 2022 10:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle #9477

Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle #9477

embik commented Mar 30, 2022 •

edited

embik commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

embik commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

embik commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

xmudrii left a comment

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle #9477

Enable etcd corruption checks for etcd 3.5 and increase to 4h cycle #9477

Conversation

embik commented Mar 30, 2022 • edited

embik commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

embik commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

embik commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

xmudrii left a comment

Choose a reason for hiding this comment

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

kubermatic-bot commented Mar 30, 2022

embik commented Mar 30, 2022 •

edited