Add resize volume functionality to Logstash operator #7519

robbavey · 2024-01-30T00:41:50Z

This commit adds support for expanding Logstash volumes by editing the storage requests
in spec.VolumeClaimTemplate, based on the existing implenentation in Elasticsearch
implemented in #3752.

The same constraints hold -

Volume size can only be increased, not decreased
Storage class must specify allowVolumeExpansion: true
Filesystem resize without pod recreation is only possible if the storage driver allows it

This is based on the Elasticsearch implementation

An update of the storage request in the volumeClaimTemplate will

Update the storage requests spec of all existing PVCs: they are immediately resized by the storage driver, if inline expansion is supported. Otherwise Pods need to be recreated.
Delete the StatefulSet, but not the pod that it owns, storing recreation details in an annotation on the owning Logstash resource
Recreate the StatefulSet with the new volumeClaimTemplate spec
Remove the recreation annotation from the Logstash resource

Note:

This commit supersedes the previous PR at Add resize volume functionality to Logstash operator #7465 to provide a simpler to follow story of implementation

Testing Notes

minikube does not have a storage class that supports volume expansion - functionality has been tested in google cloud

kaisecheng

The PVC check in validation can add a checking to ensure the request size is increase like here, otherwise, LGTM

robbavey · 2024-02-01T15:59:31Z

@kaisecheng I've added the additional check that you suggested, and reworked the webhook validation to check for this too

kaisecheng

LGTM :)

This commit moves some of the statefulset functonality from an elasticsearch package to a common one to facilitate its use in other resources

…ackage

…e to common

This commit adds support for expanding Logstash volumes by editing the storage requests in spec.VolumeClaimTemplate, based on the existing implenentation in Elasticsearch implemented in elastic#3752. The same constraints hold - * Volume size can only be increased, not decreased * Storage class must specify allowVolumeExpansion: true * Filesystem resize without pod recreation is only possible if the storage driver allows it This is based on the Elasticsearch implementation An update of the storage request in the volumeClaimTemplate will * Update the storage requests spec of all existing PVCs: they are immediately resized by the storage driver, if inline expansion is supported. Otherwise Pods need to be recreated. * Delete the StatefulSet, but not the pod that it owns, storing recreation details in an annotation on the owning Logstash resource * Recreate the StatefulSet with the new volumeClaimTemplate spec * Remove the recreation annotation from the Logstash resource

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

This commit reworks the webhook validation to validate that storage updates fulfill the requirements (only increasing storage, using a valid storage class that allows storage expansion). This required moving the webhook validation into the controller package to allow use of the k8sclient without a dependency cycle, and requires some rework in webhook registration

robbavey · 2024-02-01T23:16:20Z

@pebrc @thbkrkr @barkbay This is ready for review

pebrc · 2024-02-02T12:46:50Z

buildkite test this -f p=gke,t=TestVolumeExpansion

pebrc

This looks really good. I did a first pass, only a few superficial findings. Wanted to run a few more tests (I did test the basic functionality) but ran out of time today.

pkg/controller/elasticsearch/nodespec/statefulset.go

pkg/controller/common/statefulset/fixtures.go

pkg/controller/common/volume/pvc_expansion.go

pebrc · 2024-02-02T16:26:45Z

pkg/controller/elasticsearch/driver/pvc_expansion_test.go

-		})
-	}
-}
-
 func Test_recreateStatefulSets(t *testing.T) {


This seems to almost duplicate the test with the same name in the common/volumes package which tests the function that this recreateStatefulSets delegates too. The only difference seems the additional kind attribute.

pkg/controller/logstash/labels/labels.go

pebrc · 2024-02-02T16:50:33Z

pkg/controller/logstash/logstash_controller.go

@@ -113,7 +117,8 @@ type ReconcileLogstash struct {
 	dynamicWatches watches.DynamicWatches
 	operator.Parameters
 	// iteration is the number of times this controller has run its Reconcile method
-	iteration uint64
+	iteration    uint64
+	expectations *expectations.ClustersExpectation


ClustersExpectations seems like a name that does not fit well for anything not Elasticsearch.

We used the ClustersExpectation, rather than the standard Expectation to allow expectations for multiple sets of Logstashes to be stored safely in the same ECK instance, and my understanding was that this was required

Yes my point was more about the naming, because Logstash is not really a clustered application, not about the fact that you are using it which is appropriate.

pkg/controller/logstash/logstash_controller_test.go

Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

Also some refactoring to move NewPodTemplateValidator to common, and stop exporting validatePodTemplate function from common Adds TODO: as reminder to improve log message to include parent type

robbavey · 2024-02-02T22:53:12Z

I think this is ready for another round @pebrc

pebrc

LGTM

robbavey · 2024-02-05T14:38:11Z

Thank you @pebrc! I don't have merge access to this repo, so would you mind hitting the merge button?

botelastic bot added the triage label Jan 30, 2024

robbavey changed the title ~~Feature/logstash volumeresize~~ Add resize volume functionality to Logstash operator Jan 30, 2024

robbavey force-pushed the feature/logstash_volumeresize branch from 0d6edbd to 346d19b Compare January 30, 2024 03:35

thbkrkr added >feature Adds or discusses adding a feature to the product :logstash labels Jan 30, 2024

botelastic bot removed the triage label Jan 30, 2024

robbavey force-pushed the feature/logstash_volumeresize branch 2 times, most recently from 6b23bf1 to c0815cb Compare January 31, 2024 17:34

robbavey marked this pull request as ready for review January 31, 2024 18:46

robbavey requested a review from kaisecheng January 31, 2024 18:46

robbavey mentioned this pull request Jan 31, 2024

Add resize volume functionality to Logstash operator #7465

Closed

karenzone assigned robbavey, karenzone and kaisecheng Jan 31, 2024

kaisecheng reviewed Feb 1, 2024

View reviewed changes

robbavey force-pushed the feature/logstash_volumeresize branch from aba1347 to 86ca421 Compare February 1, 2024 15:58

kaisecheng approved these changes Feb 1, 2024

View reviewed changes

robbavey and others added 11 commits February 1, 2024 18:11

Move Logstash labels into its own package

32e7301

Move statefulset functionality from elasticsearch to common

f38bf74

This commit moves some of the statefulset functonality from an elasticsearch package to a common one to facilitate its use in other resources

Move shared volume validation features from elasticsearch to common p…

d70a2ad

…ackage

Move shared volume claim expansion features from Elasticsearch packag…

307ca94

…e to common

Update webhook to allow storage claims to have storage requests changed

e3fa13b

Add end to end test of Logstash volume expansion

836aed9

Add Docs

b4f1600

Remove reference to ES in common podTemplate

f618631

Apply suggestions from code review

f037af4

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

robbavey force-pushed the feature/logstash_volumeresize branch from 86ca421 to 1fa362a Compare February 1, 2024 23:11

pebrc reviewed Feb 2, 2024

View reviewed changes

Apply suggestions from code review

fbed4df

Co-authored-by: Peter Brachwitz <peter.brachwitz@gmail.com>

robbavey force-pushed the feature/logstash_volumeresize branch from 2cda740 to fbed4df Compare February 2, 2024 19:27

robbavey added 5 commits February 2, 2024 14:32

Code review: Rename essset import to es_set for readability

f5ac438

Use common statefulset fixtures in Elasticsearch sset package

144644e

Code review: Use getIdentityLabels to populate GetPodIdentityLabels

86d7722

Remove unnecessary function

23c1fdb

Add podTemplateValidator to Logstash statefulset reconciler

7822d0c

Also some refactoring to move NewPodTemplateValidator to common, and stop exporting validatePodTemplate function from common Adds TODO: as reminder to improve log message to include parent type

robbavey force-pushed the feature/logstash_volumeresize branch from 8f2b898 to 7822d0c Compare February 2, 2024 21:26

Removed redundant tests

697ac88

robbavey force-pushed the feature/logstash_volumeresize branch from 538238f to 697ac88 Compare February 2, 2024 22:34

thbkrkr added the v2.12.0 label Feb 5, 2024

Move annotationPrefix calculation outside of loop

14957ba

pebrc approved these changes Feb 5, 2024

View reviewed changes

pebrc merged commit 6d681b3 into elastic:main Feb 5, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add resize volume functionality to Logstash operator #7519

Add resize volume functionality to Logstash operator #7519

robbavey commented Jan 30, 2024

kaisecheng left a comment

robbavey commented Feb 1, 2024

kaisecheng left a comment

robbavey commented Feb 1, 2024

pebrc commented Feb 2, 2024

pebrc left a comment

pebrc Feb 2, 2024

pebrc Feb 2, 2024

robbavey Feb 2, 2024

pebrc Feb 5, 2024

robbavey commented Feb 2, 2024

pebrc left a comment

robbavey commented Feb 5, 2024

Add resize volume functionality to Logstash operator #7519

Add resize volume functionality to Logstash operator #7519

Conversation

robbavey commented Jan 30, 2024

kaisecheng left a comment

Choose a reason for hiding this comment

robbavey commented Feb 1, 2024

kaisecheng left a comment

Choose a reason for hiding this comment

robbavey commented Feb 1, 2024

pebrc commented Feb 2, 2024

pebrc left a comment

Choose a reason for hiding this comment

pebrc Feb 2, 2024

Choose a reason for hiding this comment

pebrc Feb 2, 2024

Choose a reason for hiding this comment

robbavey Feb 2, 2024

Choose a reason for hiding this comment

pebrc Feb 5, 2024

Choose a reason for hiding this comment

robbavey commented Feb 2, 2024

pebrc left a comment

Choose a reason for hiding this comment

robbavey commented Feb 5, 2024