Implement zoneTracker for coordinating downscales across statefulsets using a ConfigMap for the prepare_downscale admission webhook #107

JordanRushing · 2023-12-15T02:25:33Z

Replacing #96 with this PR, changing the implementation to use a ConfigMap instead of a file in object storage.

The previous PR includes context for the reasoning behind implementing the zoneTracker and ultimately settling on a ConfigMap as the mechanism to store the relevant zone metadata.

… using a file in object storage for the prepare_downscale admission webhook Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

…itial zone file if not present Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

…storage; update zoneTracker tests; run `go mod vendor` & `go mod tidy` Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

…on; update tests; go mod vendor & go mod tidy Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Update mockBucket instantiation in test; deny admission requests when failing to bootstrap the s3 bucket client Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Clean up s3 config struct Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Change s3 config creation semantics for zoneTracker Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Add support for gcs and azure object storage clients; go mod vendor & go mod tidy Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

…ones; fix deadlock in lastDownscaled Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

…g; update comment on loadZones to indicate unique sts name assumption for keys

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

…e bucket code Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

… storage dependency Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

MichelHollands

LGTM

jhalterman · 2023-12-15T17:32:50Z

cmd/rollout-operator/main.go

@@ -87,6 +94,12 @@ func (cfg config) validate() error {
 	if (cfg.kubeAPIURL == "") != (cfg.kubeConfigFile == "") {
 		return errors.New("either configure both Kubernetes API URL and config file or none of them")
 	}
+	if cfg.useZoneTracker {


Could combine these if statements

andyasp

I don't have the full background context here, but tried to help take a look.

andyasp · 2023-12-20T17:57:56Z

pkg/admission/prep_downscale.go

-			level.Warn(logger).Log("msg", msg, "err", err)
-			return deny(msg)
-		}
+		// If using zoneTracker instead of downscale annotations, check the zone file for recent downscaling.


Suggested change

// If using zoneTracker instead of downscale annotations, check the zone file for recent downscaling.

// If using zoneTracker instead of downscale annotations, check the zone ConfigMap for recent downscaling.

There are other instances of "zone file" elsewhere as well.

andyasp · 2023-12-20T18:05:44Z

pkg/admission/prep_downscale.go

-		}
+		// If using zoneTracker instead of downscale annotations, check the zone file for recent downscaling.
+		// Otherwise, check the downscale annotations.
+		if zt != nil {


Nit: If we are doing the same logical operation with two different underlying implementations that seems like a good opportunity to use an interface and have a single function that decides which implementation will be used. I think that would make the logic here easier to follow, since currently zt != nil reads to me like "are we zone tracking?", but even when zt == nil zones are being tracked through the statefulset annotations since each zone is represented by a respective statefulset.

andyasp · 2023-12-20T18:22:57Z

pkg/admission/zone_tracker_test.go

+	}
+}
+
+func TestLoadZonesEmptyBucket(t *testing.T) {


Suggested change

func TestLoadZonesEmptyBucket(t *testing.T) {

func TestLoadZonesEmptyConfigMap(t *testing.T) {

andyasp · 2023-12-20T18:32:20Z

pkg/admission/zone_tracker.go

+	return nil, nil
+}
+
+// If the zone file does not exist, populate it with the current time for each zone and upload it to the bucket.


Suggested change

// If the zone file does not exist, populate it with the current time for each zone and upload it to the bucket.

// If the zone file does not exist, populate it with the current time for each zone and upload it to the ConfigMap.

andyasp · 2023-12-20T18:37:04Z

go.mod

 	k8s.io/api v0.28.3
 	k8s.io/apimachinery v0.28.3
 	k8s.io/client-go v0.28.3
 )

+require (


A bit odd another require was added, deleting this block and rerunning go mod tidy might fix it up.

andyasp · 2023-12-20T18:51:05Z

pkg/admission/zone_tracker.go

+func (zt *zoneTracker) lastDownscaled(zone string) (string, error) {
+	zoneInfo, ok := zt.zones[zone]
+	if !ok {
+		return "", fmt.Errorf("zone %s not found", zone)


Passing the ok back in the return rather than error seems more direct.

andyasp · 2023-12-20T19:25:23Z

pkg/admission/zone_tracker.go

+		}
+
+		lastDownscaleStr, err := zt.lastDownscaled(sts.Name)
+		if err != nil {


Should the current time be assumed in this case instead since that is what the initialization does? (Genuine question, not leading, having a hard time following some of the cases here).

andyasp · 2023-12-20T19:35:15Z

pkg/admission/zone_tracker.go

+		return nil, err
+	}
+
+	// Populate initial zones after the configmap is created


This feels unnecessary. The above Create I can see to allow Update, but if we were going to create anyway you could create with the initialized zones (or leave it empty and assume missing zones as current time?)

I might be wrong here because at some point you do need to write a first value to have the confidence you've waited a certain amount of time.

…ile; run `go mod tidy` & `go mod vendor`; simplify zoneTracker config validation conditional Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

… add tests for new functions post-refactor Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

JordanRushing · 2024-01-08T22:32:35Z

Thanks so much for the review, @andyasp -- I've pushed a commit here and to another branch to implement your suggestions and feedback:

https://github.com/JordanRushing/rollout-operator/compare/prep-downscale-last-downscale-configmap...JordanRushing:rollout-operator:prep-downscale-last-downscale-configmap-refactor?expand=1

Once I'm finished iterating on the new branch, I'll merge the changes into this PR to finish the review so that we can merge and cut a new tag for the rollout operator.

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

…e-configmap-refactor Prep downscale last downscale configmap refactor

MichelHollands

One nit but otherwise LGTM

MichelHollands · 2024-01-09T16:53:12Z

pkg/admission/prep_downscale.go

 	client := &http.Client{
 		Timeout: 5 * time.Second,
 	}
+
+	var zt *zoneTracker


(nit) You don't need this variable here, it can stay inside the if useZoneTracker branch.

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

… encounters a new zone instead of erroring Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

andyasp

Thank you, looks great!

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

* Add CHANGELOG.md entry for #107 Signed-off-by: JordanRushing <rushing.jordan@gmail.com> * Update CHANGELOG.md --------- Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Co-authored-by: Andy Asp <90626759+andyasp@users.noreply.github.com>

christopherzli · 2024-01-20T09:21:28Z

Can we include this in the read.me as well? Thanks!

JordanRushing added 18 commits November 14, 2023 11:24

Implement zoneTracker for coordinating downscales across statefulsets…

f5be68c

… using a file in object storage for the prepare_downscale admission webhook Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Merge branch 'main' into prep-downscale-last-downscale-objstore

7d1958f

Expand tests; fix deadlocks in zoneTracker; add logic to create an in…

60b37ec

…itial zone file if not present Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Add zoneTracker support for s3; validate new config flags for object …

bd9ef2d

…storage; update zoneTracker tests; run `go mod vendor` & `go mod tidy` Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Fix s3 config env variable names

b11ec77

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Add endpoint to Azure bucket client config; update config validation

f2b53ef

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Add additional zoneTracker tests

93efd5d

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Remove redundant defer r.close() from loadZones

7bd2c6d

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Remove unused ctx from function definitions; update comment for saveZ…

6e4bf58

…ones; fix deadlock in lastDownscaled Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Merge branch 'main' into prep-downscale-last-downscale-objstore

0f40551

Resolve merge conflicts with main; run go mod tidy & go mod vendor

0b07920

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Convert zoneTracker zones map to use zoneInfo struct instead of strin…

c269119

…g; update comment on loadZones to indicate unique sts name assumption for keys

Clean up loadZones function

f62c470

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Fix setDownscaled to actually update the zoneInfo in the map

72c1cda

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Switch from object storage to a config map for the zoneTracker; remov…

31f1fcd

…e bucket code Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Further function cleanup

122b288

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Run go mod tidy & go mod vendor to clean up after removing object…

cbed8ff

… storage dependency Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

JordanRushing requested a review from a team as a code owner December 15, 2023 02:25

MichelHollands approved these changes Dec 15, 2023

View reviewed changes

jhalterman reviewed Dec 15, 2023

View reviewed changes

jhalterman approved these changes Dec 15, 2023

View reviewed changes

JordanRushing mentioned this pull request Dec 15, 2023

Implement zoneTracker for coordinating downscales across statefulsets using a file in object storage for the prepare_downscale admission webhook #96

Closed

3 tasks

andyasp reviewed Dec 20, 2023

View reviewed changes

JordanRushing added 2 commits January 3, 2024 11:39

Update comments to reference the zone ConfigMap instead of the zone f…

35b741f

…ile; run `go mod tidy` & `go mod vendor`; simplify zoneTracker config validation conditional Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Refactor prepareDownscale; implement prepareDownscale in zoneTracker;…

cdf9d74

… add tests for new functions post-refactor Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

JordanRushing added 3 commits January 8, 2024 16:41

Remove zt parameter from base prepareDownscale; fix test

ac0136a

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Fix test endpoint

2db86c8

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Merge pull request #1 from JordanRushing/prep-downscale-last-downscal…

0b454bb

…e-configmap-refactor Prep downscale last downscale configmap refactor

MichelHollands approved these changes Jan 9, 2024

View reviewed changes

JordanRushing added 2 commits January 9, 2024 11:42

Move zoneTracker assignment into conditional in PrepareDownscale

aa96f21

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

Create an entry in the zone map for a statefulset when lastDownscaled…

94e8d08

… encounters a new zone instead of erroring Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

andyasp approved these changes Jan 9, 2024

View reviewed changes

JordanRushing merged commit 80c46d3 into grafana:main Jan 9, 2024
6 checks passed

JordanRushing added a commit to JordanRushing/rollout-operator that referenced this pull request Jan 9, 2024

Add CHANGELOG.md entry for grafana#107

c0cc5f0

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

JordanRushing mentioned this pull request Jan 9, 2024

Add CHANGELOG.md entry for #107 #112

Merged

charleskorn mentioned this pull request Jan 12, 2024

Improve tracing #115

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement zoneTracker for coordinating downscales across statefulsets using a ConfigMap for the prepare_downscale admission webhook #107

Implement zoneTracker for coordinating downscales across statefulsets using a ConfigMap for the prepare_downscale admission webhook #107

JordanRushing commented Dec 15, 2023

MichelHollands left a comment

jhalterman Dec 15, 2023

andyasp left a comment

andyasp Dec 20, 2023

andyasp Dec 20, 2023

andyasp Dec 20, 2023

andyasp Dec 20, 2023

andyasp Dec 20, 2023

andyasp Dec 20, 2023

andyasp Dec 20, 2023

andyasp Dec 20, 2023

andyasp Dec 20, 2023

JordanRushing commented Jan 8, 2024 •

edited

MichelHollands left a comment

MichelHollands Jan 9, 2024

andyasp left a comment

christopherzli commented Jan 20, 2024

	// If using zoneTracker instead of downscale annotations, check the zone file for recent downscaling.
	// If using zoneTracker instead of downscale annotations, check the zone ConfigMap for recent downscaling.

	func TestLoadZonesEmptyBucket(t *testing.T) {
	func TestLoadZonesEmptyConfigMap(t *testing.T) {

	// If the zone file does not exist, populate it with the current time for each zone and upload it to the bucket.
	// If the zone file does not exist, populate it with the current time for each zone and upload it to the ConfigMap.

Implement zoneTracker for coordinating downscales across statefulsets using a ConfigMap for the prepare_downscale admission webhook #107

Implement zoneTracker for coordinating downscales across statefulsets using a ConfigMap for the prepare_downscale admission webhook #107

Conversation

JordanRushing commented Dec 15, 2023

MichelHollands left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyasp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JordanRushing commented Jan 8, 2024 • edited

MichelHollands left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andyasp left a comment

Choose a reason for hiding this comment

christopherzli commented Jan 20, 2024

JordanRushing commented Jan 8, 2024 •

edited