CFE-866: Add secret manager for routes to consume secret_monitor #1626

chiragkyal · 2023-11-22T07:33:33Z

Description

This PR adds route layer (introduces a new secretmanager package) on top of #1675 to consume the secret_monitor.
These changes primarily focus on enhancing the functionality of managing secrets associated with routes.

Important Functions:

RegisterRoute: Allows registering a route with a secret, enabling dynamic monitoring of secret changes associated with the route.
UnregisterRoute: Provides functionality to remove the registration of a route from the monitor.
GetSecret: Enables retrieving the secret object registered with a route.

Related: CFE-866
Implements: openshift/enhancements#1307

openshift-ci-robot · 2023-11-22T07:33:37Z

@chiragkyal: This pull request references CFE-866 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.15.0" version, but no target version was set.

In response to this:

Related: CFE-866
Implements: openshift/enhancements#1307

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-11-22T07:33:37Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

chiragkyal · 2023-11-22T07:34:34Z

Continued from #1518

chiragkyal · 2024-01-15T12:57:02Z

/retest

pkg/route/secret/manager.go

deads2k

solid start

pkg/route/secret/manager.go

pkg/route/secret/monitor.go

pkg/route/secret/manager.go

pkg/route/secret/secret_monitor.go

pkg/route/secret/manager.go

pkg/route/secret/monitor.go

candita · 2024-01-31T16:35:20Z

/assign @alebedev87

pkg/route/secret/monitor.go

pkg/route/secret/secret_monitor.go

deads2k · 2024-02-01T21:27:15Z

pkg/route/secret/manager.go

+	// each route (namespace/routeName) should be registered only once with any secret.
+	// Note: inside a namespace multiple different routes can be registered(watch) with a common secret
+	key := generateKey(namespace, routeName)
+	if _, exists := m.registeredHandlers[key]; exists {


When the secret name changes, what should happen? Probably a remove and then an add?

Correct, unregister old secret and register new secret.

deads2k · 2024-02-01T21:29:36Z

pkg/route/secret/manager.go

+		return apierrors.NewInternalError(fmt.Errorf("route already registered with key %s", key))
+	}
+
+	handlerRegistration, err := m.monitor.AddSecretEventHandler(namespace, secretName, m.secretHandler)


Probably worth considering how your consuming layers will expect the "route changed secret" to be represented to them.

deads2k · 2024-02-01T21:30:33Z

pkg/route/secret/manager.go

+	}
+
+	// wait for informer store sync, to load secret
+	if err := wait.PollImmediate(10*time.Millisecond, time.Second, func() (done bool, err error) { return handlerRegistration.HasSynced(), nil }); err != nil {


Because you waited for the HasSync in the step above, this wait should no longer be necessary, right?

That HasSync will be called only once after starting the informer. Correct me if I'm wrong, but I think we need to call HasSync for every GetSecret invocation to update the cache with the latest state of the resource.

chiragkyal · 2024-02-06T11:31:33Z

/test all

chiragkyal · 2024-02-06T11:36:04Z

As discussed, we'll split this PR into 2 PRs.
I've raised #1675 which will have only secret monitoring logic and this PR will be updated to consume that for routes.

pkg/route/secretmanager/manager_test.go

pkg/route/secret/monitor.go

alebedev87

Second iteration, mostly the nit picks.

pkg/route/secretmanager/manager_test.go

pkg/route/secret/monitor_test.go

pkg/route/secret/secret_monitor_test.go

alebedev87 · 2024-04-16T13:11:21Z

pkg/route/secret/secret_monitor.go

+	if err != nil {
+		return nil, err
+	}


Why the error is checked after the exist variable?

Informer's cache (GetByKey()) is returning error even with exists , so we should handle error instead of ignoring it.

Sorry, I didn't make the question clear. I meant that why not checking the error before we check exist. It may be misleading as exist may be false just because it's a default value returned in case of an error. This may result in a wrong error returned from the function.

Okay, that makes sense. Moved err check before exist. Thanks!

pkg/route/secretmanager/manager.go

chiragkyal · 2024-04-19T08:22:26Z

/hold
For #1675
We want to keep the secretmanager and monitor in two separate PRs.

alebedev87 · 2024-04-19T10:11:47Z

/lgtm

Miciah

This looks great. All my comments are suggestions to fix typos in comments or log or error messages and a few minor questions around error values and test code. Feel free to take or leave my suggestions, or handle them in a follow-up.
/lgtm

Miciah · 2024-04-20T04:33:37Z

pkg/secret/monitor.go

+	return i.informer.HasSynced()
+}
+
+// StartInformer starts and runs the informer util the provided context is canceled,


Suggested change

// StartInformer starts and runs the informer util the provided context is canceled,

// StartInformer starts and runs the informer until the provided context is canceled,

Miciah · 2024-04-20T04:40:19Z

pkg/secret/monitor.go

+	defer i.lock.Unlock()
+
+	if i.stopped {
+		return nil, fmt.Errorf("can not add handler %v to already stopped informer", handler)


Suggested change

return nil, fmt.Errorf("can not add handler %v to already stopped informer", handler)

return nil, fmt.Errorf("cannot add handler %v to already stopped informer", handler)

Miciah · 2024-04-20T05:07:38Z

pkg/secret/secret_monitor.go

+	// secret identifier (namespace/secret)
+	key := NewObjectKey(namespace, secretName)
+
+	// start secret informer if monitor does not exists


Suggested change

// start secret informer if monitor does not exists

// Start secret informer if monitor does not exist.

Miciah · 2024-04-20T05:22:08Z

pkg/secret/secret_monitor.go

+			return nil, fmt.Errorf("failed waiting for cache sync")
+		}
+
+		// add item key to monitors map // add watch to the list


The // add watch to the list looks conspicuous—is this a stale comment that accidentally got joined with the // add item key to monitors map comment?

Yeah, it was a stale comment. I've removed // add watch to the list comment

Miciah · 2024-04-20T05:22:52Z

pkg/secret/secret_monitor.go

+		// add item key to monitors map // add watch to the list
+		s.monitors[key] = m
+
+		klog.Info("secret informer started", " item key ", key)


Here and below, are the extra spaces intended to be there?

Suggested change

klog.Info("secret informer started", " item key ", key)

klog.Info("secret informer started", "item key", key)

Yes, these are intended because it looks like klog.Info is not putting spaces between two comma , separated strings.

Ah, I was thinking these were key-value pairs, like go-logr/logr uses. Why not use a format?

Suggested change

klog.Info("secret informer started", " item key ", key)

klog.Infof("secret informer started for item with key %v", key)

This is a minor issue and can be addressed in a follow-up if you decide to take my suggestion.

Miciah · 2024-04-20T06:49:56Z

pkg/secret/monitor_test.go

+			// for handling nil pointer dereference
+			defer func() {
+				if err := recover(); err != nil && !s.expectErr {
+					t.Errorf("unexpected error %v", err)
+				}
+			}()


This recover block highlights an oddity of this test: The "nil handler is provided" test case isn't actually testing the behavior of AddEventHandler; it's testing the behavior of the Go runtime, to verify that dereferencing a nil pointer causes a panic—right?

It was there to recover from panic when handle.GetHandler() is executed inside RemoveEventHandler, given a nil SecretEventHandlerRegistration.

I think a much easier way would be to add if statement inside RemoveEventHandler to check nil SecretEventHandlerRegistration.

I've added the following the latest commit

if handle == nil { return fmt.Errorf("nil handler registration is provided") }

I like it—the test is simpler, and the method is more robust. Thanks!

Miciah · 2024-04-20T07:10:06Z

pkg/secret/secret_monitor_test.go

+			// Delete keys if already removed
+			for _, keyRemoved := range s.alreadyRemoved {
+				delete(sm.monitors, keyRemoved)
+			}


Is this testing something that is expected to happen, or is this testing that RemoveSecretEventHandler behaves correctly if the secret monitor's internal data structures are inexplicably corrupted?

is this testing that RemoveSecretEventHandler behaves correctly if the secret monitor's internal data structures are inexplicably corrupted?

For this reason(^)

It is trying to simulate the condition when the secret monitor does not contain the intended key. This is unlikely to happen, but I think we should have a test to check RemoveSecretEventHandler catches this issue.

m, exists := s.monitors[key] if !exists { return fmt.Errorf("secret monitor already removed for item key %v", key) }

Miciah · 2024-04-20T07:13:57Z

pkg/secret/secret_monitor_test.go

+
+			gotSec, gotErr := sm.GetSecret(context.TODO(), h)
+			if (gotErr != nil) != s.expectErr {
+				t.Fatalf("expected errors to be %t, but got %t", s.expectErr, err != nil)


Suggested change

t.Fatalf("expected errors to be %t, but got %t", s.expectErr, err != nil)

t.Fatalf("expected errors to be %t, but got %t", s.expectErr, gotErr != nil)

Miciah · 2024-04-20T07:28:07Z

pkg/route/secretmanager/manager.go

+		return apierrors.NewInternalError(fmt.Errorf("route already registered with key %s", key))
+	}
+
+	// Add a secret event handler for the specified namespace and secret, with the handler functions.
+	klog.V(5).Infof("trying to add handler for key %s with secret %s", key, secretName)
+	handlerRegistration, err := m.monitor.AddSecretEventHandler(ctx, namespace, secretName, handler)
+	if err != nil {
+		return apierrors.NewInternalError(err)


Is there a reason to use apierrors.NewInternalError here? I thought apierrors.NewInternalError was intended for errors that originated from the API server. And even if it's safe to use outside of the API server context, it isn't clear to me why the error needs to be wrapped.

That said, I don't see any router code that uses apierrors.IsInternalError to do something special with such errors, so using apierrors.NewInternalError shouldn't cause any problems for now.

Okay, I also think router code will not make use of this wrapped error by calling apierrors.IsInternalError. So, to keep it consistent, I've drooped the apierrors usage from this package.

Miciah · 2024-04-20T07:39:34Z

pkg/route/secretmanager/manager.go

+		return nil, apierrors.NewInternalError(fmt.Errorf("no handler registered with key %s", key))
+	}
+
+	// get secret from secrt monitor's cache using registered handler


Suggested change

// get secret from secrt monitor's cache using registered handler

// Get the secret from the secret monitor's cache using the registered handler.

chiragkyal · 2024-04-22T08:25:59Z

Thanks @Miciah for the review comments, I've addressed them. PTAL. If there isn't any new suggestions, would appreciated your lgtm in #1675.
I've cherry-picked the secret monitor related changes (which addressed your comments) into #1675.

Miciah · 2024-04-22T17:00:20Z

Thanks!
/lgtm

Signed-off-by: chiragkyal <ckyal@redhat.com>

openshift-ci-robot · 2024-04-25T15:50:11Z

@chiragkyal: This pull request references CFE-866 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This PR adds route layer (secretmanager) on top of #1675 to consume the secret_monitor.

Related: CFE-866
Implements: openshift/enhancements#1307

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

chiragkyal · 2024-04-25T15:53:54Z

/unhold
#1675 is merged. Rebased to master.

openshift-ci · 2024-04-25T15:57:40Z

@chiragkyal: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2024-04-25T16:02:47Z

@chiragkyal: This pull request references CFE-866 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Description

This PR adds route layer (introduces a new secretmanager package) on top of #1675 to consume the secret_monitor.
These changes primarily focus on enhancing the functionality of managing secrets associated with routes.

Important Functions:

RegisterRoute: Allows registering a route with a secret, enabling dynamic monitoring of secret changes associated with the route.

UnregisterRoute: Provides functionality to remove the registration of a route from the monitor.

GetSecret: Enables retrieving the secret object registered with a route.

Related: CFE-866
Implements: openshift/enhancements#1307

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Miciah · 2024-04-25T16:07:46Z

/lgtm

openshift-ci · 2024-04-25T16:09:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alebedev87, chiragkyal, Miciah

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/route/OWNERS~~ [Miciah]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Vendoring openshift/library-go#1626 Signed-off-by: chiragkyal <ckyal@redhat.com>

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 22, 2023

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 22, 2023

chiragkyal force-pushed the feature/secret-manager branch 2 times, most recently from 2f5683e to 632b0c8 Compare January 18, 2024 10:18

chiragkyal changed the title ~~[WIP] CFE-866: Add secret monitor for Routes~~ CFE-866: Add secret monitor for Routes Jan 18, 2024

chiragkyal marked this pull request as ready for review January 18, 2024 10:18

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 18, 2024

openshift-ci bot requested review from candita and knobunc January 18, 2024 10:21

chiragkyal commented Jan 18, 2024

View reviewed changes

pkg/route/secret/manager.go Outdated Show resolved Hide resolved

deads2k requested changes Jan 25, 2024

View reviewed changes

openshift-ci bot assigned alebedev87 Jan 31, 2024