WRKLDS-728: controller/apiservice: allow to disable/delete an API service resources #1534

ingvagabund · 2023-05-30T11:49:26Z

SSIA

Needed by openshift/cluster-openshift-apiserver-operator#532

openshift-ci-robot · 2023-05-30T11:51:02Z

@ingvagabund: This pull request references WRKLDS-728 which is a valid jira issue.

In response to this:

SSIA

Needed by openshift/cluster-openshift-apiserver-operator#532

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

p0lyn0mial · 2023-05-30T12:20:27Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

+// GetAPIServicesToMangeFunc provides a full list of managed APIService items.
+// The list needs to always contain all the managed APIServices so only managed
+// objects are removed without removing user-created/unmanaged objects.
+type GetAPIServicesToMangeFunc func() ([]ManagedAPIService, error)


why this function couldn't return services that are enabled :) ?

also this is not a backward-compatible change which will break users of this function

how will you know if an api service is not enabled ?

why this function couldn't return services that are enabled :) ?

The controller needs a list of APIServices that are disabled. So they can be deleted if present.

how will you know if an api service is not enabled ?

Configurable by anyone who consumes the APIService controller. Atm, the OA operator constructs the list based on enabled capabilities.

p0lyn0mial · 2023-05-30T12:23:39Z

also the title of this pr is a bit misleading for me, are we actually going to delete something?

p0lyn0mial · 2023-05-30T12:24:42Z

pkg/operator/apiserver/controller/apiservice/apigroup.go

-			curr := coordinate{namespace: apiService.Spec.Service.Namespace, name: apiService.Spec.Service.Name}
+		for _, managedAPIService := range apiServices {
+			if !managedAPIService.Enabled {
+				continue


since we are skipping disabled services I would simply return only enabled services in GetAPIServicesToMangeFunc

Both newEndpointPrecondition and checkDiscoveryForByAPIServices can be probably given only a list of enabled services.

yes, I would prefer that :)

p0lyn0mial · 2023-05-30T12:27:43Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

+	for _, managedApiService := range apiServices {
+		// Remove disabled APIService
+		if !managedApiService.Enabled {
+			if err := c.apiregistrationv1Client.APIServices().Delete(ctx, managedApiService.APIService.Name, metav1.DeleteOptions{}); err != nil && !apierrors.IsNotFound(err) {


ah, sorry, i was reading top-to-bottom, i think this is the place that will remove an api service

p0lyn0mial · 2023-05-30T12:32:28Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

+// GetAPIServicesToMangeFunc provides a full list of managed APIService items.
+// The list needs to always contain all the managed APIServices so only managed
+// objects are removed without removing user-created/unmanaged objects.
+type GetAPIServicesToMangeFunc func() ([]ManagedAPIService, error)


have you considered adding a new function which will return services to be removed ?

or changing the signature of this method to return enabled []*apiregistrationv1.APIService, disabled []*apiregistrationv1.APIService, error)

That would be an alternative. In any case, still a breaking change.

p0lyn0mial · 2023-05-30T12:38:49Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

+type ManagedAPIService struct {
+	// Enabled signals whether an APIService is enabled.
+	// Any disabled APIService is removed from a cluster.
+	Enabled bool


will this field be used by all clients or only for some?

Once such struct is created, it's intended to be used only by the controller.

What I wanted to ask was if all operands will be required to provide this information.

In general my preference would be to check how many operators consume this function.
If the number is low then I would change the signature of the method to enabled []*apiregistrationv1.APIService, disabled []*apiregistrationv1.APIService, error).

Based on https://github.com/search?q=org%3Aopenshift%20NewAPIServiceController&type=code NewAPIServiceController is used only once.
Based on https://github.com/search?q=org%3Aopenshift%20WithAPIServiceController&type=code WithAPIServiceController is used only by openshift-apiserver and cluster authentication operators.

What I wanted to ask was if all operands will be required to provide this information.

Yeah. They have to. https://github.com/openshift/cluster-authentication-operator/blob/1b2c022034879e09e324bbdc50b84b78ce8de4ee/pkg/operator/starter.go#L793-L798 builds the list explicitly as well.

An alternative would be to provide a filter function that if not provide will include every api service. On the other hand such a filter will get invoked all over again instead of constructing a list of enabled/disabled service only once at the beginning. The list of enabled/disabled api services is expected to be static or changed very rarely.

OK, since there are only two operators using this controller I would:

Change the signature of the method to GetAPIServicesToMangeFunc func() (enabled []*apiregistrationv1.APIService, disabled []*apiregistrationv1.APIService, error)

Rename precondition to preconditionForEnabledAPIServices

Split the sync method into two flows, one for reconciling enabled services and the second for reconciling disabled services.

Does it make sense ?

Done. It looks better now.

Thanks, I have added a few more comments/questions, PTAL.

p0lyn0mial · 2023-05-31T10:58:31Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

@@ -118,7 +122,7 @@ func (c *APIServiceController) sync(ctx context.Context, syncCtx factory.SyncCon
 		return err
 	}

-	err = c.syncAPIServices(ctx, apiServices, syncCtx.Recorder())
+	err = c.syncAPIServices(ctx, enabledApiServices, disabledApiServices, syncCtx.Recorder())


How about adding a separate method for reconciling disalbedApiServices and calling it before preconditionForEnabledAPIServices (line 101)?

Having two separate methods would allow us to reconcile enabledApiServices and disabledApiServices independently. I think we always want to reconcile both and aggregate the errors. What do you think ?

p0lyn0mial · 2023-05-31T11:00:17Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

 	errs := []error{}
 	var availableConditionMessages []string

-	for _, apiService := range apiServices {
+	for _, apiService := range disabledApiServices {
+		if err := c.apiregistrationv1Client.APIServices().Delete(ctx, apiService.Name, metav1.DeleteOptions{}); err != nil && !apierrors.IsNotFound(err) {


I think we should get an APIService from the cache first as that would allow us to put less pressure on the server.
We should issue a live request only when we know the resource must be deleted.
What do you think?

p0lyn0mial · 2023-05-31T11:01:39Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

-type GetAPIServicesToMangeFunc func() ([]*apiregistrationv1.APIService, error)
+// GetAPIServicesToMangeFunc provides a two list of managed APIService items.
+// Both lists need to always contain all the managed APIServices so only managed
+// objects are removed without removing user-created/unmanaged objects.


what do you mean by without removing user-created/unmanaged objects ?

Comment updated. The controller needs to avoid deleting APIServices that are not part of the OpenShift API. I.e. the controller needs to explicitly know which APIServices are disabled instead of deleting all not enabled APIServices.

ingvagabund · 2023-06-01T11:06:59Z

/retest-required

p0lyn0mial · 2023-06-01T11:41:25Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go


 	operatorClient          v1helpers.OperatorClient
 	kubeClient              kubernetes.Interface
 	apiregistrationv1Client apiregistrationv1client.ApiregistrationV1Interface
+	apiservicelister        apiregistrationv1lister.APIServiceLister


isn't that strange that this controller didn't use apiservicelister ?

Probably not taken into account when #667 was reviewed.

p0lyn0mial · 2023-06-01T11:42:08Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

-		for _, apiService := range apiServices {
-			if err := c.apiregistrationv1Client.APIServices().Delete(ctx, apiService.Name, metav1.DeleteOptions{}); err != nil {
+		for _, apiService := range append(enabledApiServices, disabledApiServices...) {
+			if _, err := c.apiservicelister.Get(apiService.Name); err == nil {


how about simply calling syncDisabledAPIServices with combined enabledApiServices, disabledApiServices ?

p0lyn0mial · 2023-06-01T11:46:05Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go


 	for _, apiService := range apiServices {
+		if _, err := c.apiservicelister.Get(apiService.Name); err == nil {


should we also consider a case in which a service has been already marked for deletion but hasn't been deleted?
(I think that would require checking the DeletionTimestamp field)

That should be idempotent. Deleting an object that's in the process of deletion should be a no-op. While Get should still return the object without an error.

It is idempotent but we will hammer the API unnecessary. Does it make sense?

I agree this will reduce the toll. On the other hand I wonder if checking the DeletionTimestamp is something we already do in other controllers or whether this is a novel improvement. If the latter, I prefer to refactor the code more and implement generic methods like DeleteOnlyIfPresent. Probably automatically generated for all our APIs. So we don't re-invent the check all over the place. Nevertheless, as part of another PR. What do you think?

I don't think it is novel approach. For example, it is being used in:

https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authenticator/token/bootstrap/bootstrap.go#L107

https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/serviceaccount/legacy_serviceaccount_token_cleaner.go#L147

I can try to find more examples if you want.

I was wondering if we have an example in our operators. Nevertheless, checking for the DeletionTimestamp is easy to implement. Will do.

p0lyn0mial · 2023-06-01T11:48:47Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

+
+	disErr := c.syncDisabledAPIServices(ctx, disabledApiServices)
+
+	ready, err := c.preconditionForEnabledAPIServices(enabledApiServices)


what if both syncDisabledAPIServices and preconditionForEnabledAPIServices return some error?
do we want to report errors from syncDisabledAPIServices ?
i think the errors will be ignored (not reported) as it is right now.

If syncDisabledAPIServices returns an error, there are two cases that can happen:

preconditionForEnabledAPIServices returns err != nil or !ready in which case the APIServicesAvailable condition gets reported as false. Making error returned by syncDisabledAPIServices kinda obsolete since the APIServicesAvailable is already false.

preconditionForEnabledAPIServices returns err == nil and ready in which case the error returned by syncDisabledAPIServices gets checked. Producing APIServicesAvailable condition set to false. Just with a different error string.

Does setting APIServicesAvailable when syncDisabledAPIServices returns some errors make sense ?

If all services were available and we failed to remove one service would you expect to find the error in APIServicesAvailable ? Maybe we should report it as a separate condition? (Especially that we already made distinction between enabled/disabled on the API level)

The current implementation does not make the reported condition worse. Right now it will get reported with Reason: Error. Whether that happens before or after preconditionForEnabledAPIServices makes no difference. Failing to remove or create an APIservice object falls under the same category of an error. If we decide to distinguish between error related to removing and creating an APIService, we should probably introduce a new condition for both Failed to create and Failed to check discovery as well (implemented under syncEnabledAPIServices).

I wonder whether there's a value in reporting non-nil error for return c.syncDisabledAPIServices(ctx, append(enabledApiServices, disabledApiServices...)) through a condition under:

switch operatorConfigSpec.ManagementState { ... case operatorsv1.Removed: enabledApiServices, disabledApiServices, err := c.getAPIServicesToManageFn() if err != nil { return err } return c.syncDisabledAPIServices(ctx, append(enabledApiServices, disabledApiServices...))

I'm not sure if we want to go Degraded. I think that:

Type: "APIServicesDeleted", Status: operatorv1.ConditionFalse, Reason: "DisabledAPIServicesPresent", Message: err.Error(),

should be fine.

Also, We could split the APIServiceController.sync method into two flows for simplicity. I think I would do:

func (c *APIServiceController) sync(ctx context.Context, syncCtx factory.SyncContext) error { operatorConfigSpec, _, _, err := c.operatorClient.GetOperatorState() if err != nil { return err } switch operatorConfigSpec.ManagementState { case operatorsv1.Managed: case operatorsv1.Unmanaged: return nil case operatorsv1.Removed: enabledApiServices, disabledApiServices, err := c.getAPIServicesToManageFn() if err != nil { return err } return c.syncDisabledAPIServices(ctx, append(enabledApiServices, disabledApiServices...)) default: syncCtx.Recorder().Warningf("ManagementStateUnknown", "Unrecognized operator management state %q", operatorConfigSpec.ManagementState) return nil } var errors []err // or if we rename syncDisabledAPIServices to something else then // we could call it syncDisabledAPIServices (which would also set the new status) err = c.reconcileDisabledServices() if err != nil { // append to errors } err = c.syncEnabledServices() if err != nil { // append to errors } return errors }

Would syncEnabledServices include calling the preconditionForEnabledAPIServices as well?

yes, I think so :)

as an optimisation step we could collect conditions from the individual sync* methods and update the config just once :)

ingvagabund · 2023-06-01T13:31:13Z

/label tide/merge-method-squash

ingvagabund · 2023-06-01T13:41:38Z

/unlabel tide/merge-method-squash

ingvagabund · 2023-06-01T13:43:54Z

/remove-label tide/merge-method-squash

ingvagabund · 2023-06-01T13:44:20Z

/shrug

p0lyn0mial · 2023-06-16T08:28:10Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go


 	for _, apiService := range apiServices {
+		if apiServiceObj, err := c.apiservicelister.Get(apiService.Name); err == nil {
+			if apiServiceObj.DeletionTimestamp != nil {
+				continue


Technically objects can stuck in deletion (finalisers) for a long time.
Do we want to handle this case somehow (log, set something on the condition) ?

syncEnabledAPIServices returns an error when the corresponding api service Available condition is not ready. Updated the code to return error when a corresponding service is not deleted yet.

update: we decided to log instead of returning the error

p0lyn0mial · 2023-06-16T08:37:04Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

 	if err != nil {
 		if _, _, updateErr := v1helpers.UpdateStatus(ctx, c.operatorClient, v1helpers.UpdateConditionFn(operatorv1.OperatorCondition{
 			Type:    "APIServicesAvailable",
 			Status:  operatorv1.ConditionFalse,
 			Reason:  "ErrorCheckingPrecondition",
 			Message: err.Error(),
-		})); updateErr != nil {
-			return errors.NewAggregate([]error{err, updateErr})
+		}), v1helpers.UpdateConditionFn(conditionAPIServicesDegraded)); updateErr != nil {


I don't want to complicate the code but it looks like it might be easy to forget which conditions should be updated. I like the way it was done in the workload controller (

library-go/pkg/operator/apiserver/controller/workload/workload.go

Line 182 in c021d42

var updateGenerationFn func(newStatus *operatorv1.OperatorStatus) error

). Maybe we could do something similar here?

I think it boils down to:

define conditions with default values

register an update function for conditions in the defer statement

when you need to end processing, just set the condition and return (it will be updated in the defer statement).

Done. I did not find any use for updateGenerationFn though.

p0lyn0mial · 2023-06-19T13:30:36Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

+		Status: operatorv1.ConditionTrue,
+	}
+
+	if syncEnabledAPIServicesErr != nil {


I think we should check if for any error or always for simplicity, wdyt?

p0lyn0mial · 2023-06-19T13:31:07Z

pkg/operator/apiserver/controller/apiservice/apiservice_controller.go

+		// the next process will perform the checks immediately after the startup
+		select {
+		case <-ctx.Done():
+			nerr := fmt.Errorf("the operator is shutting down, skipping updating availability of the aggreaged APIs, err = %v", syncEnabledAPIServicesErr)


just adjust comment so that it is more generic

p0lyn0mial · 2023-06-19T14:07:13Z

/lgtm
/approve

openshift-ci · 2023-06-19T14:08:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ingvagabund, p0lyn0mial

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/operator/apiserver/OWNERS~~ [p0lyn0mial]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2023-06-19T14:11:34Z

@ingvagabund: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

p0lyn0mial · 2023-06-19T14:12:32Z

/hold

p0lyn0mial · 2023-06-19T14:13:41Z

just one more thing, it looks like we are going to always set the new condition, even if the toRemoveServices was empty.

p0lyn0mial · 2023-06-19T14:19:02Z

just one more thing, it looks like we are going to always set the new condition, even if the toRemoveServices was empty.

it seems that setting the new condition conditionally would make the code more complicated. Besides on clusters with this feature disabled it will always be set to false.

/hold cancel

openshift-ci bot requested review from p0lyn0mial and tkashem May 30, 2023 11:52

p0lyn0mial reviewed May 30, 2023

View reviewed changes

ingvagabund force-pushed the apiservice-controller-allow-to-disable-apiservices branch 2 times, most recently from faf15e3 to 3ca87e7 Compare May 31, 2023 09:49

p0lyn0mial reviewed May 31, 2023

View reviewed changes

p0lyn0mial reviewed Jun 1, 2023

View reviewed changes

openshift-ci bot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Jun 1, 2023

ingvagabund force-pushed the apiservice-controller-allow-to-disable-apiservices branch from 8d1b529 to fcd73e9 Compare June 1, 2023 13:40

openshift-ci bot added ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯ and removed tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. labels Jun 1, 2023

ingvagabund force-pushed the apiservice-controller-allow-to-disable-apiservices branch 3 times, most recently from 05aff02 to ea20618 Compare June 15, 2023 11:23

ingvagabund force-pushed the apiservice-controller-allow-to-disable-apiservices branch from ea20618 to 8a3367e Compare June 15, 2023 11:28

p0lyn0mial reviewed Jun 16, 2023

View reviewed changes

ingvagabund force-pushed the apiservice-controller-allow-to-disable-apiservices branch from 8a3367e to ac3fc9b Compare June 19, 2023 11:43

p0lyn0mial reviewed Jun 19, 2023

View reviewed changes

controller/apiservice: allow to disable/delete an API service resources

75c7d51

ingvagabund force-pushed the apiservice-controller-allow-to-disable-apiservices branch from ac3fc9b to 75c7d51 Compare June 19, 2023 14:04

openshift-ci bot assigned p0lyn0mial Jun 19, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 19, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 19, 2023

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 19, 2023

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 19, 2023

openshift-merge-robot merged commit 654ebf2 into openshift:master Jun 19, 2023
4 checks passed

ingvagabund deleted the apiservice-controller-allow-to-disable-apiservices branch June 19, 2023 14:23


		for _, apiService := range apiServices {
		if _, err := c.apiservicelister.Get(apiService.Name); err == nil {


		disErr := c.syncDisabledAPIServices(ctx, disabledApiServices)

		ready, err := c.preconditionForEnabledAPIServices(enabledApiServices)

WRKLDS-728: controller/apiservice: allow to disable/delete an API service resources #1534

WRKLDS-728: controller/apiservice: allow to disable/delete an API service resources #1534

Conversation

ingvagabund commented May 30, 2023

openshift-ci-robot commented May 30, 2023 • edited by openshift-ci bot

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p0lyn0mial commented May 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingvagabund May 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingvagabund Jun 1, 2023 • edited

Choose a reason for hiding this comment

ingvagabund commented Jun 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingvagabund Jun 2, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p0lyn0mial Jun 1, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingvagabund Jun 2, 2023 • edited

Choose a reason for hiding this comment

ingvagabund Jun 2, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingvagabund commented Jun 1, 2023

ingvagabund commented Jun 1, 2023

ingvagabund commented Jun 1, 2023

ingvagabund commented Jun 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p0lyn0mial commented Jun 19, 2023

openshift-ci bot commented Jun 19, 2023

openshift-ci bot commented Jun 19, 2023

p0lyn0mial commented Jun 19, 2023

p0lyn0mial commented Jun 19, 2023

p0lyn0mial commented Jun 19, 2023

openshift-ci-robot commented May 30, 2023 •

edited by openshift-ci bot

ingvagabund May 30, 2023 •

edited

ingvagabund Jun 1, 2023 •

edited

ingvagabund Jun 2, 2023 •

edited

p0lyn0mial Jun 1, 2023 •

edited

ingvagabund Jun 2, 2023 •

edited

ingvagabund Jun 2, 2023 •

edited