WRKLDS-728: Capabilities: drop build/apps APIService when capabilities are not enabled #532

ingvagabund · 2023-05-02T14:25:23Z

Unless a capability is unknown, install APIService object for corresponding API only when enabled.

openshift-ci-robot · 2023-05-02T14:25:27Z

@ingvagabund: This pull request references WRKLDS-728 which is a valid jira issue.

In response to this:

Unless a capability is unknown, install APIService object for corresponding API only when enabled.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ingvagabund · 2023-07-14T11:47:27Z

/retest-required

ingvagabund · 2023-07-17T11:25:22Z

/retest-required

ingvagabund · 2023-08-04T08:15:59Z

/retest-required

p0lyn0mial · 2023-08-04T10:24:04Z

pkg/operator/starter.go

 }

 func apiServicesReferences() []configv1.ObjectReference {
 	ret := []configv1.ObjectReference{}
-	for _, apiService := range apiServices() {
-		ret = append(ret, configv1.ObjectReference{Group: "apiregistration.k8s.io", Resource: "apiservices", Name: apiService.Spec.Version + "." + apiService.Spec.Group})
+	for _, apiService := range apiServiceGroupVersions {


why this doesn't have to operate only on the enabled services?

apiServicesReferences() is consumed by .WithClusterOperatorStatusController which uses the references to build a list of related objects. In case a reference has no relevant object, it gets ignored.

ok, it looks like this is purely informational. It doesn't have any impact on the core logic.

p0lyn0mial · 2023-08-04T10:37:28Z

pkg/operator/starter.go

-		{Group: "security.openshift.io", Version: "v1"},
-		{Group: "template.openshift.io", Version: "v1"},
+func apiServices(clusterVersionInformer configinformersconfigv1.ClusterVersionInformer) ([]*apiregistrationv1.APIService, []*apiregistrationv1.APIService, error) {
+	clusterVersion, err := clusterVersionInformer.Lister().Get("version")


I think that we should either change the NewAPIServiceController controller or this function. Otherwise we cannot guarantee the clusterVersionInformer will be synced. Am I right ?

In addition to that I think that we should add a reactor to the NewAPIServiceController to catch changes to clusterVersionInformer. WDYT?

apiServices can have various implementations so any change needs to happen inside apiService.

Are you suggesting to invoke WaitForCacheSync? Checking https://github.com/openshift/cluster-openshift-apiserver-operator/blob/master/pkg/operator/starter.go#L368-L374 noone of the informers is waiting for caches to be fully synced.

I think that we should add clusterVersionInformer to WithInformers here https://github.com/openshift/library-go/blob/75c7d51fc4155264ba71551d22d0a1968b7bf989/pkg/operator/apiserver/controller/apiservice/apiservice_controller.go#L66 because

// WithInformers is used to register event handlers and get the caches synchronized functions.

and maybe we should change GetAPIServicesToMangeFunc to accept clusterVersionInformer configinformersconfigv1.ClusterVersionInformer ?

alternative would be to add WaitForCacheSync to the apiServices function and somehow trigger the controller to call apiServices when clusterVersionInformer changes/get an event.

does it make sense?

I think that we should add clusterVersionInformer to WithInformers here

openshift/library-go#1562

+1 for passing informers as the list of managed api services may have many different implementation which do not need clusterVersionInformer.

ok, thanks.

p0lyn0mial · 2023-08-04T10:54:14Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync.go

 	configMap := resourceread.ReadConfigMapV1OrDie(v311_00_assets.MustAsset("v3.11.0/openshift-apiserver/cm.yaml"))
 	defaultConfig := v311_00_assets.MustAsset("v3.11.0/config/defaultconfig.yaml")
+
+	clusterVersion, err := clusterVersionInformer.Lister().Get("version")


I have the same question here. Should we guarantee that the clusterVersionInformer is synced?

And whether we need to add a reactor to the controller.

That's where informer passed in dc8a55f#diff-0d623dfd885adb20f991bda4c2453aebd732ca6dbb4d1d4be6e79805c3b48de6R243 takes affect. Everytime a cluster version changes, OpenShiftAPIServerWorkload.Sync triggers.

okay, so it looks like we don't need an informer, we need a Lister here :) Please change it to a Lister.

p0lyn0mial · 2023-08-04T11:56:19Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync.go

 	requiredConfigMap, _, err := resourcemerge.MergePrunedConfigMap(
 		&openshiftcontrolplanev1.OpenShiftAPIServerConfig{},
 		configMap,
 		"config.yaml",
 		nil,
 		defaultConfig,
+		bytes,


do you have a unit test to cover merging ?

dc8a55f#diff-25e7c9a2521d15c93e122ab635c1ec19358116650aa275abca5e014b514c69c2R381-R384

p0lyn0mial · 2023-08-04T12:01:09Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync.go

+			PerGroupOptions: []openshiftcontrolplanev1.PerGroupOptions{},
+		},
+	}
+	if knownCaps.Has(configv1.ClusterVersionCapabilityBuild) && !capsEnabled.Has(configv1.ClusterVersionCapabilityBuild) {


Will you have a test that will test the disabling of this APIs?

Already in dc8a55f#diff-25e7c9a2521d15c93e122ab635c1ec19358116650aa275abca5e014b514c69c2R241

I think that we should have an e2e test for it, wdyt?

Updating/adding e2es is the next step once this PR and openshift/cluster-openshift-controller-manager-operator#291 are merged.

where are you planing to add that test ? will it be this repo or in the origin ?

I am planning origin as we need to check for disabled controllers as well. E.g. read the config or parse the controllers logs. On the other hand we first need to update the origin tests to react on missing API so other tests (e.g. running in parallel) for Builds/DCs do not fail. Ultimately, each operator repo might have its own version of the e2e. The e2e (going here or into origin) will need a separate PR.

Ideally if we could add a test here and then also run it from the origin repo. Are you planing to add the test before we ship 4.14 ?

Yes. All tests need to land before 4.14 ships.

OK. I would like to avoid shipping a feature without the tests. Thanks.

p0lyn0mial · 2023-08-04T12:03:09Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync.go

+	}
+
+	if knownCaps.Has(configv1.ClusterVersionCapabilityDeploymentConfig) && !capsEnabled.Has(configv1.ClusterVersionCapabilityDeploymentConfig) {
+		klog.Infof("Capability %q not enabled, disabling 'openshift.io/apps' controller", configv1.ClusterVersionCapabilityDeploymentConfig)


won't this spam logs on every Sync ?

I will increase the log level to 4

ingvagabund · 2023-08-06T09:25:33Z

/retest-required

ingvagabund · 2023-08-06T10:09:27Z

[sig-cli] oc adm must-gather runs successfully for audit logs [apigroup:config.openshift.io][apigroup:oauth.openshift.io] [Suite:openshift/conformance/parallel]

The last two e2e-aws-own failed due to empty openshift-apiserver audit logs

All OA instances are reporting bunch of:

2023-08-03T21:07:10.547646790Z W0803 21:07:10.547610       1 logging.go:59] [core] [Channel #91 SubChannel #92] grpc: addrConn.createTransport failed to connect to {
2023-08-03T21:07:10.547646790Z   "Addr": "10.0.43.149:2379",
2023-08-03T21:07:10.547646790Z   "ServerName": "10.0.43.149",
2023-08-03T21:07:10.547646790Z   "Attributes": null,
2023-08-03T21:07:10.547646790Z   "BalancerAttributes": null,
2023-08-03T21:07:10.547646790Z   "Type": 0,
2023-08-03T21:07:10.547646790Z   "Metadata": null

… capabilities are not enabled Unless a capability is unknown, install APIService object and run apiserver for corresponding API only when enabled.

p0lyn0mial · 2023-08-10T11:23:36Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync_test.go

+			configInformers.Config().V1().ClusterVersions().Informer()
+
+			ctx := context.Background()
+			configInformers.Start(ctx.Done())


and having a lister instead of an informer will simplify this unit tests :)

p0lyn0mial · 2023-08-10T11:43:57Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync.go

+		apiServersConfig.APIServers.PerGroupOptions = append(apiServersConfig.APIServers.PerGroupOptions, openshiftcontrolplanev1.PerGroupOptions{Name: openshiftcontrolplanev1.OpenShiftAppsAPIserver, DisabledVersions: []string{"v1"}})
+	}
+
+	bytes, err := json.Marshal(apiServersConfig)


mhm, should we be passing a yaml ?

p0lyn0mial · 2023-08-10T11:53:45Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync_test.go

+				t.Fatal(err)
+			}
+
+			cm, err := kubeClient.CoreV1().ConfigMaps("openshift-apiserver").Get(ctx, "config", metav1.GetOptions{})


should we check kubeClient.Actions() instead ?

We usually do this to check what resource would be sent to the server and how many times. The Get method reads the state stored in memory, which may not necessarily correspond to the resource that would be sent to the server.

Something like https://github.com/openshift/openshift-apiserver/blob/064c2d0ef0ecaeda2bcc4387eaaa7258cee5adcf/pkg/project/apiserver/registry/project/proxy/proxy_test.go#L96-L101? Is the goal here to add just a test for Matches("update", "configmaps")? Or, to more inspect the corresponding action? Do you have an example?

Usually we verify the verbs (https://github.com/openshift/library-go/blob/abaa55e36ec42bcade37586fbeabfb5bf48cd5b8/pkg/operator/encryption/controllers/key_controller_test.go#L389)

And the content of the resource as well (https://github.com/openshift/library-go/blob/abaa55e36ec42bcade37586fbeabfb5bf48cd5b8/pkg/operator/encryption/controllers/key_controller_test.go#L302)

I think we could check both here as well.

p0lyn0mial · 2023-08-10T11:54:01Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync_test.go

+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+
+			kubeClient := fake.NewSimpleClientset(


please rename it to fakeKubeClient

TestOperatorConfigProgressingCondition uses kubeClient well. Do you wanna rename it on other test(s) as well?

sure, if this is not a problem for you that would be fine :)

p0lyn0mial · 2023-08-10T12:02:05Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync_test.go

+			}
+
+			config := openshiftcontrolplanev1.OpenShiftAPIServerConfig{}
+			if err := json.NewDecoder(bytes.NewBuffer([]byte(cm.Data["config.yaml"]))).Decode(&config); err != nil {


oas uses a versioned decoder. Could we do the same here ?

scheme := runtime.NewScheme() utilruntime.Must(openshiftcontrolplanev1.Install(scheme)) codecs := serializer.NewCodecFactory(scheme) obj, err := runtime.Decode(codecs.UniversalDecoder(openshiftcontrolplanev1.GroupVersion, configv1.GroupVersion), configContent) if err != nil { return err } config := obj.(*openshiftcontrolplanev1.OpenShiftAPIServerConfig)

https://github.com/openshift/openshift-apiserver/blob/df3ca642f426cf2f34abeae152a9fce80b44c63e/pkg/cmd/openshift-apiserver/cmd.go#L129

p0lyn0mial · 2023-08-10T12:17:27Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync_test.go

+		name                    string
+		knownCapabilities       []configv1.ClusterVersionCapability
+		enabledCapabilities     []configv1.ClusterVersionCapability
+		expectedPerGroupOptions []openshiftcontrolplanev1.PerGroupOptions


would it make sense to validate the whole config and not only PerGroupOptions ?
assuming that other fields remain unchanged, this shouldn't require much more effort

p0lyn0mial · 2023-08-11T11:42:26Z

pkg/operator/workload/workload_openshiftapiserver_v311_00_sync.go

+	}
+
+	if knownCaps.Has(configv1.ClusterVersionCapabilityDeploymentConfig) && !capsEnabled.Has(configv1.ClusterVersionCapabilityDeploymentConfig) {
+		klog.V(4).Infof("Capability %q not enabled, disabling 'openshift.io/apps' controller", configv1.ClusterVersionCapabilityDeploymentConfig)


mhm, I still think it will be logged on every sync. What is the value of spamming the log file with this information ? Ideally if we could log it just once, right ?

Maybe we need a new condition ? Does the service controller set a condition ?

With log 4 it is good to know the operator properly reads the capabilities. Otherwise, I'd need to get the CM, decode it, read the list of disabled apiservers and compare it. Too much additional code for just a single log line.

The API service controller sets the new Degraded condition for API service endpoints until the disabled APIService objects are deleted. The test is reported through the already existing conditions for operands.

openshift-ci · 2023-08-11T13:13:51Z

@ingvagabund: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-operator-encryption-rotation	`fe768dd`	link	true	`/test e2e-gcp-operator-encryption-rotation`
ci/prow/e2e-gcp-operator-encryption	`fe768dd`	link	true	`/test e2e-gcp-operator-encryption`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

tkashem · 2023-08-14T13:51:33Z

/lgtm
/approve

openshift-ci · 2023-08-14T14:07:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ingvagabund, tkashem

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tkashem]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 2, 2023

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 2, 2023

openshift-ci bot requested review from benluddy and mfojtik May 2, 2023 14:30

ingvagabund force-pushed the capabilities-build-dc branch 2 times, most recently from d54a619 to e0b42d7 Compare May 21, 2023 11:13

ingvagabund mentioned this pull request May 21, 2023

WRKLDS-728: Make Build and DeploymentConfig API optional through capabilities openshift/api#1462

Merged

This was referenced May 30, 2023

WRKLDS-728: OpenShiftAPIServerConfig: new ApiServers field to specify enabled/disabled API servers openshift/api#1475

Merged

WRKLDS-728: controller/apiservice: allow to disable/delete an API service resources openshift/library-go#1534

Merged

ingvagabund force-pushed the capabilities-build-dc branch from e0b42d7 to fe768dd Compare June 1, 2023 12:36

ingvagabund force-pushed the capabilities-build-dc branch from fe768dd to f5218fd Compare June 20, 2023 10:50

ingvagabund force-pushed the capabilities-build-dc branch from f5218fd to 09d986a Compare July 14, 2023 08:30

ingvagabund mentioned this pull request Jul 17, 2023

WRKLDS-728: Disable apiservers openshift/openshift-apiserver#366

Merged

ingvagabund force-pushed the capabilities-build-dc branch 3 times, most recently from 70ebfd5 to 96e3280 Compare July 19, 2023 12:42

ingvagabund force-pushed the capabilities-build-dc branch 2 times, most recently from c1b2527 to dc8a55f Compare August 3, 2023 20:36

ingvagabund changed the title ~~WIP: WRKLDS-728: Capabilities: drop build/apps APIService when capabilities are not enabled~~ WRKLDS-728: Capabilities: drop build/apps APIService when capabilities are not enabled Aug 4, 2023

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 4, 2023

p0lyn0mial reviewed Aug 4, 2023

View reviewed changes

ingvagabund mentioned this pull request Aug 9, 2023

APIServiceController: allow to pass informers for registering additional event handlers openshift/library-go#1562

Merged

ingvagabund added 2 commits August 9, 2023 14:33

Sync openshift/api

19eb434

Capabilities: drop build/apps APIServices and disable apiservers when…

7113150

… capabilities are not enabled Unless a capability is unknown, install APIService object and run apiserver for corresponding API only when enabled.

ingvagabund force-pushed the capabilities-build-dc branch from dc8a55f to 7113150 Compare August 9, 2023 12:42

p0lyn0mial reviewed Aug 10, 2023

View reviewed changes

ingvagabund force-pushed the capabilities-build-dc branch from 5361ad6 to 2b7de92 Compare August 11, 2023 11:14

p0lyn0mial reviewed Aug 11, 2023

View reviewed changes

ingvagabund force-pushed the capabilities-build-dc branch from 2b7de92 to 6bfe2e1 Compare August 11, 2023 14:00

Review comments

2cc5428

ingvagabund force-pushed the capabilities-build-dc branch from 6bfe2e1 to 2cc5428 Compare August 11, 2023 14:15

openshift-ci bot assigned tkashem Aug 14, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 14, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 14, 2023

openshift-merge-robot merged commit 00f7e4c into openshift:master Aug 14, 2023
7 checks passed

ingvagabund deleted the capabilities-build-dc branch August 14, 2023 16:22

WRKLDS-728: Capabilities: drop build/apps APIService when capabilities are not enabled #532

WRKLDS-728: Capabilities: drop build/apps APIService when capabilities are not enabled #532

Conversation

ingvagabund commented May 2, 2023

openshift-ci-robot commented May 2, 2023 • edited by openshift-ci bot

ingvagabund commented Jul 14, 2023

ingvagabund commented Jul 17, 2023

ingvagabund commented Aug 4, 2023

Choose a reason for hiding this comment

ingvagabund Aug 9, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingvagabund Aug 10, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ingvagabund commented Aug 6, 2023

ingvagabund commented Aug 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Aug 11, 2023 • edited

tkashem commented Aug 14, 2023

openshift-ci bot commented Aug 14, 2023

openshift-ci-robot commented May 2, 2023 •

edited by openshift-ci bot

ingvagabund Aug 9, 2023 •

edited

ingvagabund Aug 10, 2023 •

edited

ingvagabund commented Aug 6, 2023 •

edited

openshift-ci bot commented Aug 11, 2023 •

edited