✨ Registration-agent supports multiple bootstrapkubeconfigs. #443

xuezhaojun · 2024-04-25T09:06:07Z

Summary

Klusterlet supports configure multiple bootstrapkubeconfigs and switch when hub doesn't accpet it or failed to connect to a hub.

Special Notes:

The feature is depends on the auto-approve enabled and configured.

qiujian16

can we split this to two PRs. the first one only update the registration part.

...anager/olm-catalog/0.13.1/manifests/operator.open-cluster-management.io_clustermanagers.yaml

pkg/operator/operators/klusterlet/controllers/klusterletcontroller/klusterlet_controller.go

pkg/registration/spoke/bootstrapkubeconfigsmanager.go

xuezhaojun · 2024-04-25T10:03:21Z

can we split this to two PRs. the first one only update the registration part.

Yes, I have split the PR and now it only contains registration part.

codecov · 2024-04-25T15:27:40Z

Codecov Report

Attention: Patch coverage is 34.84848% with 172 lines in your changes are missing coverage. Please review.

Project coverage is 62.09%. Comparing base (bcbe4d2) to head (983b8da).
Report is 49 commits behind head on main.

Files	Patch %	Lines
...pkubeconfigsmanager/bootstrapkubeconfigsmanager.go	38.53%	54 Missing and 13 partials ⚠️
pkg/registration/spoke/spokeagent.go	0.00%	37 Missing ⚠️
...istration/bootstrapkubeconfig_secret_controller.go	0.00%	18 Missing ⚠️
...ation/spoke/registration/hub_timeout_controller.go	40.74%	13 Missing and 3 partials ⚠️
...ration/spoke/registration/hub_accept_controller.go	26.31%	12 Missing and 2 partials ⚠️
...bootstrapkubeconfigsmanager/bootstrapkubeconfig.go	69.69%	5 Missing and 5 partials ⚠️
...trapkubeconfigsmanager/bootstrapkubeconfiginuse.go	62.50%	3 Missing and 3 partials ⚠️
pkg/registration/spoke/options.go	20.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #443      +/-   ##
==========================================
+ Coverage   61.54%   62.09%   +0.54%     
==========================================
  Files         133      142       +9     
  Lines       14078    11823    -2255     
==========================================
- Hits         8665     7342    -1323     
+ Misses       4664     3687     -977     
- Partials      749      794      +45

Flag	Coverage Δ
unit	`62.09% <34.84%> (+0.54%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

qiujian16 · 2024-04-29T02:45:40Z

pkg/registration/spoke/spokeagent.go

@@ -411,6 +424,16 @@ func (o *SpokeAgentConfig) RunSpokeAgentWithSpokeInformers(ctx context.Context,
 		)
 	}

+	hubAcceptController := registration.NewHubAcceptController(
+		o.agentOptions.SpokeClusterName,


the agent will be unauthorized when accept is false. So this controller does not always work.

Yes, the feature depends on the auto-approve enabled and configured, I will note this point in the doc in the future.

pkg/registration/spoke/lease/lease_controller.go

qiujian16 · 2024-04-29T02:55:26Z

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfiginuse.go

+)
+
+// bootstrapKubeConfigInUse is the registration spoke's current in used bootstrap kubeconfig.
+type bootstrapKubeConfigInUse interface {


why interface, we always use the bootstrap kubeconfig from file system, isn't it?

No, the current bootstrap kubeconfig used to trigger rebootstrap is based on the secret, and if in the future we want to change to another type such as file, then the interface is a good practice to prevent that change impact other parts of code.

qiujian16 · 2024-04-29T02:58:17Z

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfigsmanager.go

+
+// ReSelect is the only public method of a bootstrapkubeconfigManager
+func (m *bootstrapkubeconfigsManager) ReSelect(ctx context.Context) error {
+	if m.reselected {


when reslect will be set false again?

Never in the same lifecycle, the pod will restart and in the next lifecycle of container, the reselect will be set as false initially.

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfig.go

xuezhaojun · 2024-04-29T06:12:53Z

pkg/registration/spoke/registration/hub_accept_controller.go

+	if cluster.Spec.HubAcceptsClient {
+		return nil
+	}


If hubClientAccpet set to false, is there a controller to remove the clusterrole and prevent agent to get ManagedCluster?

The related code: https://github.com/xuezhaojun/OCM/blob/fc574c1be4fcfe1dfb0e250b41ef9cd3b9c1705c/pkg/registration/hub/managedcluster/controller.go#L132

Comment added: the hubAccpetController can only handle the case that "true" -> "false".

PS: from this line of code we can see that an event is send to all watchers at the same time. So if the informer on the hub side and informer on the agent side will both get the update event.

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfigsmanager.go

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfiginuse.go

xuezhaojun · 2024-04-29T06:42:32Z

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfiginuse.go

+
+type bootstrapKubeConfigInUseImpl struct {
+	secretName      string
+	secretNamespace string


remove secretName and use from constant pkg.

@qiujian16

From the code we know, the spoke compoennt namespace is not always the defaultSpokeComponentNamespace, it depends on which ns the agent is installed, which means it's neccessary to keep the secretNamespace.

// NewAgentOptions returns the flags with default value set func NewAgentOptions() *AgentOptions { opts := &AgentOptions{ HubKubeconfigDir: "/spoke/hub-kubeconfig", ComponentNamespace: defaultSpokeComponentNamespace, CommoOpts: NewOptions(), } // get component namespace of spoke agent nsBytes, err := os.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/namespace") if err == nil { opts.ComponentNamespace = string(nsBytes) } return opts }

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfigsmanager.go

openshift-ci · 2024-05-04T08:12:52Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xuezhaojun
Once this PR has been reviewed and has the lgtm label, please assign skeeey for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/registration/OWNERS
test/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…hub. Signed-off-by: xuezhaojun <zxue@redhat.com>

openshift-ci bot requested review from elgnay and zhiweiyin318 April 25, 2024 09:06

qiujian16 reviewed Apr 25, 2024

View reviewed changes

xuezhaojun force-pushed the feature-switch-hub branch 2 times, most recently from 7115d11 to 51b4c3b Compare April 25, 2024 10:02

xuezhaojun force-pushed the feature-switch-hub branch 3 times, most recently from 649f468 to 55d1b74 Compare April 25, 2024 10:16

openshift-ci bot added the do-not-merge/hold label Apr 25, 2024

xuezhaojun changed the title ~~✨ Klusterlet support multiple bootstrapkubeconfigs.~~ ✨ Klusterlet support multiple bootstrapkubeconfigs. [WIP] Apr 25, 2024

xuezhaojun changed the title ~~✨ Klusterlet support multiple bootstrapkubeconfigs. [WIP]~~ ✨ Klusterlet support multiple bootstrapkubeconfigs. Apr 28, 2024

xuezhaojun force-pushed the feature-switch-hub branch from 4645a2a to 3b52b6e Compare April 28, 2024 14:50

openshift-merge-robot added the needs-rebase label Apr 28, 2024

xuezhaojun force-pushed the feature-switch-hub branch 2 times, most recently from eeeee2a to 73f58c6 Compare April 28, 2024 15:20

openshift-merge-robot removed the needs-rebase label Apr 28, 2024

openshift-ci bot removed the do-not-merge/hold label Apr 28, 2024

xuezhaojun requested a review from qiujian16 April 28, 2024 15:21

xuezhaojun force-pushed the feature-switch-hub branch 3 times, most recently from 2baf7de to fa7196e Compare April 29, 2024 05:59

qiujian16 reviewed Apr 29, 2024

View reviewed changes

xuezhaojun commented Apr 29, 2024

View reviewed changes

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfigsmanager.go Show resolved Hide resolved

xuezhaojun commented Apr 29, 2024

View reviewed changes

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfiginuse.go Show resolved Hide resolved

xuezhaojun commented Apr 29, 2024

View reviewed changes

pkg/registration/spoke/bootstrapkubeconfigsmanager/bootstrapkubeconfigsmanager.go Outdated Show resolved Hide resolved

xuezhaojun force-pushed the feature-switch-hub branch from fa7196e to fc574c1 Compare May 4, 2024 08:12

xuezhaojun force-pushed the feature-switch-hub branch 4 times, most recently from 9719311 to 8ccb0b3 Compare May 12, 2024 12:39

xuezhaojun requested a review from qiujian16 May 12, 2024 12:46

xuezhaojun changed the title ~~✨ Klusterlet support multiple bootstrapkubeconfigs.~~ ✨ Klusterlet agent support multiple bootstrapkubeconfigs. May 12, 2024

xuezhaojun changed the title ~~✨ Klusterlet agent support multiple bootstrapkubeconfigs.~~ ✨ Registration-agent supports multiple bootstrapkubeconfigs. May 12, 2024

xuezhaojun force-pushed the feature-switch-hub branch from 8ccb0b3 to affc8c9 Compare May 16, 2024 09:21

Registration-agent supports multiple bootstrapkubeconfigs and switch-…

983b8da

…hub. Signed-off-by: xuezhaojun <zxue@redhat.com>

xuezhaojun force-pushed the feature-switch-hub branch from affc8c9 to 983b8da Compare May 16, 2024 12:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Registration-agent supports multiple bootstrapkubeconfigs. #443

✨ Registration-agent supports multiple bootstrapkubeconfigs. #443

xuezhaojun commented Apr 25, 2024 •

edited

qiujian16 left a comment

xuezhaojun commented Apr 25, 2024

codecov bot commented Apr 25, 2024 •

edited

qiujian16 Apr 29, 2024

xuezhaojun May 4, 2024

qiujian16 Apr 29, 2024

xuezhaojun Apr 29, 2024

xuezhaojun May 4, 2024

qiujian16 Apr 29, 2024

xuezhaojun May 4, 2024

xuezhaojun Apr 29, 2024

xuezhaojun May 4, 2024

xuezhaojun May 4, 2024

xuezhaojun May 12, 2024

xuezhaojun Apr 29, 2024

xuezhaojun May 4, 2024

openshift-ci bot commented May 4, 2024

✨ Registration-agent supports multiple bootstrapkubeconfigs. #443

Are you sure you want to change the base?

✨ Registration-agent supports multiple bootstrapkubeconfigs. #443

Conversation

xuezhaojun commented Apr 25, 2024 • edited

Summary

qiujian16 left a comment

Choose a reason for hiding this comment

xuezhaojun commented Apr 25, 2024

codecov bot commented Apr 25, 2024 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented May 4, 2024

xuezhaojun commented Apr 25, 2024 •

edited

codecov bot commented Apr 25, 2024 •

edited