Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

join 1.24版本的k8s的集群会失败 #1961

Closed
zgfh opened this issue Jun 5, 2022 · 11 comments · Fixed by #1972
Closed

join 1.24版本的k8s的集群会失败 #1961

zgfh opened this issue Jun 5, 2022 · 11 comments · Fixed by #1972
Assignees
@zgfh
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@zgfh
Copy link
Contributor

zgfh commented Jun 5, 2022

What happened:
测试了下join 1.24版本的k8s的集群，发现会失败

日志

(base) [root@10-29-22-116 karmada]# kubectl karmada --kubeconfig=/etc/karmada/karmada-apiserver.config join member1 --cluster-kubeconfig=/root/.kube/kind-k8s-1-24 --cluster-context=kind-kind-k8s-1-24 -v 6
I0605 15:40:05.902449  374049 join.go:148] joining cluster. cluster name: member1
I0605 15:40:05.902512  374049 join.go:149] joining cluster. cluster namespace: karmada-cluster
(base) [root@10-29-22-116 karmada]# kubectl karmada --kubeconfig=/etc/karmada/karmada-apiserver.config join member1 --cluster-kubeconfig=/root/.kube/kind-k8s-1-24 --cluster-context=kind-kind-k8s-1-24 -v 6
I0605 16:21:50.706944  430078 join.go:148] joining cluster. cluster name: member1
I0605 16:21:50.707218  430078 join.go:149] joining cluster. cluster namespace: karmada-cluster
I0605 16:21:50.708664  430078 loader.go:372] Config loaded from file:  /etc/karmada/karmada-apiserver.config
I0605 16:21:50.709382  430078 loader.go:372] Config loaded from file:  /root/.kube/kind-k8s-1-24
I0605 16:21:50.710384  430078 join.go:172] joining cluster config. endpoint: https://kind.debug.daozzg.com:57443
I0605 16:21:50.726892  430078 round_trippers.go:553] GET https://172.18.0.7:32443/api/v1/namespaces/karmada-cluster 200 OK in 16 milliseconds
I0605 16:21:50.748153  430078 round_trippers.go:553] GET https://kind.debug.daozzg.com:57443/api/v1/namespaces/karmada-cluster 200 OK in 19 milliseconds
I0605 16:21:50.750648  430078 round_trippers.go:553] GET https://kind.debug.daozzg.com:57443/api/v1/namespaces/karmada-cluster/serviceaccounts/karmada-member1 200 OK in 1 milliseconds
I0605 16:21:50.758322  430078 round_trippers.go:553] GET https://kind.debug.daozzg.com:57443/api/v1/namespaces/karmada-cluster/serviceaccounts/karmada-impersonator 200 OK in 6 milliseconds
I0605 16:21:50.760966  430078 round_trippers.go:553] GET https://kind.debug.daozzg.com:57443/apis/rbac.authorization.k8s.io/v1/clusterroles/karmada-controller-manager:karmada-member1 200 OK in 2 milliseconds
I0605 16:21:50.761387  430078 join.go:367] ensure ClusterRole succeed as already exist. ClusterRole: karmada-controller-manager:karmada-member1
I0605 16:21:50.763847  430078 round_trippers.go:553] GET https://kind.debug.daozzg.com:57443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/karmada-controller-manager:karmada-member1 200 OK in 2 milliseconds
I0605 16:21:50.764051  430078 join.go:391] ensure ClusterRole succeed as already exist. ClusterRole: karmada-controller-manager:karmada-member1
I0605 16:21:51.767682  430078 round_trippers.go:553] GET https://kind.debug.daozzg.com:57443/api/v1/namespaces/karmada-cluster/serviceaccounts/karmada-member1 200 OK in 2 milliseconds
Error: failed to get serviceAccount secret from cluster(member1), error: failed to get serviceAccount secret, error: no specific secret found in namespace: karmada-cluster
(base) [root@10-29-22-116 karmada]#

What you expected to happen:
join 成功

How to reproduce it (as minimally and precisely as possible):

kubectl karmada --kubeconfig=/etc/karmada/karmada-apiserver.config join member1 --cluster-kubeconfig=/root/.kube/kind-k8s-1-24 --cluster-context=kind-kind-k8s-1-24 -v 6

Anything else we need to know?:

排查发现是因为新版本的k8s 不会自动为sa创建secret 导致的
image

参考这里 :
https://kubernetes.io/docs/concepts/configuration/secret/#service-account-token-secrets
image

在https://github.com/karmada-io/karmada/blob/master/pkg/util/serviceaccount.go#L76 增加当sa.secrets为空时，创建secret
再次测试，发现join 成功

是否可以这样修复？

Environment: kind

  • Karmada version: v1.2.0
  • kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version): v1.2.0
  • Others:
@zgfh zgfh added the kind/bug Categorizes issue or PR as related to a bug. label Jun 5, 2022
@calvin0327
Copy link
Contributor

如果 1.24 的版本，secret 为空，需要手动创建吗？

@zgfh
Copy link
Contributor Author

zgfh commented Jun 5, 2022

https://kubernetes.io/docs/concepts/configuration/secret/#service-account-token-secrets

参考这里，有几种方式，这里的场景个人感觉创建 secret 比较合适（需要永久有效的token）

所以想的方法是：如果k8s 没有自动为sa创建secret（sa.secrets为空），则手动创建一个secret

@XiShanYongYe-Chang
Copy link
Member

这里的场景个人感觉创建 secret 比较合适（需要永久有效的token）

我个人赞同这一点。使用kubectl create token SERVICE_ACCOUNT_NAME获取token不是很方便：

./kubectl create token cztest01 -n cz
eyJhbGciOiJSUzI1NiIsImtpZCI6ImJvWldRbEtqUzJBMHNkczF2RHJ6SkQ3Rmptb0lZcEhJYS1nNFhKUFplSzAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjU0NTc0NzE1LCJpYXQiOjE2NTQ1NzExMTUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJjeiIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJjenRlc3QwMSIsInVpZCI6IjdiMmQ1M2IwLTE3OWMtNGQ3MC04YmUyLTNhNjc1OWRiYjJjNCJ9fSwibmJmIjoxNjU0NTcxMTE1LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Y3o6Y3p0ZXN0MDEifQ.d3GspiRtbbE4qsFLPNH74zhKRHz3eskESQzbRcKAZqc8qKG7P-3BkEsrVru9sghJUjXWgNwDD2Pdysh2S_MiCv7je9nc5XvpiZNlgUt5X9YPxh2UYmLHC9-xgyYu85xlzl4AIV9dvcSIkgGX-I5WTlpjFWNgsEPR9Q9uo5587hR_QOa5eaBF5np5d5KmtaslPvw6UmJJKHkfi62XTC1Mk68nqya8b_0Me5wNcWO7OE9djGVLAyAR8gKovyx9d988c_o_lJQg1fyQOHJ3nbiVrdeHirZPx3JvKtr3HH7rA8oahbxBG16JSVY7TIEd6s7Qb7jHcDy5Wpz1UFVHe8jwzw

如果k8s 没有自动为sa创建secret（sa.secrets为空），则手动创建一个secret

手动创建secret的过程可以说明下么？另外通过手动创建出的secret，如何通过sa找到该secret呢？

@zgfh zgfh mentioned this issue Jun 8, 2022
Merged
@zgfh
Copy link
Contributor Author

zgfh commented Jun 8, 2022

这里的场景个人感觉创建 secret 比较合适（需要永久有效的token）

我个人赞同这一点。使用kubectl create token SERVICE_ACCOUNT_NAME获取token不是很方便：

./kubectl create token cztest01 -n cz
eyJhbGciOiJSUzI1NiIsImtpZCI6ImJvWldRbEtqUzJBMHNkczF2RHJ6SkQ3Rmptb0lZcEhJYS1nNFhKUFplSzAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjU0NTc0NzE1LCJpYXQiOjE2NTQ1NzExMTUsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJjeiIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJjenRlc3QwMSIsInVpZCI6IjdiMmQ1M2IwLTE3OWMtNGQ3MC04YmUyLTNhNjc1OWRiYjJjNCJ9fSwibmJmIjoxNjU0NTcxMTE1LCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6Y3o6Y3p0ZXN0MDEifQ.d3GspiRtbbE4qsFLPNH74zhKRHz3eskESQzbRcKAZqc8qKG7P-3BkEsrVru9sghJUjXWgNwDD2Pdysh2S_MiCv7je9nc5XvpiZNlgUt5X9YPxh2UYmLHC9-xgyYu85xlzl4AIV9dvcSIkgGX-I5WTlpjFWNgsEPR9Q9uo5587hR_QOa5eaBF5np5d5KmtaslPvw6UmJJKHkfi62XTC1Mk68nqya8b_0Me5wNcWO7OE9djGVLAyAR8gKovyx9d988c_o_lJQg1fyQOHJ3nbiVrdeHirZPx3JvKtr3HH7rA8oahbxBG16JSVY7TIEd6s7Qb7jHcDy5Wpz1UFVHe8jwzw

如果k8s 没有自动为sa创建secret（sa.secrets为空），则手动创建一个secret

手动创建secret的过程可以说明下么？另外通过手动创建出的secret，如何通过sa找到该secret呢？

提交pr了：#1972

@RainbowMango
Copy link
Member

Echo from Kubernetes v1.24 Release Note

The LegacyServiceAccountTokenNoAutoGeneration feature gate is beta, and enabled by default. When enabled, Secret API objects containing service account tokens are no longer auto-generated for every ServiceAccount. Use the TokenRequest API to acquire service account tokens, or if a non-expiring token is required, create a Secret API object for the token controller to populate with a service account token by following this guide. (kubernetes/kubernetes#108309, @zshihang)

@XiShanYongYe-Chang
Copy link
Member

XiShanYongYe-Chang commented Jun 10, 2022 •
edited

让我们先来整理下整个事件的背景。

Kubernetes社区为了提高token使用的安全性和可扩展性，提出了一个KEP-1205，旨在引入一种新的机制来使用ServiceAccount token，而不是直接将ServiceAccount生成的Secret挂载到pod中，具体方式见ServiceAccount automation。这个特性名为BoundServiceAccountTokenVolume，在Kubernetes 1.22版本中已经GA。

随着BoundServiceAccountTokenVolume特性的GA，社区认为已经没必要为ServiceAccount自动生成token了，因为这样并不安全，于是又提出了KEP-2799，这个KEP的一个目的是不再为ServiceAccount自动生成token Secret，另外一个目的是要清除未被使用的ServiceAccount产生的token Secret。

对于第一个目的，社区提供了LegacyServiceAccountTokenNoAutoGeneration feature gate，该feature gate在Kubernetes 1.24版本中已进入Beta阶段，这也正是当前Issue中集群加入失败的直接原因。当然了，如果用户仍想使用之前的方式，为ServiceAccount生成Secret，可以参考此处进行操作。

那么，我们现有就有两种方式获取token，一种是手动为ServiceAccount创建Secret；另一种是访问TokenRequest API。使用第一种方式需要由我们承担安全风险，使用第二种方式我们需要解决token过期的问题。又或者，我们可以直接disable LegacyServiceAccountTokenNoAutoGeneration，但这并不是一个好的解决办法。

以上分析为后续方案的决策做准备。

@RainbowMango
Copy link
Member

cc @lonelyCZ @prodanlabs who might be interested in it.

@calvin0327
Copy link
Contributor

https://kubernetes.io/docs/concepts/configuration/secret/#service-account-token-secrets

如果要解决 token 过期的问题，使用第二种方式调用 token API 好像是行不通的。官方提供了一个这样的方式来创建不过期的 token：

  1. 先创建 serviceaccount
  2. 再创建 secret
  3. controller 会自动帮我们填充 token 到 data 里面

既然官方也提供出了这样的一个方式，我感觉没啥安全风险？个人看法😄

@RainbowMango
Copy link
Member

controller 会自动帮我们填充 token 到 data 里面

这种方式后面会不会被弃用？ 如果不会，我赞成这个方案。

@calvin0327
Copy link
Contributor

kubernetes/website#33675

这个 PR 可以作为参考，在 1.24 更新的了对 secret 生成的描述，可以推测不会被弃用。

@XiShanYongYe-Chang
Copy link
Member

controller 会自动帮我们填充 token 到 data 里面

+1

@zgfh 对问题的修改 #1972 正是使用了这种方式，让我们来检视这个PR吧。

@ada2468 ada2468 mentioned this issue Aug 31, 2022
Open
@XiShanYongYe-Chang XiShanYongYe-Chang mentioned this issue Mar 15, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zgfh zgfh

Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
Karmada Release 1.3
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

bugfix: support join 1.24 k8s: auto create secret when sa not create zgfh/karmada
4 participants
@RainbowMango @zgfh @calvin0327 @XiShanYongYe-Chang