New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RBAC/v1 apis in 1.8.4 take a long time to be available, rbac/v1beta1 apis are immediately available #56899

Closed
puneetguptanitj opened this Issue Dec 6, 2017 · 5 comments

Comments

Projects
None yet
5 participants
@puneetguptanitj

puneetguptanitj commented Dec 6, 2017

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

While upgrading a cluster from 1.7.7 to 1.8.4 we are seeing an issue where rbac/v1 apis are taking a long time to be available while rbac/v1beta1 apis are immediately available.

Following shows portion of api server logs where repeated attempts to create a clusterrolebinding kept hitting v1beta1 API and failing (as the yaml for clusterrolebinding is using v1) only after around 6 mins was the v1 api available

I1206 02:18:26.159788 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (849.554µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57596]
I1206 02:18:36.443903 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (882.707µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57626]
I1206 02:18:47.681015 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (739.119µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57646]
I1206 02:18:57.950529 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (868.266µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57674]
I1206 02:19:08.229472 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (819.409µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57714]
I1206 02:19:18.509022 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (855.587µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57742]
I1206 02:19:28.788274 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (852.857µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57770]
I1206 02:19:39.061476 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (863.969µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57804]
I1206 02:19:49.334077 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (831.841µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57820]
I1206 02:19:59.612576 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (794.052µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57848]
I1206 02:20:09.891490 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (873.745µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57872]
I1206 02:20:20.167194 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (833.252µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57888]
I1206 02:20:30.559923 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (726.542µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57916]
I1206 02:20:40.841377 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (778.105µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57946]
I1206 02:20:51.130115 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (809.75µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57974]
I1206 02:21:01.561022 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (919.476µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:57990]
I1206 02:21:11.850537 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (899.759µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58026]
I1206 02:21:22.140556 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (876.984µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58042]
I1206 02:21:32.433602 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (836.9µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58070]
I1206 02:21:42.723118 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (843.08µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58088]
I1206 02:21:53.005680 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (818.855µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58116]
I1206 02:22:03.407608 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (790.238µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58132]
I1206 02:22:13.697983 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (2.185418ms) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58174]
I1206 02:22:23.983548 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (885.403µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58194]
I1206 02:22:34.268977 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (856.222µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58222]
I1206 02:22:44.552567 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (912.803µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58240]
I1206 02:22:54.840928 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (826.622µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58272]
I1206 02:23:05.136700 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (902.401µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58290]
I1206 02:23:15.419510 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (813.022µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58308]
I1206 02:23:25.699921 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (2.79864ms) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58336]
I1206 02:23:35.981540 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (827.741µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58366]
I1206 02:23:46.269816 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (846.222µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58396]
I1206 02:23:56.552978 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (805.646µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58414]
I1206 02:24:06.837554 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (914.559µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58442]
I1206 02:24:17.116817 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access: (838.708µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58464]
I1206 02:24:27.543595 1 wrap.go:42] GET /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/usera-access: (676.322µs) 404 [[kubectl/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:58492]

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
Upgrade a 1.7.7 cluster to 1.8.4.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): 1.8.4
  • Cloud provider or hardware configuration: AWS, t2 medium instances for both master and worker
  • OS (e.g. from /etc/os-release):seen on both unbutu and centos (even slower in centos)
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

/sig auth

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Dec 13, 2017

Member

is this a HA API server environment?

Member

liggitt commented Dec 13, 2017

is this a HA API server environment?

@puneetguptanitj

This comment has been minimized.

Show comment
Hide comment
@puneetguptanitj

puneetguptanitj Dec 13, 2017

Thanks Jordan for taking a look, its a single API server setup. Please let me know if I can grab more logs, it is fairly reproducible in our setup

puneetguptanitj commented Dec 13, 2017

Thanks Jordan for taking a look, its a single API server setup. Please let me know if I can grab more logs, it is fairly reproducible in our setup

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Jan 5, 2018

Member

while the server is in this state, what do the following return:

kubectl get apiservice/v1.rbac.authorization.k8s.io -o yaml
kubectl get apiservice/v1beta1.rbac.authorization.k8s.io -o yaml
kubectl api-versions
kubectl get clusterroles.v1.rbac.authorization.k8s.io
Member

liggitt commented Jan 5, 2018

while the server is in this state, what do the following return:

kubectl get apiservice/v1.rbac.authorization.k8s.io -o yaml
kubectl get apiservice/v1beta1.rbac.authorization.k8s.io -o yaml
kubectl api-versions
kubectl get clusterroles.v1.rbac.authorization.k8s.io
@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Apr 5, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot commented Apr 5, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Apr 17, 2018

Member

cannot reproduce, closing for lack of information

/close

Member

liggitt commented Apr 17, 2018

cannot reproduce, closing for lack of information

/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment