Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Istio 1.4 ingress-gateway RBAC not found issue #19371

Closed
whatsupbuddy opened this issue Dec 4, 2019 · 18 comments · Fixed by istio/operator#668
Closed

Istio 1.4 ingress-gateway RBAC not found issue #19371

whatsupbuddy opened this issue Dec 4, 2019 · 18 comments · Fixed by istio/operator#668

Comments

@whatsupbuddy
Copy link

@whatsupbuddy whatsupbuddy commented Dec 4, 2019

Bug description
I have a clean install of Istio 1.4 running on an AKS K8's cluster. HTTP traffic is all good to a pod i am running. However after running through the Cert Manager Setup (Istio documentation may need updating to reflect their changes) I cannot get to my service over HTTPS following the issuing of certificates.

The error that my Istio-ingressgateway pod is logging is:

error k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:98: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:istio-system:istio-ingressgateway-service-account" cannot list resource "secrets" in API group "" in the namespace "istio-system": RBAC: role.rbac.authorization.k8s.io "istio-ingressgateway-sds" not found

Expected behavior
Be able to have https traffic routed to my pods when hitting the public endpoint & leveraging a lets encrypt certificate.

Steps to reproduce the bug
Its constant for all services service i attempt to communicate with over https

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
Istioctl:client version: 1.4.0 - control plane version: 1.4.0 - data plane version: 1.4.0 (2 proxies)
Kubectl: Major:"1", Minor:"13", GitVersion:"v1.13.12"
Helm: 3.0
Cert Manager: v0.12

How was Istio installed?

istioctl manifest apply
--set values.gateways.istio-ingressgateway.sds.enabled=true
--set values.global.k8sIngress.enabled=true
--set values.global.k8sIngress.enableHttps=true
--set values.global.k8sIngress.gatewayName=ingressgateway
--set values.grafana.enabled=true
--set values.grafana.security.enabled=true
--set values.kiali.enabled=true
--set values.tracing.enabled=true \

Then followed this along with cert manager steps for v0.12: https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/#configuring-dns-name-and-gateway
Environment where bug was observed (cloud vendor, OS, etc)

AKS - Azure

@howardjohn

This comment has been minimized.

Copy link
Member

@howardjohn howardjohn commented Dec 4, 2019

Looks like its not creating the role properly. We should see:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: istio-ingressgateway-sds
  namespace: istio-system
  labels:
    release: istio
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
---


apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: istio-ingressgateway-sds
  namespace: istio-system
  labels:
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: istio-ingressgateway-sds
subjects:
- kind: ServiceAccount
  name: istio-ingressgateway-service-account

created based on the command above. Can you check that the role exists?

@whatsupbuddy

This comment has been minimized.

Copy link
Author

@whatsupbuddy whatsupbuddy commented Dec 4, 2019

@howardjohn - i will check now and get back to you.

@whatsupbuddy

This comment has been minimized.

Copy link
Author

@whatsupbuddy whatsupbuddy commented Dec 4, 2019

Ok here it is: kubectl describe rolebinding istio-ingressgateway-sds -n istio-system

Name: istio-ingressgateway-sds
Labels: operator.istio.io/component=IngressGateway
operator.istio.io/managed=Reconcile
operator.istio.io/version=1.4.0
release=istio
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{},"labels":{"operator.istio.io/component":"In...
Role:
Kind: Role
Name: istio-ingressgateway-sds
Subjects:
Kind Name Namespace


ServiceAccount istio-ingressgateway-service-account

Then when running: kubectl get role -n istio-system
No resources found.

@howardjohn

This comment has been minimized.

Copy link
Member

@howardjohn howardjohn commented Dec 4, 2019

When I used the flags you provided above (with generate, not apply) the role was there. So seems like it's either an issue with apply, or you somehow deleted the role

@whatsupbuddy

This comment has been minimized.

Copy link
Author

@whatsupbuddy whatsupbuddy commented Dec 4, 2019

When I used the flags you provided above (with generate, not apply) the role was there. So seems like it's either an issue with apply, or you somehow deleted the role

Ok the istioctl command you mean. Those flags were based off that Istio ingress-certmgr article and then the last four flags were just what I figured I would need to pass through to have them features. Previously I installed istio 1.1 via Helm earlier this year but opted for istio 1.4 via istioctl this time around.

The installation was done just yesterday and dont believe i've removed anything, either way, do you foresee any issues running the below manually with kubectl apply?

Thanks @howardjohn !!

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: istio-ingressgateway-sds
namespace: istio-system
labels:
release: istio
rules:

  • apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "watch", "list"]

@howardjohn

This comment has been minimized.

Copy link
Member

@howardjohn howardjohn commented Dec 4, 2019

it should be fine, you could also try rerunning the istioctl command

@JimmyCYJ

This comment has been minimized.

Copy link
Member

@JimmyCYJ JimmyCYJ commented Dec 4, 2019

@whatsupbuddy is the problem solved now?

@whatsupbuddy

This comment has been minimized.

Copy link
Author

@whatsupbuddy whatsupbuddy commented Dec 4, 2019

@whatsupbuddy is the problem solved now?

Ill let you know in 5 minutes!

@JimmyCYJ

This comment has been minimized.

Copy link
Member

@JimmyCYJ JimmyCYJ commented Dec 4, 2019

Assign to @howardjohn as John is helping with this issue. Feel free to reassign. Thanks!

@whatsupbuddy

This comment has been minimized.

Copy link
Author

@whatsupbuddy whatsupbuddy commented Dec 4, 2019

Ok - that issue is solved - i still have an issue with accessing over https but i think thats a seperate thing. Out of curiousity where is the best spot to seek advice on setting stuff like istio with cert manager up? As an issue on here or somewhere else?

@whatsupbuddy

This comment has been minimized.

Copy link
Author

@whatsupbuddy whatsupbuddy commented Dec 4, 2019

it should be fine, you could also try rerunning the istioctl command

Just re running the istioctl command on a fresh cluster and that role does not get provisioned.

@whatsupbuddy

This comment has been minimized.

Copy link
Author

@whatsupbuddy whatsupbuddy commented Dec 4, 2019

Question - are theyre meant to be two gateways provisioned in istio-system?

  • ingressgateway
  • istio-autogenerated-k8s-ingress
@howardjohn

This comment has been minimized.

Copy link
Member

@howardjohn howardjohn commented Dec 4, 2019

@richardwxn I confirmed the role shows up from generate and is not there from apply

Component IngressGateway installed successfully:
================================================

serviceaccount/istio-ingressgateway-service-account created
deployment.apps/istio-ingressgateway created
gateway.networking.istio.io/ingressgateway created
gateway.networking.istio.io/istio-autogenerated-k8s-ingress created
sidecar.networking.istio.io/default created
poddisruptionbudget.policy/ingressgateway created
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
service/istio-ingressgateway created
@howardjohn howardjohn assigned richardwxn and unassigned howardjohn Dec 4, 2019
@howardjohn howardjohn added this to the 1.4 milestone Dec 4, 2019
@richardwxn

This comment has been minimized.

Copy link
Contributor

@richardwxn richardwxn commented Dec 5, 2019

@howardjohn this should not happen since the logic of generating manifests is shared between manifest-generate and manifest-apply. I would try that out to see what happens

@so-jelly

This comment has been minimized.

Copy link

@so-jelly so-jelly commented Dec 26, 2019

I ran into this myself. In summary, use istioctl manifest generate | kubectl apply apply -f -

@fabianotessarolo

This comment has been minimized.

Copy link

@fabianotessarolo fabianotessarolo commented Dec 29, 2019

Yep me either @so-jelly:

istioctl version && istioctl manifest generate | grep -i "Kind: Role"
client version: 1.4.2
control plane version: 1.4.2
data plane version: 1.4.2 (14 proxies)

@howardjohn if I'm not mistaken, on your output #19371 (comment) I see the Role Binding, but not the Kind Role.

Applying the role manually seems to work, but actually istioctl is not creating the required Role.

@howardjohn

This comment has been minimized.

Copy link
Member

@howardjohn howardjohn commented Dec 29, 2019

Yes, that is what I meant @fabianotessarolo - it is not working as expected. this was fixed in the linked PR which will be in 1.4.3 coming out next week or so

@NelsonJeppesen

This comment has been minimized.

Copy link

@NelsonJeppesen NelsonJeppesen commented Jan 15, 2020

I still see this with 1.4.3

Completely deleted the istio-system namespace and re-deployed istio 1.4.3 from scratch but I still see this issue. Just like in 1.4.2, creating the istio-ingressgateway-sds Role by hand fixed the issue

Using istio-operator if thats relevant

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.