Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: not creating RBAC roles in NodePort mode #524

Closed
julianvmodesto opened this issue Sep 22, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@julianvmodesto
Copy link
Contributor

commented Sep 22, 2017

I'm creating an ingress of type ingress.appscode.com/type: NodePort, and it looks like the ServiceAccount + RBAC roles aren't getting created.

voyager-operator-3067620550-3qvbb voyager-operator Caused By:
voyager-operator-3067620550-3qvbb voyager-operator configmaps "voyager-frontend" not found
voyager-operator-3067620550-3qvbb voyager-operator Stack trace:
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/pkg/ingress/nodeport.go:408                           (*nodePortController).Delete
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/pkg/operator/ingress_tprs.go:189                      (*Operator).DeleteEngress
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/pkg/operator/ingress_tprs.go:89                       (*Operator).initIngressTPRWatcher.func5
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:206 ResourceEventHandlerFuncs.OnDelete
voyager-operator-3067620550-3qvbb voyager-operator <autogenerated>:56                                                                        (*ResourceEventHandlerFuncs).OnDelete
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:276 NewInformer.func1
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 (*DeltaFIFO).Pop
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:147 (*controller).processLoop
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:121 processLoop)-fm
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96   JitterUntil.func1
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97   JitterUntil
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52   Until
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:121 (*controller).Run
voyager-operator-3067620550-3qvbb voyager-operator /usr/local/go/src/runtime/asm_amd64.s:2197                                                goexit
voyager-operator-3067620550-3qvbb voyager-operator I0922 15:40:26.992192       1 nodeport.go:414]
voyager-operator-3067620550-3qvbb voyager-operator Caused By:
voyager-operator-3067620550-3qvbb voyager-operator rolebindings.rbac.authorization.k8s.io "voyager-frontend" not found
voyager-operator-3067620550-3qvbb voyager-operator Stack trace:
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/pkg/ingress/nodeport.go:413                           (*nodePortController).Delete
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/pkg/operator/ingress_tprs.go:189                      (*Operator).DeleteEngress
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/pkg/operator/ingress_tprs.go:89                       (*Operator).initIngressTPRWatcher.func5
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:206 ResourceEventHandlerFuncs.OnDelete
voyager-operator-3067620550-3qvbb voyager-operator <autogenerated>:56                                                                        (*ResourceEventHandlerFuncs).OnDelete
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:276 NewInformer.func1
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 (*DeltaFIFO).Pop
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:147 (*controller).processLoop
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:121 processLoop)-fm
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96   JitterUntil.func1
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97   JitterUntil
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52   Until
voyager-operator-3067620550-3qvbb voyager-operator /go/src/github.com/appscode/voyager/vendor/k8s.io/client-go/tools/cache/controller.go:121 (*controller).Run
voyager-operator-3067620550-3qvbb voyager-operator /usr/local/go/src/runtime/asm_amd64.s:2197                                                goexit

$ kubectl get sa,clusterrole,configmap voyager-frontend
Error from server (NotFound): serviceaccounts "voyager-frontend" not found
Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "voyager-frontend" not found
Error from server (NotFound): configmaps "voyager-frontend" not found

@julianvmodesto julianvmodesto changed the title Bug: Bug: not creating RBAC roles in NodePort mode Sep 22, 2017

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Sep 23, 2017

What is the Kubernetes version? What is the Voyager version?

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Sep 25, 2017

We have talked internally about what might have caused this issue. The stack traces above point to deleting RBAC portion of the code.

I think this is what happened. You had a cluster which was not using RBAC. Then you switched to RBAC and also upgraded Voyager. Due to #448 , the rbac stuff was not created. Then you tried to delete the Ingress. But the RBAC objects failed to delete, since they did not exist in the first place. "Failed-to-delete" is ok. We just log it so that there is some record that this happened.

Also, earlier we will create RBAC roles only when an Ingress was added. In #542, we are going to create or update RBAC roles if needed. So, this should be fixed.

@julianvmodesto

This comment has been minimized.

Copy link
Contributor Author

commented Sep 25, 2017

Thanks Tamal!

For reference, here is the version info.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:22:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl --namespace=kube-system exec voyager-operator-3067620550-3qvbb /voyager version
Version = 3.2.1
VersionStrategy = tag
Os = alpine
Arch = amd64
CommitHash = be28c1ad0423d8e4c1bbc1f72188307ec711c09c
GitBranch = release-3.2
GitTag = 3.2.1
CommitTimestamp = 2017-09-19T11:47:32

Here are the Pod templates for the old + new versions we deployed:

$ kubectl --namespace=kube-system rollout history deploy voyager-operator --revision 6
deployments "voyager-operator" with revision #6
Pod Template:
  Labels:       app=voyager-operator
        pod-template-hash=3294440648
  Service Account:      voyager-operator
  Containers:
   voyager-operator:
    Image:      appscode/voyager:3.2.0
    Port:       56790/TCP
    Args:
      run
      --cloud-provider=aws
      --v=6
      --ingress-class=voyager
      --rbac=true
      --custom-templates=/srv/voyager/custom/*.cfg
    Environment:        <none>
    Mounts:
      /etc/kubernetes from cloudconfig (ro)
      /srv/voyager/custom from templates (ro)
  Volumes:
   cloudconfig:
    Type:       HostPath (bare host directory volume)
    Path:       /etc/kubernetes
   templates:
    Type:       ConfigMap (a volume populated by a ConfigMap)
    Name:       voyager-templates
    Optional:   false

$ kubectl --namespace=kube-system rollout history deploy voyager-operator --revision 7
deployments "voyager-operator" with revision #7
Pod Template:
  Labels:       app=voyager-operator
        pod-template-hash=2824153801
  Service Account:      voyager-operator
  Containers:
   voyager-operator:
    Image:      appscode/voyager:3.2.1
    Port:       56790/TCP
    Args:
      run
      --cloud-provider=aws
      --v=3
      --ingress-class=voyager
      --rbac
      --custom-templates=/srv/voyager/custom/*.cfg
    Environment:        <none>
    Mounts:
      /etc/kubernetes from cloudconfig (ro)
      /srv/voyager/custom from templates (ro)
  Volumes:
   cloudconfig:
    Type:       HostPath (bare host directory volume)
    Path:       /etc/kubernetes
   templates:
    Type:       ConfigMap (a volume populated by a ConfigMap)
    Name:       voyager-templates
    Optional:   false

@tamalsaha tamalsaha added this to the 4.0.0 milestone Oct 4, 2017

@tamalsaha tamalsaha closed this Oct 4, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.