poll-period option does not respect the service account's namespace

[jayakarra]

provided poll-period and -----guess-kube-api-settings. The API calls to kubernetes does not consider the namespace of the service account thats configured.

When using the --once option, it works fine. However when we use periodic option it fails.
Run ONCE option:
/tmp/helm $ kube-template --guess-kube-api-settings --config=/tmp/kube-template.yaml --alsologtostderr --logtos
tderr=true --v=6 --once
I0531 02:40:37.390509 446 cfg.go:100] using config file: /tmp/kube-template.yaml
I0531 02:40:37.390542 446 cfg.go:187] poll period set to 1s
I0531 02:40:37.390557 446 cfg.go:189] command timeout set to 2s
I0531 02:40:37.390573 446 cfg.go:240] adding template from config file: /tmp/haproxy_cfg.sh.tmpl
I0531 02:40:37.392091 446 app.go:105] run once templates processing...
I0531 02:40:37.392110 446 app.go:118] processing template: haproxy_cfg.sh.tmpl
I0531 02:40:37.392143 446 client_gen.go:33] fetching pods, namespace: "jkompella-temp6-3486", selector: "appFamily=aoobm"
I0531 02:40:37.408308 446 round_trippers.go:405] GET **https://10.100.0.1:443/api/v1/namespaces/jkompella-temp6-3486/pods?**labelSelector=appFamily%3Daoobm 200 OK in 16 milliseconds
I0531 02:40:37.428623 446 client_gen.go:33] fetching pods, namespace: "jkompella-temp6-3486", selector: "app=aoobm-sshd"

PERIODIC POLL:
/tmp/helm $ kube-template --guess-kube-api-settings --config=/tmp/kube-template.yaml --alsologtostderr --logtos
tderr=true --v=6
I0531 02:40:28.574389 435 cfg.go:100] using config file: /tmp/kube-template.yaml
I0531 02:40:28.574419 435 cfg.go:187] poll period set to 1s
I0531 02:40:28.574431 435 cfg.go:189] command timeout set to 2s
I0531 02:40:28.574447 435 cfg.go:240] adding template from config file: /tmp/haproxy_cfg.sh.tmpl
I0531 02:40:28.576073 435 app.go:80] starting templates processing...
I0531 02:40:28.576097 435 app.go:118] processing template: haproxy_cfg.sh.tmpl
I0531 02:40:28.576154 435 client_gen.go:33] fetching pods, namespace: "jkompella-temp6-3486", selector: "appFamily=aoobm"
I0531 02:40:28.576418 435 reflector.go:202] Starting reflector *v1.Pod (0s) from kube-template/client_gen.go:50
I0531 02:40:28.576435 435 reflector.go:240] Listing and watching *v1.Pod from kube-template/client_gen.go:50
I0531 02:40:28.583677 435 round_trippers.go:405] GET https://10.100.0.1:443/api/v1/pods?limit=500&resourceVersion=0 403 Forbidden in 7 milliseconds
*E0531 02:40:28.583984 435 reflector.go:205] kube-template/client_gen.go:50: Failed to list v1.Pod: pods is forbidden: User "system:serviceaccount:jkompella-temp6-3486:lurker-service-account" cannot list resource "pods" in API group "" at the cluster scope

Looks like the client_gen.go, informerFactory is the cause of this. Could you please help us out?
podLister, found := c.listers["podLister"]
if !found {
podInformer := c.informerFactory.Core().V1().Pods()

[jayakarra]

kubernetes/kubernetes#54660

Looks like this relates to the above issue.Let me know if you need any further information.

[3cky]

@jayakarra thank you for the report and the issue investigation! I'll look into it.

Also, isn't this issue related to #9? If so, could you please close #9 as a duplicate and we'll continue the work here.

[jayakarra]

Thank you for helping us out. I tried to switch to --kube-config=/tmp/sa.kubeconfig instead of --guess-kube-api-settings and still see same problem. If need be and using POD_NAMESPACE to pass to informerfactory, I can change my code to use --kube-config. Let me know. Thanks again.

[3cky]

@jayakarra I did some fixes in the kube-template namespace handling code, please test.

[jayakarra]

Thank you. I don't see permission denied error any longer but see the following.
*go@v0.18.3/tools/cache/reflector.go:125: Failed to watch v1.Pod: unknown (get pods)
I0603 06:55:45.971028 8 app.go:118] processing template: haproxy_cfg.sh.tmpl

I do notice a delay in the backend of haproxy timing out. Will investigate further and let you know.

[3cky]

Could you check is your RBAC role descriptor declares 'watch' verb for pods?Something like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: mycustomnamespace
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

[jayakarra]

Seems better, will watch for sometime and close the issue. Thanks a lot.

[3cky]

Hi @jayakarra, how it's going? Is the fix worked as intended?

[jayakarra]

Thanks so much. It works and we have deployed in production.