Halyard: 403 (forbidden) on spin-orca-bootstrap:8083/orchestrate

[spinnaker-release]

Migrated from https://github.com/spinnaker/halyard/issues/718

Opened by: @wheleph (2017-10-13 14:24:37) I'm trying to deploy Spinnaker 1.4.1 using Halyard 0.34.0-20170908160431. When I execute hal deploy apply I see the following cryptic error:

! ERROR com.fasterxml.jackson.databind.JsonMappingException:
  Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using
  backslash to be included in string value
at [Source:
  retrofit.ExceptionCatchingTypedInput$ExceptionCatchingInputStream@30f842ca;
  line: 2236, column: 176]
at [Source:
  retrofit.ExceptionCatchingTypedInput$ExceptionCatchingInputStream@30f842ca;
  line: 2236, column: 21] (through reference chain:
  com.netflix.spinnaker.halyard.core.tasks.v1.DaemonTask["children"]->java.util.ArrayList[0]->com.netflix.spinnaker.halyard.core.tasks.v1.DaemonTask["fatalError"])

--debug flag gives the following additional output:

      "url" : "http://localhost:43081/api/v1/proxy/namespaces/spinnaker/services/spin-orca-bootstrap:8083/orchestrate",
      "response" : {
        "url" : "http://localhost:43081/api/v1/proxy/namespaces/spinnaker/services/spin-orca-bootstrap:8083/orchestrate",
        "status" : 403,

I suppose this is because I'm trying to deploy Spinnaker using a restricted service account that has access to only 1 namespace (similar to https://blog.spinnaker.io/spinnaker-kubernetes-rbac-c40f1f73c172) and it lacks some permissions. I couldn't find anything relevant in orca or clouddriver logs.

But how to find which ones are actually needed?

Comments:

---

@lwander (2017-09-29 17:55:07): Does it have access to the spinnaker namespace?

---

@wmuizelaar (2017-09-29 18:01:40): Yes, it has the 'edit' clusterrole in the spinnaker namespace:

$ kubectl get rolebinding -n spinnaker -o yaml spinnaker-role-binding
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"RoleBinding","metadata":{"annotations":{},"name":"spinnaker-role-binding","namespace":"spinnaker"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"edit"},"subjects":[{"kind":"ServiceAccount","name":"spinnaker-service-account"}]}
  creationTimestamp: 2017-09-29T11:54:21Z
  name: spinnaker-role-binding
  namespace: spinnaker
  resourceVersion: "390019"
  selfLink: /apis/rbac.authorization.k8s.io/v1beta1/namespaces/spinnaker/rolebindings/spinnaker-role-binding
  uid: ee524a23-a50c-11e7-9645-42010a8400e2
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
- kind: ServiceAccount
  name: spinnaker-service-account

---

@lwander (2017-09-29 18:03:37): ClusterRoles are not namespace dependent, so I'm not sure what you mean

---

@lwander (2017-09-29 18:03:57): https://kubernetes.io/docs/admin/authorization/rbac/#role-and-clusterrole

---

@wmuizelaar (2017-09-29 18:05:56): Well, as described in your link, the spinnaker service account has a rolebinding in the spinnaker namespace, giving it 'edit' rights (which happens to be defined as a clusterrole, but because this is a rolebinding and not a clusterrolebinding, it only applies within the spinnaker namespace)

---

@wmuizelaar (2017-10-13 14:24:36): We needed to give the spinnaker service-account 'cluster-admin' rights to make things work eventually.

Is it possible to specify what rights actually are needed, so we can restrict it accordingly?

[wheleph]

With Spinnaker 1.6.1 and Halyard 1.2.0 I get more meaningful error message:

User "system:serviceaccount:vso-spin:spinnaker-service-account" cannot create services/proxy in the namespace "vso-spin"

[wheleph]

After investigation of logs it turned out that it's necessary to add the following permissions:

- apiGroups: [""]
  resources: ["services/proxy", "pods/portforward"]
  verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]

So the complete role definition looks like:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: spinnaker-role
rules:
- apiGroups: [""]
  resources: ["namespaces", "configmaps", "events", "replicationcontrollers", "serviceaccounts", "pods/logs"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods", "services", "secrets"]
  verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["list", "get"]
- apiGroups: ["apps"]
  resources: ["controllerrevisions", "statefulsets"]
  verbs: ["list"]
- apiGroups: ["extensions", "apps"]
  resources: ["deployments", "replicasets", "ingresses"]
  verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
# These permissions are necessary for halyard to operate. We use this role also to deploy Spinnaker itself.
- apiGroups: [""]
  resources: ["services/proxy", "pods/portforward"]
  verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]

[wheleph]

@lwander, @wmuizelaar the issue can be closed.

[lwander]

Nice work!