Having install issue with Linkerd 2.0 on OpenShift

[mhausenblas]

To reproduce my install issue, please use this Katacoda scenario. It seems like the pod security contexts need to be aligned with OpenShift's security context constraints. Certain caps are not set or allowed. I could monkey patch as I did with stuff in previous steps but I think a dedicated version might be more sustainable?

CC: @grampelberg

[kinash000]

For fix errors:
Error creating: pods "controller-d6f584b7d-" is forbidden: unable to validate against any security context constraint: [capabilities.add: Invalid value: "NET_ADMIN": capability may not be added capabilities.add: Invalid value: "NET_ADMIN": capability can not be added spec.containers [4] .securityContext.securityContext.runAsUser: Invalid value: 2102: must be in the ranges: [1000080000, 1000089999]]

Need execute:
oc adm policy add-scc-to-user privileged -z linkerd-controller -n linkerd
oc adm policy add-scc-to-user privileged -z linkerd-prometheus -n linkerd

[grampelberg]

Awesome, I'll open an issue to get that added to the docs!

[lostiniceland]

Having the same problem. Executing the provided commands works for controller and prometheus, but the grafana and web pods are not starting with the same error.

After some attempts I found that the following works
oc adm policy add-scc-to-user privileged -z default -n linkerd

I think this is related to the init-container trying to modify iptables. Should that be reverted later on?

[grampelberg]

@lostiniceland there's probably a more specific policy that we can use. I went down the podsecuritypolicy hole last week. If there was a SCC YAML we could add to the docs, that would be ideal ... I just don't have an openshift cluster around to play with right now.

[mhausenblas]

@grampelberg https://learn.openshift.com/playgrounds/ :P

[mshutt]

Here is what I am working with... It's restricted + NET_ADMIN... and I just change the uid from 2102 to something in the valid range for the namespace in the installation YAMLs both as the securityContext and the arguments for the proxies.

We are going to test more tomorrow to see if this is an approach that will work.

THAT SAID, I am still quite reticent to give out cap NET_ADMIN to any individual user who is sharing a cluster with other projects, other business units, and other applications... It's clearly not "all about the multitenancy"

allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: security.openshift.io/v1
defaultAddCapabilities: ['NET_ADMIN']
fsGroup:
  type: MustRunAs
groups:
- system:authenticated
kind: SecurityContextConstraints
metadata:
  annotations:
  name: linkerd2
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities:
- KILL
- MKNOD
- SETUID
- SETGID
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users: []
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret

[calston]

After some attempts I found that the following works
oc adm policy add-scc-to-user privileged -z default -n linkerd

Before anyone else sees this as a solution, it's the kube equivalent of chmod -R 0777 /

[lostiniceland]

After some attempts I found that the following works
oc adm policy add-scc-to-user privileged -z default -n linkerd

Before anyone else sees this as a solution, it's the kube equivalent of chmod -R 0777 /

Fair enough. Thats why I did not investigate Linkerd on Openshift at the moment. I hope we can get a clean setup (maybe an operator for Linkerd).

[calston]

Fair enough. Thats why I did not investigate Linkerd on Openshift at the moment. I hope we can get a clean setup (maybe an operator for Linkerd).

NB. It's not a criticism of the idea itself to see if it fixes the permission to then distil it again. I'm just the resident [insert evil regime] about privileged mode because it's the first thing every user I support asks me for as soon as they hit a permission issue and it's 99.99999999999999999999999999999999% of the time totally unnecessary :)

eg. the docker PHP container still runs as root 🤦‍♂️

[kinash000]

@calston @lostiniceland @mshutt @mhausenblas
Gays, Do you have a successful installation experience linkerd in OpenShift?
After installation, the collection of metrics does not work:

linkerd stat pods --all-namespaces
NAMESPACE NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 TLS
default docker-registry-1-xk7mz 0/1 - - - - - -
default router-1-d9f9k 0/1 - - - - - -
emojivoto emoji-8466d478db-8gsnf 1/1 - - - - - -
emojivoto vote-bot-77f8478f6b-g6f5j 1/1 - - - - - -
emojivoto voting-858656b5c8-gzv6l 1/1 - - - - - -
emojivoto web-68656ddd64-b7g2j 1/1 - - - - - -
kube-system kubernetes-dashboard-6c664cf6c5-knnlh 0/1 - - - - - -
linkerd controller-d6f584b7d-gntjg 1/1 - - - - - -
linkerd grafana-d57f845dd-5k8l4 1/1 - - - - - -
linkerd prometheus-6b979dc67d-mqrss 1/1 - - - - - -
linkerd web-5bc657776c-bpkwc 1/1 - - - - - -
openshift-web-console webconsole-5f567f6dc5-vjqbg 0/1 - - - - - -

But i try installed Linkerd in Katakoda's OpenShift and all works.

[jotak]

Katakoda scenario is now broken due to curl commands changed in the install script.
But locally I could make it work with openshift 3.11 and adding privileged mode. Metrics correctly being collected after waiting a little bit. However the links to dashboard / grafana are incorrect.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[jkassis]

what is the accepted solution here? is it

> oc adm policy add-scc-to-user privileged -z linkerd-controller -n linkerd                                    08.07 10:43
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "linkerd-controller"
> oc adm policy add-scc-to-user privileged -z linkerd-prometheus -n linkerd                                        08.07 10:43
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "linkerd-prometheus"
> oc adm policy add-scc-to-user privileged -z default -n linkerd                                                   08.07 10:43
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "default"

?

[jkassis]

i found errors related to scc for all of the service accounts and did the install this way...

> brew install linkerd
> oc login
> oc new-project linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-controller -n linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-prometheus -n linkerd
> oc adm policy add-scc-to-user privileged -z default -n linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-destination -n linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-grafana -n linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-proxy-injector -n linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-sp-validator -n linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-tap -n linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-web -n linkerd
> oc adm policy add-scc-to-user privileged -z linkerd-identity -n linkerd
> oc describe rolebinding.rbac -n linkerd
> linkerd install | oc apply -f -
> linkerd check

pods still refused to show up with no logs at all, so afaict... linkerd is complete incompatible with openshift 4.5...

[ihcsim]

@jkassis Thanks for bringing this up. Let's follow-up the conversation at #4851.