Enable defaut recommended admission plugins

[drpaneas]

This PR enables the default recommended admission controller
plugins for kubernetes. In addition, it includes the plugins
from kubeadm (NodeRestriction) and lastly, it adds the
PodSecurityPolicy plugin since skuba is using it.

Why is this PR needed?

Fixes https://github.com/SUSE/avant-garde/issues/220
Fixes https://github.com/SUSE/avant-garde/issues/569

Anything else a reviewer needs to know?

Special test cases, manual steps, links to resources or anything else that could be helpful to the reviewer.

Info for QA

As agreed, we will cover only the sonobuoy tests as minimum sufficient coverage for that. Here's my results:

$ sonobuoy version --kubeconfig ~/.kube/config
Sonobuoy Version: v0.15.3
MinimumKubeVersion: 1.13.0
MaximumKubeVersion: 1.15.99
GitSHA: 
API Version:  v1.15.2

$ sonobuoy run --wait
Running plugins: e2e, systemd-logs
INFO[0002] created object                                name=heptio-sonobuoy namespace= resource=namespaces
INFO[0002] created object                                name=sonobuoy-serviceaccount namespace=heptio-sonobuoy resource=serviceaccounts
INFO[0002] created object                                name=sonobuoy-serviceaccount-heptio-sonobuoy namespace= resource=clusterrolebindings
INFO[0003] created object                                name=sonobuoy-serviceaccount namespace= resource=clusterroles
INFO[0003] created object                                name=sonobuoy-config-cm namespace=heptio-sonobuoy resource=configmaps
INFO[0003] created object                                name=sonobuoy-plugins-cm namespace=heptio-sonobuoy resource=configmaps
INFO[0003] created object                                name=sonobuoy namespace=heptio-sonobuoy resource=pods
INFO[0003] created object                                name=sonobuoy-master namespace=heptio-sonobuoy resource=services

$ results=$(sonobuoy retrieve)
$ sonobuoy e2e $results
failed tests: 0

Related info

Status BEFORE applying the patch

• kube-apiserver should be running

• only NodeRestriction and PodSecurityPolicy admission plugins are loaded

Status AFTER applying the patch

kube-apiserver should be running:

$ kubectl get pods -A -o wide | grep kube-apiserver
kube-system       kube-apiserver-master-1            1/1     Running   0          5h39m   10.84.72.49    master-1   <none>           <none>
kube-system       kube-apiserver-master-2            1/1     Running   0          5h33m   10.84.73.39    master-2   <none>           <none>
kube-system       kube-apiserver-master-3            1/1     Running   0          5h27m   10.84.72.77    master-3   <none>           <none>

Make sure it loads the new plugins:

$ pid=$(ssh sles@10.84.72.49 pgrep kube-apiserver)
$ ssh sles@10.84.72.49 ps -fp $pid
$ ssh sles@10.84.72.49 ps -fp $pid | awk -F 'enable-admission-plugins=' '{ print $2 }' | awk -F ' ' '{ print $1 }'

NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,PodSecurityPolicy

Docs

SUSE/doc-caasp#477

Merge restrictions

(Please do not edit this)

We are in v4-maintenance phase, so we will restrict what can be merged to prevent unexpected surprises:

What can be merged (merge criteria):
    2 approvals:
        1 developer: code is fine
        1 QA: QA is fine
    there is a PR for updating documentation (or a statement that this is not needed)

[ereslibre]

Aside from the admission plugins themselves, a small comment regarding forcing our own.

[Klaven]

I agree with Rafa, and this /lgtm

[suseclee]

LGTM