webhook-create-signed-cert.sh fails on OpenShift 3.9.0

[mattmi88]

Is this a BUG or FEATURE REQUEST?: BUG

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?: YES

Bug:
Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version 0.7.1
kubectl version 1.9.6
kubernetes server version 1.9.1
oc version 3.9.0
minishift version 1.15.1

Is Istio Auth enabled or not ? NO
Did you install istio.yaml, istio-auth.yaml.... istio.yaml

What happened:

webhook-create-signed-cert.sh fails when on OpenShift 3.9.0.

What you expected to happen:

webhook-create-signed-cert.sh should succeed or provide alternate instructions to create the cert.

How to reproduce it:

1. Install Minishift VM. My environment is VirtualBox on Windows, but should be similar on linux. Note: The docker bip change is to avoid network collision on our corporate network.

   pushd ~
   minishift config set openshift-version v3.9.0
   minishift config set cpus 4
   minishift config set memory 8192
   minishift config set disk-size 60g
   minishift config set vm-driver virtualbox
   minishift config set docker-opt bip=172.29.0.1/16
   
   minishift start
   popd
   

2. Config kubelet args. Increase pods-per-core.

   minishift_node_kubelet_args="'{\"kubeletArguments\": {\"pods-per-core\": [\"20\"], \"volume-plugin-dir\": [\"/var/run/kubelet/volumeplugins\"]}}'"
   eval minishift openshift config set --target node --patch ${minishift_node_kubelet_args}
   
   sleep 5
   

3. Install istio.

   oc login -u system:admin
   
   oc adm policy add-cluster-role-to-user cluster-admin admin
   
   oc adm policy add-scc-to-user anyuid -z istio-ingress-service-account -n istio-system
   oc adm policy add-scc-to-user anyuid -z default -n istio-system
   
   kubectl apply -f istio.yaml
   
   ~~~ Output ~~~
   namespace "istio-system" created
   clusterrole "istio-pilot-istio-system" created
   clusterrole "istio-sidecar-injector-istio-system" created
   clusterrole "istio-mixer-istio-system" created
   clusterrole "istio-mixer-validator-istio-system" created
   clusterrole "istio-ca-istio-system" created
   clusterrole "istio-sidecar-istio-system" created
   clusterrolebinding "istio-pilot-admin-role-binding-istio-system" created
   clusterrolebinding "istio-sidecar-injector-admin-role-binding-istio-system" created
   clusterrolebinding "istio-ca-role-binding-istio-system" created
   clusterrolebinding "istio-ingress-admin-role-binding-istio-system" created
   clusterrolebinding "istio-sidecar-role-binding-istio-system" created
   clusterrolebinding "istio-mixer-admin-role-binding-istio-system" created
   clusterrolebinding "istio-mixer-validator-admin-role-binding-istio-system" created
   configmap "istio-mixer" created
   service "istio-mixer" created
   serviceaccount "istio-mixer-service-account" created
   deployment "istio-mixer" created
   customresourcedefinition "rules.config.istio.io" created
   customresourcedefinition "attributemanifests.config.istio.io" created
   customresourcedefinition "circonuses.config.istio.io" created
   customresourcedefinition "deniers.config.istio.io" created
   customresourcedefinition "fluentds.config.istio.io" created
   customresourcedefinition "kubernetesenvs.config.istio.io" created
   customresourcedefinition "listcheckers.config.istio.io" created
   customresourcedefinition "memquotas.config.istio.io" created
   customresourcedefinition "noops.config.istio.io" created
   customresourcedefinition "opas.config.istio.io" created
   customresourcedefinition "prometheuses.config.istio.io" created
   customresourcedefinition "rbacs.config.istio.io" created
   customresourcedefinition "servicecontrols.config.istio.io" created
   customresourcedefinition "solarwindses.config.istio.io" created
   customresourcedefinition "stackdrivers.config.istio.io" created
   customresourcedefinition "statsds.config.istio.io" created
   customresourcedefinition "stdios.config.istio.io" created
   customresourcedefinition "apikeys.config.istio.io" created
   customresourcedefinition "authorizations.config.istio.io" created
   customresourcedefinition "checknothings.config.istio.io" created
   customresourcedefinition "kuberneteses.config.istio.io" created
   customresourcedefinition "listentries.config.istio.io" created
   customresourcedefinition "logentries.config.istio.io" created
   customresourcedefinition "metrics.config.istio.io" created
   customresourcedefinition "quotas.config.istio.io" created
   customresourcedefinition "reportnothings.config.istio.io" created
   customresourcedefinition "servicecontrolreports.config.istio.io" created
   customresourcedefinition "tracespans.config.istio.io" created
   customresourcedefinition "serviceroles.config.istio.io" created
   customresourcedefinition "servicerolebindings.config.istio.io" created
   configmap "istio" created
   customresourcedefinition "destinationpolicies.config.istio.io" created
   customresourcedefinition "egressrules.config.istio.io" created
   customresourcedefinition "routerules.config.istio.io" created
   customresourcedefinition "virtualservices.networking.istio.io" created
   customresourcedefinition "destinationrules.networking.istio.io" created
   customresourcedefinition "externalservices.networking.istio.io" created
   service "istio-pilot" created
   serviceaccount "istio-pilot-service-account" created
   deployment "istio-pilot" created
   service "istio-ingress" created
   serviceaccount "istio-ingress-service-account" created
   deployment "istio-ingress" created
   serviceaccount "istio-ca-service-account" created
   deployment "istio-ca" created
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=attributemanifest
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=attributemanifest
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=stdio
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=logentry
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=rule
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=metric
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=metric
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=metric
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=metric
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=metric
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=metric
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=prometheus
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=rule
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=rule
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=kubernetesenv
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=rule
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=rule
   unable to recognize "istio.yaml": no matches for config.istio.io/, Kind=kubernetes
   ~~~ End Output ~~~
   
   ... Run it a second time to confirm no errors of: 'unable to recognize ...'
   

4. Wait for istio services to start.

5. Minor update to webhook-create-signed-cert.sh so openssl will accept the subject on Git-bash for Windows. Add the following just before the first invocation of openssl.

   # Windows mingw needs an extra leading slash in the subject name.
   subject="/CN=${service}.${namespace}.svc"
   if [[ $(uname -s) == MINGW* ]]; then subject="/${subject}"; fi
   

6. Run webhook-create-signed-cert.sh.

   ./webhook-create-signed-cert.sh
   
   ~~~ Output ~~~
   
   creating certs in tmpdir /tmp/tmp.nmOQyZvHWL
   Generating RSA private key, 2048 bit long modulus
   .........................................................................+++
   ....+++
   e is 65537 (0x10001)
   certificatesigningrequest "istio-sidecar-injector.istio-system" created
   NAME                                  AGE       REQUESTOR      CONDITION
   istio-sidecar-injector.istio-system   0s        system:admin   Pending
   certificatesigningrequest "istio-sidecar-injector.istio-system" approved
   ERROR: After approving csr istio-sidecar-injector.istio-system, the signed certificate did not appear on the resource. Giving up after 10 attempts.
   See https://istio.io/docs/setup/kubernetes/sidecar-injection.html for more details on troubleshooting.
   
   ~~~ End Output ~~~
   

Feature Request:
N

Describe the feature:

[linsun]

Could you try reproduce this on minikube pls?

[mattmi88]

Unfortunately, no, I can't. Our target platform is OpenShift so I really need to make this work on OpenShift. I suspect it will work on Minikube since it has a working controller that will generate the cert.

I'm not sure this is a regression. It's likely something to do with how OpenShift implements their controllers and the CertificateSigningRequest object isn't processed somehow because Redhat would rather you use oc adm ca instead.

[linsun]

cc @ayj for web hook script @christian-posta for openshift.

[ayj]

@yusuoh just updated cert provisioning to use Istio CA instead of k8s CSR API. He removed webhook-create-signed-cert.sh and the caBundle patching script in the process. I don't believe he's updated the istio.io documentation to reflect this yet.

[john-deng]

we faced the same issue.

[john-deng]

@christian-posta @linsun any plan to solve this issue?

[christian-posta]

@geoand is this issue addressed by istio/istio#4389

[louiscryan]

@geoand bump can this be closed?

[geoand]

Hi, I would guess that it can