service "istio-galley" not found

[rnkhouse]

Bug description
I am installing istio with below configuration:

---
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default
  components:
    galley:
      enabled: true
    egressGateways:
    - enabled: true
    ingressGateways:
    - enabled: true
      k8s:
        hpaSpec:
          maxReplicas: 5
          metrics:
          - resource:
              name: cpu
              targetAverageUtilization: 80
            type: Resource
          minReplicas: 1
          scaleTargetRef:
            apiVersion: apps/v1
            kind: Deployment
            name: istio-ingressgateway
        resources:
          limits:
            cpu: 1000m
            memory: 256Mi
          requests:
            cpu: 200m
            memory: 256Mi
        strategy:
          rollingUpdate:
            maxSurge: 100%
            maxUnavailable: 25%
        service:
          loadBalancerIP: x.x.x.x
          externalTrafficPolicy: Local
          ports:
          - port: 443
            targetPort: 443
            name: https
          - port: 80
            targetPort: 80
            name: http
        serviceAnnotations:
          service.beta.kubernetes.io/azure-load-balancer-resource-group: rg-name
  meshConfig:
    enableAutoMtls: true
  addonComponents:
    grafana:
      enabled: true
    kiali:
      enabled: true
  values:
    global:
      logging:
        level: "default:info"
      controlPlaneSecurityEnabled: true
    gateways:
      istio-ingressgateway:
        sds:
          enabled: true

Error:

    ✔ Istio core installed
    2020-05-22T14:11:35.511996Z	error	installer	failed to create "EnvoyFilter/istio-system/metadata-exchange-1.4": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.586916Z	error	installer	failed to create "EnvoyFilter/istio-system/metadata-exchange-1.5": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.626408Z	error	installer	failed to create "EnvoyFilter/istio-system/metadata-exchange-1.6": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.664555Z	error	installer	failed to create "EnvoyFilter/istio-system/stats-filter-1.4": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.712217Z	error	installer	failed to create "EnvoyFilter/istio-system/stats-filter-1.5": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.750036Z	error	installer	failed to create "EnvoyFilter/istio-system/stats-filter-1.6": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.833979Z	error	installer	failed to create "EnvoyFilter/istio-system/tcp-metadata-exchange-1.5": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.881527Z	error	installer	failed to create "EnvoyFilter/istio-system/tcp-metadata-exchange-1.6": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.926121Z	error	installer	failed to create "EnvoyFilter/istio-system/tcp-stats-filter-1.5": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    2020-05-22T14:11:35.982427Z	error	installer	failed to create "EnvoyFilter/istio-system/tcp-stats-filter-1.6": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    ✘ Istiod encountered an error: failed to create "EnvoyFilter/istio-system/metadata-exchange-1.4": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/metadata-exchange-1.5": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/metadata-exchange-1.6": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/stats-filter-1.4": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/stats-filter-1.5": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/stats-filter-1.6": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/tcp-metadata-exchange-1.5": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/tcp-metadata-exchange-1.6": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/tcp-stats-filter-1.5": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    failed to create "EnvoyFilter/istio-system/tcp-stats-filter-1.6": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    ✔ Egress gateways installed
    ✔ Ingress gateways installed
    - Processing resources for Addons.                                                                     2020-05-22T14:11:36.750506Z	error	installer	failed to create "PeerAuthentication/istio-system/grafana-ports-mtls-disabled": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    ✘ Addons encountered an error: failed to create "PeerAuthentication/istio-system/grafana-ports-mtls-disabled": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
    - Pruning removed resources                                                                            Error: failed to apply manifests: errors occurred during operation

Steps to reproduce the bug
I had istio 1.4.3. I deleted first and then Install istio 1.6.0 using this command istioctl install -f ./config.yaml with given config file.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
istio version:
client version: 1.6.0
control plane version: 1.6.0
data plane version: 1.6.0 (4 proxies)
kubectl version: 1.15.7
helm version:
Client: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}

How was Istio installed?
istioctl install -f ./config.yaml

Environment where bug was observed (cloud vendor, OS, etc)
AKS

[howardjohn]

When you uninstalled 1.4.3 you likely did not fully clean up so the Validating webhook was still around. You'll need to properly delete that

[bharath0208]

@howardjohn I am getting the same error.

Error from server (InternalError): error when creating "https://github.com/knative/net-istio/releases/download/v0.15.0/release.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found
Error from server (InternalError): error when creating "https://github.com/knative/net-istio/releases/download/v0.15.0/release.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: service "istio-galley" not found

how do i remove the Validating webhook for istio. Also is there a proper way to clean up istio and crds from Kubernetes cluster.

[howardjohn]

kubectl delete validatingwebhookconfigurations istio-galley should do it

[bharath0208]

@howardjohn thank you I can now see that the installation is happening