failed calling webhook "mutating-create-update-seldondeployment.seldon.io": error

[RudraprakashR]

Hello,
I have brought up kubeflow 0.7.x with seldon-core-operator 1.0.1 and tried running a seldon's XGBoost
Prepackaged Model Servers example.
The example is:

`apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
name: xgboost
spec:
name: iris
predictors:

• graph:
  children: []
  implementation: XGBOOST_SERVER
  modelUri: pvc://xgboostlocal
  name: classifier
  name: default
  replicas: 1`

I am getting an error mentioned below :

Error from server (InternalError): error when creating "seldon_example.yaml": Internal error occurred: failed calling webhook "mutating-create-update-seldondeployment.seldon.io": Post https://webhook-server-service.kubeflow.svc:443/mutating-create-update-seldondeployment?timeout=30s: service "webhook-server-service" not found

Please help

thanks in advance

[ukclivecox]

How did you install Seldon 1.0.1 as its not in Kubeflow 0.7?
You probably have a stale or missing webhooks. Can you check
kubectl get validatingwebhookconfguration
kubectl get mutatingwebhookconfiguration

[AndriiNeverov]

To me this has happened after installing KF 1.0 into the cluster which previously had KF 0.7.1. Deleting the old webhooks helped:

kubectl delete validatingwebhookconfiguration -A validating-webhook-configuration
kubectl delete mutatingwebhookconfiguration -A mutating-webhook-configuration

Would be good if this could be taken care of automatically as part of Seldon 1+ install.

[RudraprakashR]

Hello @AndriiNeverov and @cliveseldon thank you so much for your responses i deleted old weebhooks and it worked.

[ukclivecox]

Great to hear its we working. I wonder if this is because of the name change between Seldon 0.4.1 which is part of KF 0.7 and Seldon 1.0.x which is part of KF 1.0.
@adriangonz any comments on this?

[adriangonz]

Depending on the upgrade process, it may not be cleaning the old Kubernetes resources coming from the previous version. That's a good point @cliveseldon, if the name was the same I assume that Kubernetes itself would replace them implicitly.

How did you upgrade from KF 0.7 to 1.0 @AndriiNeverov @RudraprakashR? It would be good to learn more about this so that we can add it as part of the upgrade process, or at least document it somewhere.

[AndriiNeverov]

@adriangonz The official upgrade process is described at https://www.kubeflow.org/docs/upgrading/.

I didn't follow it though. Just deployed 1.0 on top of the old cluster which created webhooks with the new names (seldon-validating-webhook-configuration-kubeflow and seldon-mutating-webhook-configuration-kubeflow) and the old ones were still there.

I realize now that it might be a not supported scenario. Especially, if proper upgrade (which I haven't tried) would have taken care of deleting the old ones. However, just preserving the old webhook names would have avoided the problem in the first place.

[ukclivecox]

Post 1.0 we should open an issue on Kubeflow if this happens