seldon-core-operator CRD's incompatible with K8s v1.18

[iamlittle]

Installing the seldon-core-operator on a fresh v1.18 Kubernetes install throws an error:

Error: CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[containerSpec].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property]

I believe this is due to an API change in v1.18.

CustomResourceDefinition schemas that use x-kubernetes-list-map-keys to specify properties that uniquely identify list items must make those properties required or have a default value, to ensure those properties are present for all list items. See https://kubernetes.io/docs/reference/using-api/api-concepts/#merge-strategy for details. (#88076, @eloyekunle) [SIG API Machinery and Testing]

I was able to work around by installing a crd definition exported from a successful v1.17 install and installing seldon-core-operator with crd.create=false.

Steps to reproduce:

curl https://get.helm.sh/helm-v3.1.2-linux-amd64.tar.gz -o helm-v3.1.2-linux-amd64.tar.gz
tar -xzvf helm-v3.1.2-linux-amd64.tar.gz
chmod a+x linux-amd64/helm
mv linux-amd64/helm /usr/local/bin/

helm repo add ambassador https://www.getambassador.io
helm repo add seldon https://storage.googleapis.com/seldon-charts

helm upgrade ambassador ambassador/ambassador --install
helm upgrade seldon-core seldon/seldon-core-operator --install --version=1.0.3-SNAPSHOT --set ambassador.enabled=true --set crd.create=true

[ukclivecox]

Thanks for these details. We will need to see what we need to add to the Kustomize definitions.

[DenisOgr]

Same problem! Downgrade Kubernetes to 1.17.0 and it fixes.

[xaniasd]

FYI this affects local development as well, because kind images are not pinned to a specific version

[vega42]

I can confirm. After downgrading my minikube using:

minikube start --kubernetes-version v1.17.0

I can get past the issue in the seldon-core setup.

[Atharex]

The same error came up with the prometheus operator.
prometheus-operator/prometheus-operator#3122

I can confirm that when I adapted the schema to make the property required, it fixed the problem and the install works again on 1.18+ (though not sure if that is a proper "fix" for your operator)

In the prometheus operator they fixed the problem by upgrading their CRD API to v1 for kubernetes >= 1.18 and keeping also the v1beta1 API for kubernetes < 1.16

[ukclivecox]

Thanks for the info we will be looking to solve this post 1.2.0 release for 1.3.0 release

[shawnzhu]

I hit this problem when installing kubeflow (w/ seldon-core-operator:1.1.0) on k8s v1.18. is this caused by kubernetes/kubernetes#91395 ?

[ukclivecox]

Yes that is the core reason. We need to upgrade to v1 CRDs but need at same time to keep backwards compataibility for older k8s. So I think we will need options to create both.

[shadowmodder]

+1 need this backward compatible

[ukclivecox]

It should be. The helm chart checks for k8s version and deploys a v1beta1 CRD for k8s <1.18 and a v1 CRD for >- 1.18.
There is #2367 which we need to investigate before release.

[arunbenoyv]

kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Helm install is resulting in the same error

helm install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set usageMetrics.enabled=true --set ambassador.enabled=true
Error: CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[containerSpec].properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[componentSpecs].items.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property]

[ukclivecox]

The fix is in master and will be in next release @arunbenoyv

[daddydrac]

Separate but related ques here: When we set the --set crd.create=true to true, what does that CRD do and what is it for? I could not find anything in the docs that explains this.

[ukclivecox]

This is used for cases where we want the operator to create the CRD - its usually used just in Marketplaces where we can't install the CRD separately first

[daddydrac]

So on a vanilla kubernetes install, it can be set to false and model serving with mlflow and metrics in grafana will work?

…

On Thu, Feb 17, 2022 at 10:00 AM cliveseldon ***@***.***> wrote: This is used for cases where we want the operator to create the CRD - its usually used just in Marketplaces where we can't install the CRD separately first — Reply to this email directly, view it on GitHub <#1675 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHVQHEWDCNKBZGRSJBSAA3U3ULQ7ANCNFSM4MEF3WDA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[ukclivecox]

Worth following our install methods to see if they fit your use. With Helm the chart will install the CRD.