feat: CA bundle mount options for storage initializer

[Jooho]

What this PR does / why we need it:
This PR is a collaboration between @Jooho and @danilopeixoto based on the work done by @danilopeixoto in PR #2906 and some additional fixes contributed by @Jooho

It addresses #2897

Related issues listed in #3171

This PR is for self signed certificate for storage-initializer and this feature provides 2 ways

• Using global configuration (inferenceservice-config ConfigMap)
• Using storage-config Secret in a user namespace

If you set global configuration, the cabundle secret in a namespace where kserve controller is running will be copied to a user namespace in where ISVC object created.
However, if you want to set the cabundle secret for a specific infereceService, you can set an annotation 'serving.kserve.io/s3-cabundle-secret'

You can find the detail information from here

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Type of changes
Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Notice
The test model runtime becomes CrashLoopBackOff after running status but it is ok. The most important thing from this test is whether of not to pull a model from SSL Minio pod. So please ignore when the run pod failed at the end.

Test cluster:

• kind v0.20.0
• Kubernetes 1.27.3

  Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean",   BuildDate:"2023-06-15T00:36:28Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
  

Test environment setup:

• Deploy minio with SSL

# Create demo folder
export DEMO_HOME=/tmp/demo
mkdir $DEMO_HOME
cd $DEMO_HOME

# Deploy kubernetes
kind create cluster
kubectl cluster-info --context kind-kind

# Deploy Minio with SSL
git clone git@github.com:Jooho/jhouse_openshift.git
cd jhouse_openshift/Minio/minio-tls-kserve/
source env.sh
./1.setup.sh 
./2.generate-cert.sh 
kubectl create ns minio
kubectl config set-context --current --namespace minio

sed 's+quay.io/opendatahub/modelmesh-minio-examples:caikit-flan-t5+kserve/modelmesh-minio-examples+g'  -i ${DEMO_HOME}/minio.yaml
./3.deploy-minio.sh 
cp /tmp/minio/minio_certs/root.crt /tmp/minio/minio_certs/cabundle.crt

• Deploy KServe

cd /tmp/demo 

git clone git@github.com:Jooho/kserve.git
cd kserve

# Create the cabundle secret in kserve namespace
kubectl create ns kserve
kubectl config set-context --current --namespace kserve
kubectl create configmap cabundle --from-file=/tmp/minio/minio_certs/cabundle.crt -n kserve

# Deploy Cert Manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

# Deploy KServer (RawDeployment version) - it should work with serverless version too
# Update controller/storage-initializer images built based on this PR
# Set caBundleSecretName in infereneservice-config cm

KSERVE_VERSION=0.11.1
wget https://github.com/kserve/kserve/releases/download/v${KSERVE_VERSION}/kserve.yaml
sed 's/Serverless/RawDeployment/g' -i ./kserve.yaml
sed 's+kserve/storage-initializer:v0.11.1+quay.io/jooholee/kserve-storage-initializer:cabundle-configmap+g' -i ./kserve.yaml
sed 's+kserve/kserve-controller:v0.11.1+quay.io/jooholee/kserve-controller:cabundle-configmap+g' -i ./kserve.yaml
kubectl apply -f ./kserve.yaml

TEST

• 1. namespace scope test
• 1-1. using json style storage-config

kubectl create ns kserve-demo
kubectl config set-context --current --namespace kserve-demo 
kustomize build ./config/runtimes/| sed 's/ClusterServingRuntime/ServingRuntime/g' |kubectl create -n kserve-demo -f -

kubectl create configmap local-cabundle --from-file=/tmp/minio/minio_certs/cabundle.crt -n kserve-demo

cat <<EOF|kubectl apply -f -
apiVersion: v1
stringData:
  localMinIO: |
    {
      "type": "s3",
      "access_key_id": "THEACCESSKEY",
      "secret_access_key": "THEPASSWORD",
      "endpoint_url": "https://minio.minio.svc:9000",
      "bucket": "modelmesh-example-models",
      "region": "us-south",
      "cabundle_configmap": "local-cabundle"
    }
kind: Secret
metadata:
  name: storage-config
type: Opaque   
EOF

cat<<EOF | kubectl create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    "serving.kserve.io/deploymentMode": "RawDeployment"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storage:
        key: localMinIO
        path: sklearn/mnist-svm.joblib
EOF

• 1-2. using annotation style storage-config

# Clean up previous test
kubectl delete isvc,secret --all --force

cat <<EOF|kubectl apply -f -
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: VEhFQUNDRVNTS0VZ
  AWS_SECRET_ACCESS_KEY: VEhFUEFTU1dPUkQ=
kind: Secret
metadata:
  annotations:
    serving.kserve.io/s3-endpoint: minio.minio.svc:9000
    serving.kserve.io/s3-region: us-east-2
    serving.kserve.io/s3-useanoncredential: "false"
    serving.kserve.io/s3-usehttps: "1"
    serving.kserve.io/s3-cabundle-configmap: "local-cabundle"
  name: storage-config
  namespace: kserve-demo
type: Opaque
EOF

cat<<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
- name: storage-config
EOF


cat<<EOF | kubectl create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    "serving.kserve.io/deploymentMode": "RawDeployment"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    serviceAccountName: sa
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storageUri: s3://modelmesh-example-models/sklearn/mnist-svm.joblib
EOF

• 2. global scope test
• 2-1. using inferenceservice-config configmap

kubectl delete secret,isvc --all --force
kubectl edit configmap inferenceservice-config -n kserve
  storageInitializer: |-
    {
        ...
        "caBundleConfigMapName": "cabundle", #<==
        "enableDirectPvcVolumeMount": false
    }
kubectl delete pod -l control-plane=kserve-controller-manager --force -n kserve

kubectl config set-context --current --namespace kserve-demo 

cat <<EOF|kubectl apply -f -
apiVersion: v1
stringData:
  localMinIO: |
    {
      "type": "s3",
      "access_key_id": "THEACCESSKEY",
      "secret_access_key": "THEPASSWORD",
      "endpoint_url": "https://minio.minio.svc:9000",
      "bucket": "modelmesh-example-models",
      "region": "us-south"
    }
kind: Secret
metadata:
  name: storage-config
type: Opaque   
EOF

cat<<EOF | kubectl create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    "serving.kserve.io/deploymentMode": "RawDeployment"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storage:
        key: localMinIO
        path: sklearn/mnist-svm.joblib
EOF

• 2-2. override the global cabundle by cabundle_configmap in StorageSpec

kubectl delete secret,isvc --all --force

cat <<EOF|kubectl apply -f -
apiVersion: v1
stringData:
  localMinIO: |
    {
      "type": "s3",
      "access_key_id": "THEACCESSKEY",
      "secret_access_key": "THEPASSWORD",
      "endpoint_url": "https://minio.minio.svc:9000",
      "bucket": "modelmesh-example-models",
      "region": "us-south",
      "cabundle_configmap": "local-cabundle"    #<==
    }
kind: Secret
metadata:
  name: storage-config
type: Opaque   
EOF

cat<<EOF | kubectl create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    "serving.kserve.io/deploymentMode": "RawDeployment"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storage:
        key: localMinIO
        path: sklearn/mnist-svm.joblib
EOF

• 2-3. override the global cabundle by annotation serving.kserve.io/s3-cabundle-configmap in storage-config

# Clean up previous test
kubectl delete isvc,secret --all --force

cat <<EOF|kubectl apply -f -
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: VEhFQUNDRVNTS0VZ
  AWS_SECRET_ACCESS_KEY: VEhFUEFTU1dPUkQ=
kind: Secret
metadata:
  annotations:
    serving.kserve.io/s3-endpoint: minio.minio.svc:9000
    serving.kserve.io/s3-region: us-east-2
    serving.kserve.io/s3-useanoncredential: "false"
    serving.kserve.io/s3-usehttps: "1"
    serving.kserve.io/s3-cabundle-configmap: "local-cabundle"   #<==
  name: storage-config
  namespace: kserve-demo
type: Opaque
EOF

cat<<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
secrets:
- name: storage-config
EOF


cat<<EOF | kubectl create -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    "serving.kserve.io/deploymentMode": "RawDeployment"
  name: sklearn-iris-v2-rest
  namespace: kserve-demo
spec:
  predictor:
    serviceAccountName: sa
    model:
      modelFormat:
        name: sklearn
      name: ""
      resources: {}
      runtime: kserve-sklearnserver
      storageUri: s3://modelmesh-example-models/sklearn/mnist-svm.joblib
EOF

• Logs

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

• Have you added unit/e2e tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Release note:

CA bundle mount options for storage initializer

[israel-hdez]

@Jooho I left a few additional comments. I think nothing is a blocker. I'm happy to approve if you think it is all OK

[israel-hdez]

This PR is now good for me.

[kserve-oss-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: israel-hdez, Jooho, spolti
To complete the pull request process, please assign yuzisun after the PR has been reviewed.
You can assign the PR to them by writing /assign @yuzisun in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Jooho]

@yuzisun any update on this ticket?

[yuzisun]

nit: Kubernetes uses key ca.crt

[Jooho]

that's right bit I tried to use a slightly different name so as not to overlap with anything else.

[yuzisun]

why do we need this env? seems like the presence of AWS_CA_BUNDLE is good enough.

[Jooho]

AWS_CABUNDLE represents the path to the CA bundle certificate, while CA_BUNDLE_CONFIGMAP_NAME represents the configmap name to attach to the storage initialization container.
For backward compatibility reasons, I did not change the semantics of AWS_CABUNDLE.

[yuzisun]

Thanks for the awesome work!!

please help create an issue on website repo to update the doc.
/lgtm
/approve

[oss-prow-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: israel-hdez, Jooho, spolti, yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [yuzisun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Jooho]

doc issue : kserve/website#324