Deployment fails when using Istio 1.14

[mkls6]

Checks

• I have searched the existing issues.
• This issue is NOT specific to the CLI. (If so, please open an issue on the CLI repo)

deployKF Version

0.1.2

Kubernetes Version

Client Version: v1.24.16
Kustomize Version: v4.5.4
Server Version: v1.24.16

Description

I tried to deploy Kubeflow using deployKF to cluster with separately installed Istio 1.14.6. According to documentation deployKF should be compatible with version 1.14, but is does not seem to be the case. When applying the generated applications to cluster oauth2-proxy fails to start, istiod logs errors (see logs below). I was able to reproduce the issue in k3s/minikube environment.

With istio 1.15+ everything starts successfully. It seems to me that the issue is with deploykf-istio-gateway--ext-authz-disable EnvoyFilter. It uses envoy.extensions.filters.http.lua.v3.LuaPerRoute, which has been introduced in Envoy 1.15.

Relevant Logs

istiod:
2023-10-26T12:55:59.552785Z     warn    ads     ADS:RDS: ACK ERROR deploykf-gateway-d5b485fb6-pqrk5.deploykf-istio-gateway-1 Internal:Didn't find a registered implementation for name: 'envoy.filters.http.lua.set_userid_header'

deploykf-gateway:
2023-10-26T12:42:01.839962Z     warning envoy config    gRPC config for type.googleapis.com/envoy.config.route.v3.RouteConfiguration rejected: Didn't find a registered implementation for name: 'envoy.filters.http.lua.set_userid_header'
2023-10-26T12:42:12.341233Z     warning envoy config    gRPC config for type.googleapis.com/envoy.config.route.v3.RouteConfiguration rejected: Didn't find a registered implementation for name: 'envoy.filters.http.lua.set_userid_header'
2023-10-26T12:55:59.552330Z     warning envoy config    gRPC config for type.googleapis.com/envoy.config.route.v3.RouteConfiguration rejected: Didn't find a registered implementation for name: 'envoy.filters.http.lua.set_userid_header'

oauth2-proxy:
[2023/10/26 13:05:24] [provider.go:55] Performing OIDC Discovery...
[2023/10/26 13:05:24] [main.go:60] ERROR: Failed to initialise OAuth2 Proxy: error intiailising provider: could not create provider data: error building OIDC ProviderVerifier: could not get verifier builder: error while discovery OIDC configuration: failed to discover OIDC configuration: unexpected status "403": ExtAuthz did not set 'x-auth-request-email' header

deployKF Values (Optional)

# Basically default values with some components disabled
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: deploykf-app-of-apps
  namespace: argocd
  labels:
    app.kubernetes.io/name: deploykf-app-of-apps
    app.kubernetes.io/part-of: deploykf
spec:
  project: "default"
  source:
    repoURL: "https://github.com/deployKF/deployKF.git"
    targetRevision: "v0.1.2"
    path: "."

    plugin:
      name: "deploykf"
      parameters:
        - name: "source_version"
          string: "0.1.2"

        - name: "values_files"
          array:
            - "./sample-values.yaml"

        - name: "values"
          string: |
            deploykf_dependencies:
              cert_manager:
                enabled: true
                namespace: cert-manager

                clusterIssuer:
                  enabled: true
                  issuerName: kf-selfsigned-issuer

              istio:
                # Istio is deployed externally
                enabled: false
                namespace: istio-system

              kyverno:
                enabled: true
                namespace: kyverno

            deploykf_core:
              deploykf_auth:
                dex:
                  staticPasswords:
                    - email: "admin@example.com"
                      password:
                        value: "admin"
                    - email: "user1@example.com"
                      password:
                        value: "user1"
                    - email: "user2@example.com"
                      password:
                        value: "user2"
                  clients:
                    oauth2Proxy:
                      clientId: "oauth2-proxy"
                      clientSecret:
                        existingSecret: "generated--dex-oauth2-proxy-client"
                        existingSecretKey: "client_secret"
                        generateSecret: true

              deploykf_istio_gateway:
                namespace: deploykf-istio-gateway
                gatewayService.type: "ClusterIP"

                charts:
                  istioGateway:
                    enabled: true
                    name: "gateway"
                    version: "1.14.6"

                gateway:
                  hostname: deploykf.example.com
                  ports:
                    http: 8080
                    https: 8443
                  tls:
                    enabled: false
                  selectorLabels:
                    app: deploykf-gateway
                    istio: deploykf-gateway

                gatewayDeployment:
                  serviceAccount:
                    name: deploykf-gateway
                    annotations: {}

                gatewayService:
                  name: deploykf-gateway
                  annotations: {}
                  type: ClusterIP  # Do not expose it at least for now

              deploykf_profiles_generator:
                profileDefaults:
                  profileNamePrefix: ""
                  ownerEmail: "admin@example.com"

                  memberAccess:
                    role: view
                    notebooksAccess: false

                  plugins: []
                  tools: {}

                users:
                  - id: user-1
                    email: "user1@example.com"
                  - id: user-2
                    email: "user2@example.com"

                groups:
                  - id: team-1
                    users:
                      - user-1
                      - user-2

                profiles:
                  - name: team-1
                    members:
                      - group: team-1
                        access:
                          role: edit
                          notebooksAccess: true

                  - name: team-1-prod
                    members:
                      - group: team-1
                        access:
                          role: view
                          notebooksAccess: false

            deploykf_opt:
              deploykf_minio:
                enabled: false
              deploykf_mysql:
                enabled: false

            kubeflow_dependencies:
              kubeflow_argo_workflows:
                enabled: false

            kubeflow_tools:
              katib:
                enabled: false
              notebooks:
                enabled: true
              pipelines:
                enabled: false

  destination:
    server: "https://kubernetes.default.svc"
    namespace: "argocd"

[thesuperzapper]

@mkls6 thanks for raising this, I did a bit of testing and also ran into the same issue when using Istio 1.14.6 with deployKF 0.1.2.

Either way, given how old Istio 1.14 is, I strongly recommend NOT using it, and I will update the website to reflect its incompatibility with deployKF (which will also let me use some of the newer envoy features in future releases of deployKF).

Finally, there is no clear reason why this should not work in Istio 1.14.6, which actually uses Envoy 1.22.7. I suspect this is actually an Istio or Envoy bug that was patched before the release of Envoy 1.23.0 or Istio 1.15.0.

You can check the envoy version of any Istio proxy by using this docker command:

docker run --entrypoint "/usr/local/bin/envoy" docker.io/istio/proxyv2:1.14.6-distroless --version
# /usr/local/bin/envoy  version: a757eae4ff3ad133fc75efa953e6642436316f98/1.22.7-dev/Clean/RELEASE/BoringSSL