[Feature] Add mTLS Support via Cert Manager to RayCluster #4566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

andrewsykim merged 76 commits into ray-project:master from opendatahub-io:enhanced-security-features

Jul 27, 2026

.buildkite/test-e2e.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -214,6 +214,27 @@ @@
         - KUBERAY_TEST_TIMEOUT_SHORT=1m KUBERAY_TEST_TIMEOUT_MEDIUM=5m KUBERAY_TEST_TIMEOUT_LONG=10m go test -timeout 30m -v ./test/e2erayjobsubmitter 2>&1 | awk -f ../.buildkite/format.awk | tee $$KUBERAY_TEST_OUTPUT_DIR/gotest.log || (kubectl logs --tail -1 -l app.kubernetes.io/name=kuberay | tee $$KUBERAY_TEST_OUTPUT_DIR/kuberay-operator.log && cd $$KUBERAY_TEST_OUTPUT_DIR && find . -name "*.log" | tar -cf /artifact-mount/e2e-log.tar -T - && exit 1)
         - echo "--- END:RayJob Light Weight Submitter E2E (nightly operator) tests finished"
+    - label: 'Test RayCluster mTLS E2E (nightly operator)'
+      instance_size: large
+      image: golang:1.26-bookworm
+      commands:
+        - source .buildkite/setup-env.sh
+        - kind create cluster --wait 900s --config ./ci/kind-config-buildkite.yml
+        - kubectl config set clusters.kind-kind.server https://docker:6443
+        # Install cert-manager (required for RayClusterMTLS auto-generate e2e tests)
+        - pushd ray-operator
+        - make certmanager
+        - source ../.buildkite/build-start-operator.sh
+        - kubectl wait --timeout=90s --for=condition=Available=true deployment kuberay-operator
+        # Run mTLS e2e tests and print KubeRay operator logs if tests fail
+        - echo "--- START:Running RayCluster mTLS E2E (nightly operator) tests"
+        - if [ -n "$${KUBERAY_TEST_RAY_IMAGE}" ]; then echo "Using Ray Image $${KUBERAY_TEST_RAY_IMAGE}"; fi
+        - set -o pipefail
+        - mkdir -p "$(pwd)/tmp" && export KUBERAY_TEST_OUTPUT_DIR=$(pwd)/tmp
+        - echo "KUBERAY_TEST_OUTPUT_DIR=$$KUBERAY_TEST_OUTPUT_DIR"
+        - KUBERAY_TEST_TIMEOUT_SHORT=1m KUBERAY_TEST_TIMEOUT_MEDIUM=5m KUBERAY_TEST_TIMEOUT_LONG=10m go test -timeout 30m -v -run TLS ./test/e2e 2>&1 | awk -f ../.buildkite/format.awk | tee $$KUBERAY_TEST_OUTPUT_DIR/gotest.log || (kubectl logs --tail -1 -l app.kubernetes.io/name=kuberay | tee $$KUBERAY_TEST_OUTPUT_DIR/kuberay-operator.log && cd $$KUBERAY_TEST_OUTPUT_DIR && find . -name "*.log" | tar -cf /artifact-mount/e2e-mtls-log.tar -T - && exit 1)
+        - echo "--- END:RayCluster mTLS E2E (nightly operator) tests finished"
     - label: 'Test RayCronJob E2E (nightly operator)'
       instance_size: large
       image: golang:1.26-bookworm
@@ Expand Down @@

.buildkite/values-kuberay-operator-override.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -22,3 +22,5 @@ featureGates: @@
         enabled: true
       - name: SidecarSubmitterRestart
         enabled: true
+      - name: RayClusterMTLS
+        enabled: true

docs/reference/api.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -492,6 +492,7 @@ _Appears in:_ @@
     | `enableInTreeAutoscaling` _boolean_ | EnableInTreeAutoscaling indicates whether operator should create in tree autoscaling configs |  |  |
     | `gcsFaultToleranceOptions` _[GcsFaultToleranceOptions](#gcsfaulttoleranceoptions)_ | GcsFaultToleranceOptions for enabling GCS FT |  |  |
     | `networkPolicy` _[NetworkPolicyConfig](#networkpolicyconfig)_ | NetworkPolicy specifies optional configuration for network isolation.<br />When set, separate NetworkPolicies are created for head and worker pods.<br />The reconciler always permits intra-cluster pod-to-pod traffic.<br />Note: under DenyAll/DenyAllEgress, DNS egress is not added<br />automatically; since Ray pods reach the head via its service FQDN, you must<br />allow DNS egress via Head/Worker EgressRules or the cluster will fail to start. |  |  |
+    | `tlsOptions` _[TLSOptions](#tlsoptions)_ | TLSOptions specifies optional TLS encryption settings for the RayCluster.<br />If omitted or Enabled is false, TLS is disabled. When Enabled is true,<br />the operator enables mTLS using cert-manager to provision and manage certificates.<br />Requires the RayClusterMTLS feature gate on the operator. |  |  |
     | `headGroupSpec` _[HeadGroupSpec](#headgroupspec)_ | HeadGroupSpec is the spec for the head pod |  |  |
     | `rayVersion` _string_ | RayVersion is used to determine the command for the Kubernetes Job managed by RayJob |  |  |
     | `workerGroupSpecs` _[WorkerGroupSpec](#workergroupspec) array_ | WorkerGroupSpecs are the specs for the worker pods |  |  |
@@ Expand Down Expand Up / @@ -759,6 +760,26 @@ _Appears in:_ @@
     | `backoffLimit` _integer_ | BackoffLimit of the submitter. In K8sJobMode, this is the K8s Job backoffLimit.<br />In SidecarMode with SidecarSubmitterRestart enabled, this is the maximum container restart count. |  |  |
+    #### TLSOptions
+    TLSOptions configures TLS encryption for the RayCluster.
+    When TLSOptions is nil or Enabled is nil/false, TLS is disabled.
+    When Enabled is true, the operator uses cert-manager to automatically
+    provision a full PKI (self-signed CA, head and worker leaf certificates)
+    and keeps certificates up to date as pod IPs change during autoscaling.
+    _Appears in:_
+    - [RayClusterSpec](#rayclusterspec)
+    | Field | Description | Default | Validation |
+    | --- | --- | --- | --- |
+    | `enabled` _boolean_ | Enabled controls whether mTLS is active for this RayCluster.<br />Defaults to false when omitted. Set to true to enable mTLS. |  |  |
     #### UpscalingMode
     _Underlying type:_ _string_
@@ Expand Down @@

helm-chart/kuberay-operator/README.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -183,6 +183,8 @@ spec: @@
     | featureGates[3].enabled | bool | `false` |  |
     | featureGates[4].name | string | `"RayCronJob"` |  |
     | featureGates[4].enabled | bool | `false` |  |
+    | featureGates[5].name | string | `"RayClusterMTLS"` |  |
+    | featureGates[5].enabled | bool | `false` |  |
     | metrics.enabled | bool | `true` | Whether KubeRay operator should emit control plane metrics. |
     | metrics.serviceMonitor.enabled | bool | `false` | Enable a prometheus ServiceMonitor |
     | metrics.serviceMonitor.interval | string | `"30s"` | Prometheus ServiceMonitor interval |
@@ Expand Down @@

helm-chart/kuberay-operator/crds/ray.io_rayclusters.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

helm-chart/kuberay-operator/crds/ray.io_raycronjobs.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -5082,6 +5082,11 @@ spec: @@
                             type: string
                           suspend:
                             type: boolean
+                          tlsOptions:
+                            properties:
+                              enabled:
+                                type: boolean
+                            type: object
                           upgradeStrategy:
                             properties:
                               type:
@@ Expand Down @@

helm-chart/kuberay-operator/crds/ray.io_rayjobs.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

helm-chart/kuberay-operator/crds/ray.io_rayservices.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

helm-chart/kuberay-operator/templates/_helpers.tpl

-Original file line number
+Diff line change
@@ Expand Up / @@ -195,8 +195,10 @@ rules: @@
       - secrets
       verbs:
       - create
+      - delete
       - get
       - list
+      - update
       - watch
     - apiGroups:
       - ""
@@ Expand Down Expand Up / @@ -229,6 +231,31 @@ rules: @@
       - patch
       - update
       - watch
+    - apiGroups:
+      - cert-manager.io
+      resources:
+      - certificates
+      verbs:
+      - create
+      - get
+      - list
+      - update
+      - watch
+    - apiGroups:
+      - cert-manager.io
+      resources:
+      - certificates/status
+      verbs:
+      - get
+    - apiGroups:
+      - cert-manager.io
+      resources:
+      - issuers
+      verbs:
+      - create
+      - get
+      - list
+      - watch
     - apiGroups:
       - coordination.k8s.io
       resources:
@@ Expand Down @@

helm-chart/kuberay-operator/values.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -157,6 +157,8 @@ featureGates: @@
       enabled: false
     - name: RayCronJob
       enabled: false
+    - name: RayClusterMTLS
+      enabled: false
     # Configurations for KubeRay operator metrics.
     metrics:
@@ Expand Down @@

ray-operator/Makefile

-Original file line number
+Diff line change
@@ Expand Up / @@ -180,7 +180,10 @@ api-docs: crd-ref-docs ## Generate CRD documentation. @@
         --output-path=../docs/reference/api.md
     certmanager:
-    	kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
+    	kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.4/cert-manager.yaml
+    	kubectl wait --namespace cert-manager --for=condition=Available deployment/cert-manager --timeout=120s
+    	kubectl wait --namespace cert-manager --for=condition=Available deployment/cert-manager-webhook --timeout=120s
+    	kubectl wait --namespace cert-manager --for=condition=Available deployment/cert-manager-cainjector --timeout=120s
     deploy-with-webhooks: manifests kustomize certmanager ## Deploy controller with webhooks to the K8s cluster specified in ~/.kube/config.
     	cd config/default-with-webhooks && $(KUSTOMIZE) edit set image kuberay/operator=${IMG}
@@ Expand Down @@

ray-operator/apis/ray/v1/raycluster_types.go

-Original file line number
+Diff line change
@@ Expand Up / @@ -51,6 +51,12 @@ type RayClusterSpec struct { @@
     	// allow DNS egress via Head/Worker EgressRules or the cluster will fail to start.
     	// +optional
     	NetworkPolicy *NetworkPolicyConfig `json:"networkPolicy,omitempty"`
+    	// TLSOptions specifies optional TLS encryption settings for the RayCluster.
+    	// If omitted or Enabled is false, TLS is disabled. When Enabled is true,
+    	// the operator enables mTLS using cert-manager to provision and manage certificates.
+    	// Requires the RayClusterMTLS feature gate on the operator.
+    	// +optional
+    	TLSOptions *TLSOptions `json:"tlsOptions,omitempty"`
     	// HeadGroupSpec is the spec for the head pod
     	HeadGroupSpec HeadGroupSpec `json:"headGroupSpec"`
     	// RayVersion is used to determine the command for the Kubernetes Job managed by RayJob
@@ Expand All / @@ -61,6 +67,18 @@ type RayClusterSpec struct { @@
     	WorkerGroupSpecs []WorkerGroupSpec `json:"workerGroupSpecs,omitempty"`
     }
+    // TLSOptions configures TLS encryption for the RayCluster.
+    // When TLSOptions is nil or Enabled is nil/false, TLS is disabled.
+    // When Enabled is true, the operator uses cert-manager to automatically
+    // provision a full PKI (self-signed CA, head and worker leaf certificates)
+    // and keeps certificates up to date as pod IPs change during autoscaling.
+    type TLSOptions struct {
+    	// Enabled controls whether mTLS is active for this RayCluster.
+    	// Defaults to false when omitted. Set to true to enable mTLS.
+    	// +optional
+    	Enabled *bool `json:"enabled,omitempty"`
+    }
     // +kubebuilder:validation:Enum=Recreate;None
     type RayClusterUpgradeType string
@@ Expand Down @@

ray-operator/apis/ray/v1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

ray-operator/config/crd/bases/ray.io_rayclusters.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

ray-operator/config/crd/bases/ray.io_raycronjobs.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -5082,6 +5082,11 @@ spec: @@
                             type: string
                           suspend:
                             type: boolean
+                          tlsOptions:
+                            properties:
+                              enabled:
+                                type: boolean
+                            type: object
                           upgradeStrategy:
                             properties:
                               type:
@@ Expand Down @@

ray-operator/config/crd/bases/ray.io_rayjobs.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

ray-operator/config/crd/bases/ray.io_rayservices.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

ray-operator/config/overlays/test-overrides/deployment-override.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -9,4 +9,4 @@ spec: @@
           containers:
           - name: kuberay-operator
             args:
-            - --feature-gates=RayClusterStatusConditions=true,RayJobDeletionPolicy=true,RayMultiHostIndexing=true,RayCronJob=true,RayServiceIncrementalUpgrade=true,SidecarSubmitterRestart=true,GCSFaultToleranceEmbeddedStorage=true
+            - --feature-gates=RayClusterStatusConditions=true,RayJobDeletionPolicy=true,RayMultiHostIndexing=true,RayCronJob=true,RayServiceIncrementalUpgrade=true,SidecarSubmitterRestart=true,GCSFaultToleranceEmbeddedStorage=true,RayClusterMTLS=true

ray-operator/config/rbac/role.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -52,8 +52,10 @@ rules: @@
       - secrets
       verbs:
       - create
+      - delete
       - get
       - list
+      - update
       - watch
     - apiGroups:
       - ""
@@ Expand Down Expand Up / @@ -86,6 +88,31 @@ rules: @@
       - patch
       - update
       - watch
+    - apiGroups:
+      - cert-manager.io
+      resources:
+      - certificates
+      verbs:
+      - create
+      - get
+      - list
+      - update
+      - watch
+    - apiGroups:
+      - cert-manager.io
+      resources:
+      - certificates/status
+      verbs:
+      - get
+    - apiGroups:
+      - cert-manager.io
+      resources:
+      - issuers
+      verbs:
+      - create
+      - get
+      - list
+      - watch
     - apiGroups:
       - coordination.k8s.io
       resources:
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Feature] Add mTLS Support via Cert Manager to RayCluster #4566

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewsykim Jul 23, 2026

Uh oh!

chipspeak Jul 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Feature] Add mTLS Support via Cert Manager to RayCluster #4566

Uh oh!

[Feature] Add mTLS Support via Cert Manager to RayCluster #4566

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewsykim Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

chipspeak Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!