Skip to content

Commit

Permalink
Support exposing extra TCP ports in Spark Driver via K8s Ingress (#1998)
Browse files Browse the repository at this point in the history 
* Add driverIngressOptions in SparkApplication CRD

Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>

* Update chart version to 1.3.0

Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>

* Update helm chart README

Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>

* Fix make detect-crds-drift

Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>

* Update api-docs.md

Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>

---------

Signed-off-by: Bo (AIML) Yang <bo_yang6@apple.com>
  • Loading branch information
@hiboyang
hiboyang committed May 20, 2024
1 parent 153537e commit b723367
Show file tree
Hide file tree
Showing 14 changed files with 1,535 additions and 260 deletions.
4 changes: 2 additions & 2 deletions charts/spark-operator-chart/Chart.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
apiVersion: v2
name: spark-operator
description: A Helm chart for Spark on Kubernetes operator
version: 1.2.15
appVersion: v1beta2-1.4.6-3.5.0
version: 1.3.0
appVersion: v1beta2-1.4.2-3.5.0
keywords:
- spark
home: https://github.com/kubeflow/spark-operator
Expand Down
2 changes: 1 addition & 1 deletion charts/spark-operator-chart/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# spark-operator

![Version: 1.2.15](https://img.shields.io/badge/Version-1.2.15-informational?style=flat-square) ![AppVersion: v1beta2-1.4.6-3.5.0](https://img.shields.io/badge/AppVersion-v1beta2--1.4.6--3.5.0-informational?style=flat-square)
![Version: 1.3.0](https://img.shields.io/badge/Version-1.3.0-informational?style=flat-square) ![AppVersion: v1beta2-1.4.2-3.5.0](https://img.shields.io/badge/AppVersion-v1beta2--1.4.2--3.5.0-informational?style=flat-square)

A Helm chart for Spark on Kubernetes operator

Expand Down
37 changes: 37 additions & 0 deletions charts/spark-operator-chart/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3779,6 +3779,43 @@ spec:
serviceType:
type: string
type: object
driverIngressOptions:
items:
properties:
serviceAnnotations:
additionalProperties:
type: string
type: object
serviceLabels:
additionalProperties:
type: string
type: object
ingressURLFormat:
type: string
ingressAnnotations:
additionalProperties:
type: string
type: object
ingressTLS:
items:
properties:
hosts:
items:
type: string
type: array
secretName:
type: string
type: object
type: array
servicePort:
format: int32
type: integer
servicePortName:
type: string
serviceType:
type: string
type: object
type: array
sparkVersion:
type: string
timeToLiveSeconds:
Expand Down
37 changes: 37 additions & 0 deletions charts/spark-operator-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3767,6 +3767,43 @@ spec:
serviceType:
type: string
type: object
driverIngressOptions:
items:
properties:
serviceAnnotations:
additionalProperties:
type: string
type: object
serviceLabels:
additionalProperties:
type: string
type: object
ingressURLFormat:
type: string
ingressAnnotations:
additionalProperties:
type: string
type: object
ingressTLS:
items:
properties:
hosts:
items:
type: string
type: array
secretName:
type: string
type: object
type: array
servicePort:
format: int32
type: integer
servicePortName:
type: string
serviceType:
type: string
type: object
type: array
sparkVersion:
type: string
timeToLiveSeconds:
Expand Down
144 changes: 144 additions & 0 deletions docs/api-docs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -601,6 +601,20 @@ SparkUIConfiguration
</tr>
<tr>
<td>
<code>driverIngressOptions</code><br/>
<em>
<a href="#sparkoperator.k8s.io/v1beta2.DriverIngressConfiguration">
[]DriverIngressConfiguration
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>DriverIngressOptions allows configuring the Service and the Ingress to expose ports inside Spark Driver</p>
</td>
</tr>
<tr>
<td>
<code>dynamicAllocation</code><br/>
<em>
<a href="#sparkoperator.k8s.io/v1beta2.DynamicAllocation">
Expand Down Expand Up @@ -991,6 +1005,122 @@ string
</tr>
</tbody>
</table>
<h3 id="sparkoperator.k8s.io/v1beta2.DriverIngressConfiguration">DriverIngressConfiguration
</h3>
<p>
(<em>Appears on:</em><a href="#sparkoperator.k8s.io/v1beta2.SparkApplicationSpec">SparkApplicationSpec</a>)
</p>
<div>
<p>DriverIngressConfiguration is for driver ingress specific configuration parameters.</p>
</div>
<table>
<thead>
<tr>
<th>Field</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<code>servicePort</code><br/>
<em>
int32
</em>
</td>
<td>
<p>ServicePort allows configuring the port at service level that might be different from the targetPort.</p>
</td>
</tr>
<tr>
<td>
<code>servicePortName</code><br/>
<em>
string
</em>
</td>
<td>
<p>ServicePortName allows configuring the name of the service port.
This may be useful for sidecar proxies like Envoy injected by Istio which require specific ports names to treat traffic as proper HTTP.</p>
</td>
</tr>
<tr>
<td>
<code>serviceType</code><br/>
<em>
<a href="https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#servicetype-v1-core">
Kubernetes core/v1.ServiceType
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>ServiceType allows configuring the type of the service. Defaults to ClusterIP.</p>
</td>
</tr>
<tr>
<td>
<code>serviceAnnotations</code><br/>
<em>
map[string]string
</em>
</td>
<td>
<em>(Optional)</em>
<p>ServiceAnnotations is a map of key,value pairs of annotations that might be added to the service object.</p>
</td>
</tr>
<tr>
<td>
<code>serviceLabels</code><br/>
<em>
map[string]string
</em>
</td>
<td>
<em>(Optional)</em>
<p>ServiceLables is a map of key,value pairs of labels that might be added to the service object.</p>
</td>
</tr>
<tr>
<td>
<code>ingressURLFormat</code><br/>
<em>
string
</em>
</td>
<td>
<p>IngressURLFormat is the URL for the ingress.</p>
</td>
</tr>
<tr>
<td>
<code>ingressAnnotations</code><br/>
<em>
map[string]string
</em>
</td>
<td>
<em>(Optional)</em>
<p>IngressAnnotations is a map of key,value pairs of annotations that might be added to the ingress object. i.e. specify nginx as ingress.class</p>
</td>
</tr>
<tr>
<td>
<code>ingressTLS</code><br/>
<em>
<a href="https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#ingresstls-v1-networking">
[]Kubernetes networking/v1.IngressTLS
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>TlsHosts is useful If we need to declare SSL certificates to the ingress object</p>
</td>
</tr>
</tbody>
</table>
<h3 id="sparkoperator.k8s.io/v1beta2.DriverSpec">DriverSpec
</h3>
<p>
Expand Down Expand Up @@ -2493,6 +2623,20 @@ SparkUIConfiguration
</tr>
<tr>
<td>
<code>driverIngressOptions</code><br/>
<em>
<a href="#sparkoperator.k8s.io/v1beta2.DriverIngressConfiguration">
[]DriverIngressConfiguration
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>DriverIngressOptions allows configuring the Service and the Ingress to expose ports inside Spark Driver</p>
</td>
</tr>
<tr>
<td>
<code>dynamicAllocation</code><br/>
<em>
<a href="#sparkoperator.k8s.io/v1beta2.DynamicAllocation">
Expand Down
37 changes: 37 additions & 0 deletions manifest/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3779,6 +3779,43 @@ spec:
serviceType:
type: string
type: object
driverIngressOptions:
items:
properties:
serviceAnnotations:
additionalProperties:
type: string
type: object
serviceLabels:
additionalProperties:
type: string
type: object
ingressURLFormat:
type: string
ingressAnnotations:
additionalProperties:
type: string
type: object
ingressTLS:
items:
properties:
hosts:
items:
type: string
type: array
secretName:
type: string
type: object
type: array
servicePort:
format: int32
type: integer
servicePortName:
type: string
serviceType:
type: string
type: object
type: array
sparkVersion:
type: string
timeToLiveSeconds:
Expand Down
37 changes: 37 additions & 0 deletions manifest/crds/sparkoperator.k8s.io_sparkapplications.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3767,6 +3767,43 @@ spec:
serviceType:
type: string
type: object
driverIngressOptions:
items:
properties:
serviceAnnotations:
additionalProperties:
type: string
type: object
serviceLabels:
additionalProperties:
type: string
type: object
ingressURLFormat:
type: string
ingressAnnotations:
additionalProperties:
type: string
type: object
ingressTLS:
items:
properties:
hosts:
items:
type: string
type: array
secretName:
type: string
type: object
type: array
servicePort:
format: int32
type: integer
servicePortName:
type: string
serviceType:
type: string
type: object
type: array
sparkVersion:
type: string
timeToLiveSeconds:
Expand Down

0 comments on commit b723367

Please sign in to comment.