[bitnami/spark] Allow to access Spark worker page easily

[hongbo-miao]

Name and Version

bitnami/spark 3.3.2-debian-11-r19

What is the problem this feature will solve?

It would be great to allow to access Spark worker page.

Prefect also has a master / worker pattern.
In Prefect server (master) helm chart, there is a publicApiUrl, so that all UI related URLs will point to that URL.

If existing helm values not able to do it, it would be great to provide something like that, thanks! 😃

What is the feature you are proposing to solve the problem?

Originally asked at Stack Overflow. Below is a copy:

---

I have a local k3s Kubernetes created by Rancher Desktop.

I installed Spark by

helm upgrade \
  spark \
  spark \
  --install \
  --repo=https://charts.bitnami.com/bitnami \
  --namespace=hm-spark \
  --create-namespace

➜ kubectl get services -n hm-spark
NAME                                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
spark-headless                                        ClusterIP   None            <none>        <none>                       32h
spark-master-svc                                      ClusterIP   10.43.200.139   <none>        7077/TCP,80/TCP              32h

➜ kubectl get pods -n hm-spark
NAME                                              READY   STATUS      RESTARTS   AGE
spark-master-0                                    1/1     Running     0          32h
spark-worker-0                                    1/1     Running     0          32h
spark-worker-1                                    1/1     Running     0          32h

Currently I am doing port-forward

➜ kubectl port-forward service/spark-master-svc --namespace=hm-spark 4040:80

to access the Spark master UI at http://localhost:4040/

However, I won't be able to open Spark worker job pages as they are using Kubernentes internal cluster IP.

Here is this chart values: https://github.com/bitnami/charts/blob/main/bitnami/spark/values.yaml

Is there any value I can set to help me access the Spark worker job page? Thanks!

I found a similar question deployed by Docker Swarm but also has no accepted answers.

What alternatives have you considered?

spark-on-k8s-operator would be potential option.

[javsalgar]

Hi,

It seems to me that this would be a case where add a public service endpoint to each node might be necessary. You can check the externalAccess section in other charts like bitnami/redis-cluster if I'm not mistaken. Would that work for your use case?

[hongbo-miao]

Thanks @javsalgar ! I think externalAccess is the thing I want.

However, for redis-cluster helm chart, externalAccess is defined at here in the values.yaml file.
Unfortunately, Spark helm chart values.yaml does not have this externalAccess currently.

[javsalgar]

Thanks! I will forward this to the engineering team, but we cannot guarantee an ETA. However, if you want to speed up the process, would you like to submit a PR?

[hongbo-miao]

Thanks @javsalgar ! Unfortunately, I think I don't have enough knowledge for this task.
No hurry, it is nice-to-have feature. Definitely gonna attract more people using this chart 😃

[rafariossaa]

Hi,
I am creating an internal tasks in order to implement this. However, I can not provide an ETA.
If someone if interested in sending a PR we will be glad to review and merge it.

[jeluizferreira]

Hi.
Did you tried the following config option, at master and worker sections:

  configOptions: >-
    -Dspark.ui.reverseProxy=true
    -Dspark.ui.reverseProxyUrl=https://spark.youdomain.com

[hongbo-miao]

Thanks @jeluizferreira ! And thanks @javsalgar and @rafariossaa too!

I found the related document at https://github.com/bitnami/charts/tree/main/bitnami/spark#configuring-spark-master-as-reverse-proxy

In my case I am using Cloudflare Tunnel. So I don't need ingress related part in the doc.

Here is my step:

First I pointed http://spark-master-svc.hm-spark.svc:80 to spark.mydomain.com in my Cloudflare Tunnel.

Then deploy Spark by

helm upgrade \
  spark \
  spark \
  --install \
  --repo=https://charts.bitnami.com/bitnami \
  --namespace=hm-spark \
  --create-namespace \
  --values=my-values.yaml

my-values.yaml

master:
  configOptions:
    -Dspark.ui.reverseProxy=true
    -Dspark.ui.reverseProxyUrl=https://spark.mydomain.com
worker:
  configOptions:
    -Dspark.ui.reverseProxy=true
    -Dspark.ui.reverseProxyUrl=https://spark.mydomain.com

Then I am able to visit both master and worker pages! 😃