[LIVY-703] Show SparkUI link on Livy UI when running on Kubernetes

[jahstreet]

What changes were proposed in this pull request?

Jira

This PR is one of the PRs in the series related to the splitting of the PR #167 to multiple PRs to ease and speed up review and merge processes.

This PR targets branch of the base PR #249 and should be merged after it.

This PR implements a way to build Spark UI links on Livy UI when running on Kubernetes. This patch is required to unlock further integrations with Kubernetes Ingress resources (ref: #167).

Refer jahstreet#35 to compare changes.

How was this patch tested?

Unit tests.

Manual testing with Kubernetes on Docker Desktop for Mac v2.1.0.1.
Environment - Helm charts:

• cluster-base with custom-values.yaml:

nginx-ingress:
  controller:
    service:
      loadBalancerIP: 127.0.0.1 # my-cluster.example.com IP address (from /etc/hosts)
      loadBalancerSourceRanges: []
cluster-autoscaler:
  enabled: false
oauth2-proxy:
  enabled: false

• spark-cluster with custom-values.yaml:

livy:
  image:
    pullPolicy: Never
    tag: 0.7.0-incubating-spark_2.4.3_2.11-hadoop_3.2.0-dev
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
    path: /livy/?(.*)
    hosts:
    - my-cluster.example.com
    tls:
    - secretName: spark-cluster-tls
      hosts:
      - my-cluster.example.com
  persistence:
    enabled: true
  env:
    LIVY_LIVY_UI_BASE1PATH: {value: "/livy"}
    LIVY_SPARK_KUBERNETES_CONTAINER_IMAGE_PULL1POLICY: {value: "Never"}
    LIVY_SPARK_KUBERNETES_CONTAINER_IMAGE: {value: "sasnouskikh/livy-spark:0.7.0-incubating-spark_2.4.3_2.11-hadoop_3.2.0-dev"}
    LIVY_LIVY_SERVER_SESSION_STATE0RETAIN_SEC: {value: "300s"}
    LIVY_LIVY_SERVER_KUBERNETES_ALLOWED1NAMESPACES: {value: "default,test"}
    LIVY_LIVY_UI_KUBERNETES_SPARKUI_ENABLED: {value: "true"}
    LIVY_LIVY_UI_KUBERNETES_SPARKUI_LINK0FORMAT: {value: "/spark-ui/%s"}
historyserver:
  enabled: false
jupyterhub:
  enabled: true
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
    hosts:
    - my-cluster.example.com
    pathSuffix: ''
    tls:
    - secretName: spark-cluster-tls
      hosts:
      - my-cluster.example.com
  hub:
    baseUrl: /jupyterhub
    publicURL: "https://my-cluster.example.com"
    activeServerLimit: 10
    # $> openssl rand -hex 32
    cookieSecret: 41b85e5f50222b1542cc3b38a51f4d744864acca5e94eeb78c6e8c19d89eb433
    pdb:
      enabled: true
      minAvailable: 0
  proxy:
    # $> openssl rand -hex 32
    secretToken: cc52356e9a19a50861b22e08c92c40b8ebe617192f77edb355b9bf4b74b055de
    pdb:
      enabled: true
      minAvailable: 0
  cull:
    enabled: false
    timeout: 300
    every: 60

• Interactive sessions - Jupyter notebook on JupyterHub with Sparkmagic
• Batch sessions - SparkPi:

curl -k -H 'Content-Type: application/json' -X POST \
  -d '{
        "name": "SparkPi-01",
        "className": "org.apache.spark.examples.SparkPi",
        "numExecutors": 2,
        "file": "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.3.jar",
        "args": ["10000"],
        "conf": {
            "spark.kubernetes.namespace": "<namespace>"
        }
      }' "https://my-cluster.example.com/livy/batches"

[codecov-io]

Codecov Report

Merging #252 into master will decrease coverage by 1.22%.
The diff coverage is 42.58%.

@@             Coverage Diff              @@
##             master     #252      +/-   ##
============================================
- Coverage     67.92%   66.69%   -1.23%     
- Complexity      939      956      +17     
============================================
  Files           102      103       +1     
  Lines          5876     6189     +313     
  Branches        891      946      +55     
============================================
+ Hits           3991     4128     +137     
- Misses         1314     1464     +150     
- Partials        571      597      +26

Impacted Files	Coverage Δ	Complexity Δ
...ain/java/org/apache/livy/rsc/driver/RSCDriver.java	79.25% <0%> (-1.51%)	45 <0> (ø)
rsc/src/main/java/org/apache/livy/rsc/RSCConf.java	88.18% <100%> (+0.33%)	9 <1> (+1)	⬆️
...main/scala/org/apache/livy/server/LivyServer.scala	32.88% <20%> (-0.46%)	11 <1> (ø)
...e/livy/server/interactive/InteractiveSession.scala	68.93% <33.33%> (-0.44%)	46 <0> (ø)
...ala/org/apache/livy/utils/SparkKubernetesApp.scala	42.2% <42.2%> (ø)	14 <14> (?)
...rc/main/scala/org/apache/livy/utils/SparkApp.scala	45.23% <5.55%> (-30.77%)	1 <0> (ø)
...rver/src/main/scala/org/apache/livy/LivyConf.scala	95.02% <85%> (-1%)	22 <1> (+1)
...ain/scala/org/apache/livy/utils/SparkYarnApp.scala	67.32% <0%> (+1.3%)	40% <0%> (ø)	⬇️
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7313c5...4050c0d. Read the comment docs.

[ajbozarth]

Following my review on #249 on a couple typos here, again I'm not the best to review you new files, but afaik the changes look fine as far as code styling and formatting is concerned.

Feel free to tag me for review on future incremental PRs that touch the Livy core, conf, and ui code (especially the UI stuff since I wrote that whole library)

[ajbozarth]

typo: "Whether'

[ajbozarth]

typo: "Whether'

[jahstreet]

Feel free to tag me for review on future incremental PRs that touch the Livy core, conf, and ui code (especially the UI stuff since I wrote that whole library)

Thanks man, I appreciate that!

[codecov-commenter]

Codecov Report

❌ Patch coverage is 42.58675% with 182 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.69%. Comparing base (ba12b51) to head (4050c0d).
⚠️ Report is 87 commits behind head on master.

Files with missing lines	Patch %	Lines
...ala/org/apache/livy/utils/SparkKubernetesApp.scala	42.20%	130 Missing and 22 partials ⚠️
...rc/main/scala/org/apache/livy/utils/SparkApp.scala	5.55%	15 Missing and 2 partials ⚠️
...main/scala/org/apache/livy/server/LivyServer.scala	20.00%	3 Missing and 1 partial ⚠️
...e/livy/server/interactive/InteractiveSession.scala	33.33%	1 Missing and 3 partials ⚠️
...rver/src/main/scala/org/apache/livy/LivyConf.scala	85.00%	3 Missing ⚠️
...ain/java/org/apache/livy/rsc/driver/RSCDriver.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #252      +/-   ##
============================================
- Coverage     68.06%   66.69%   -1.37%     
- Complexity      940      956      +16     
============================================
  Files           101      103       +2     
  Lines          5855     6189     +334     
  Branches        886      946      +60     
============================================
+ Hits           3985     4128     +143     
- Misses         1299     1464     +165     
- Partials        571      597      +26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jahstreet]

It was merged in another MR.