fix: spark application check fails on missing section

[pdrastil]

This PR fixes issue #6035.

Modify the check to safely validate optional executorState section.

Running state immediately after job has been created

obj.status.executorState is not present yet and health check also throws exception.

kind: SparkApplication
apiVersion: sparkoperator.k8s.io/v1beta2
metadata:
  name: python
spec:
  type: Python
  sparkVersion: 3.0.1
  mode: cluster
  # The rest was omitted for clarity
status:
  applicationState:
    state: RUNNING
  driverInfo:
    podName: python-driver
    webUIAddress: '172.21.157.105:4040'
    webUIPort: 4040
    webUIServiceName: python-ui-svc
  executionAttempts: 1
  lastSubmissionAttemptTime: '2021-04-14T17:51:00Z'
  sparkApplicationId: spark-635eea7212384dc2beda105497a57b1a
  submissionAttempts: 1
  submissionID: b589f35c-5963-43b0-b28f-3653a890cb5b
  terminationTime: '2021-04-14T17:51:57Z'

Running state after executors have been created

Please notice that obj.status.executorState is present now.

kind: SparkApplication
apiVersion: sparkoperator.k8s.io/v1beta2
metadata:
  name: python
spec:
  type: Python
  sparkVersion: 3.0.1
  mode: cluster
  # The rest was omitted for clarity
status:
  applicationState:
    state: RUNNING
  driverInfo:
    podName: python-driver
    webUIAddress: '172.21.157.105:4040'
    webUIPort: 4040
    webUIServiceName: python-ui-svc
  executionAttempts: 1
  executorState:
    pythonstreamingqueuestream-8630eb78d17ee8fa-exec-4: RUNNING
  lastSubmissionAttemptTime: '2021-04-14T17:51:00Z'
  sparkApplicationId: spark-635eea7212384dc2beda105497a57b1a
  submissionAttempts: 1
  submissionID: b589f35c-5963-43b0-b28f-3653a890cb5b
  terminationTime: '2021-04-14T17:51:57Z'

Failed state

obj.status.executorState is not present at all.

kind: SparkApplication
apiVersion: sparkoperator.k8s.io/v1beta2
metadata:
  name: scala
spec:
  type: Scala
  sparkVersion: 3.0.1
  mode: cluster
  # The rest was omitted for clarity
status:
  applicationState:
    errorMessage: "Exception in thread \"main\" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.21.0.1/api/v1/namespaces/demo-customer/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods \"scala-driver\" is forbidden: exceeded quota: demo-customer, requested: memory=896Mi, used: memory=3268Mi, limited: memory=4Gi."
    state: FAILED
  driverInfo: {}
  lastSubmissionAttemptTime: '2021-04-14T17:51:14Z'
  submissionAttempts: 1
  terminationTime: null

[codecov]

Codecov Report

Merging #6036 (7c17ede) into master (8f53bd5) will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #6036      +/-   ##
==========================================
+ Coverage   40.94%   40.97%   +0.03%     
==========================================
  Files         147      147              
  Lines       19682    19691       +9     
==========================================
+ Hits         8059     8069      +10     
- Misses      10516    10518       +2     
+ Partials     1107     1104       -3     

Impacted Files	Coverage Δ
util/git/creds.go	9.62% <0.00%> (ø)
util/helm/helm.go	60.27% <0.00%> (ø)
util/cache/redis.go	48.83% <0.00%> (ø)
util/helm/client.go	45.02% <0.00%> (ø)
server/cache/cache.go	73.33% <0.00%> (ø)
util/oidc/provider.go	0.00% <0.00%> (ø)
util/session/state.go	69.81% <0.00%> (ø)
util/cache/inmemory.go	35.13% <0.00%> (ø)
util/ksonnet/ksonnet.go	40.00% <0.00%> (ø)
util/kustomize/kustomize.go	63.57% <0.00%> (ø)
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f53bd5...7c17ede. Read the comment docs.

[pdrastil]

Returned the check for the number of running executors with additional guard condition.

[alexmt]

Thank you! Health check looks great now. Unit test broke - I think you just need to change the expected health check message.

[alexmt]

LGTM