Exposing livenessProbe and readinessProbe Configuration in Task Manager and Job Manager Pod Spec

[hjwalt]

Right now the following is the default provided:

  livenessProbe:
    failureThreshold: 5
    initialDelaySeconds: 5
    periodSeconds: 60
    successThreshold: 1
    tcpSocket:
      port: 6123
    timeoutSeconds: 10

Occasionally we encounter an unusual situation where the task manager did not reach RPC port start up, and was terminated by k8s. But when the port starts up while the pod is in terminating state, the job gets submitted, and killed eventually, creating a restart loop (which after some time resolves when magically the task manager starts before 25 seconds).

I can work on this and create pull request here if you think this is useful.

[regadas]

Hi @hjwalt yeah I think that exposing these can be useful indeed; If you can create a PR please do.

[numbnut]

Hi @hjwalt and @regadas ,
I was as well facing this and raised it in issue 94. I will remove it there, because now it is duplicated. I have two more points.

1. In the GCP Operator it was actually working to set livenessProbe and readynessProbe. I'm not sure, why it has been removed. I thought it might be helpful to mention this.
2. Would you mind to implement it for the jobmanager as well?

[hjwalt]

@numbnut sure, I'll add the same for job manager

[numbnut]

Hi @hjwalt , thank you!

[regadas]

Closed via #111