Handle unschedulable jobs in k8s runner

[almahmoud]

Addresses #10480

• Adds k8s_unschedulable_walltime_limit which specifies the time limit for a job to wait while pods are unschedulable.
• Makes job not be marked as running in Galaxy if Job is dispatched but job is still unschedulable
• Handles marking the job as failed with a reasonable message and cleaning up the job when unschedulable past the specified limit

Testing everything together:

• Job is not marked as running if the pod is unschedulable
• Specified time limit is respected (tested with 30 seconds, defaulted to 30 mins)
• Job error is properly reported
  
• Job resource is cleaned up in k8s
• Schedulable jobs still run unaffected

[nuwang]

I think it would be good to guard against the condition where this file doesn't exist, because I've seen several instances (in other places in the runner) where writing to the error file fails.

[mvdbeek]

What do we need this file for ? Can we eliminate this ? This was used in older traditional HPC runners that would kill a job and then write something to a specific file, but kubernetes doesn't do that, and we can write to job_state.fail_message. I would assume that ends up on the job_stderr column of the job table.

[almahmoud]

I was just following what is done to handle failures in other places eg https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/kubernetes.py#L521 . Can we just change this as part of moving to the pulsar runner? Not really worth the time sink now? I can just remove the file and do fail message

[mvdbeek]

Thanks @almahmoud!