Cancel job after a certain queueing time

[vkarak]

This a feature request coming from @mand35:

We want to submit our jobs daily. But in some cases nodes can be occupied for longer, than we would like to skin these data points and neglect these runs for nodes which are busy the whole day. We do not want to queue up more and more jobs for one node.

[mand35]

This could be similar to the SLURM option --deadline, e.g. with (datetime.timedelta(days=0,hours=24)+datetime.datetime.now()).isoformat(timespec='seconds')
Thus we can schedule daily jobs and prevent queuing up more and more jobs in our multi day partition.