[batch] shorten worker timeout

[cseed]

After watching batch a bit, it is clear 5m is way too long.

[jigold]

Wow! That seems aggressive. But I guess if the startup time is approximately 60-90 seconds then this makes sense?

[cseed]

So basically watching batch load, if there's no work to do now, there won't be work to do for a while.

Here is an example: Konrad has big, slow jobs that gate many (~3K) medium fast jobs. So when a big job finishes, we spin up 3K cores and run jobs that take 5m. Then we hang out for 5m more waiting for work (there is none), and then wind down, making the node usage about 50% efficient.

I actually want to make it smaller (5s? ... long enough that children of any jobs that just finished will get rescheduled) and add a minimum up time of 1m (since we pay for at least one minute of compute). So I was planning to make it more aggressive in a later PR.

We need to start scraping the database and analyzing it. We could actually quantify this explicitly.

[cseed]

The startup seems closer to 1-2m.