Satisfiability check within resource

[dongahn]

Creating this issue so that this can be addressed separate from Issue #468.

resource has two match operations: match allocate and match allocate_orelse_reserve. The former doesn't tell whether the allocation request failed because of the current resource state (i.e., no resource available now) or the jobspec cannot be satisfied at all. By contrast, if the latter fails, that means the jobspec is not satisfiable -- No matter how far in the timeline I move my schedule point (as far as the resource state becomes pristine), the jobspec cannot be reserved.

I propose we add match allocate_orelse_satisfiability to overcome this challenge. This is required to complete the qmanager integration with job-manager: if the jobspec is unsatisfiable, qmanager needs to send back a "request denied" to the alloc request; otherwise, it will keep trying to allocate on every job and resource event.

[SteVwonder]

Great idea!

It may also be worthwhile to have a match satisfiability command that can be run once per job at ingestion, to give users an early feedback about their job (at the expense of extra work at the scheduler).

[dongahn]

Conceptually, that's a good idea. But implementation wise, we will have to be very careful.

In particular, I am worried about the performance implication that this will have on resource-match. A satisfiability check can be very expensive (in fact much more expensive then match_allocate itself) as it has to walk the entire resource graph. So it would be wise to avoid a full walk unless we have to.

I was thinking to add this for the alloc request handler within qmanager for the first cut. It will be only called if the match allocate request of a jobspec fails so it shouldn't affect the scheduler performance for those "normal" jobs at all. Users won't get early feedback. But they will eventually get their feedback when the job is at least checked at least once within the schedule loop.

Now, for early feedback, I can see why we can add a lightweight check (check various limits) though. This won't be a full satisfiability check. But this would be useful as the first filter. Such a lightweight check will probably also belong to resource match but I have't given much thought there.

[dongahn]

PR #504 resolves this.