scheduler: RescheduleTracker dropped if follow-up fails placements

[tgross]

When an allocation fails it triggers an evaluation. The evaluation is processed and the scheduler sees it needs to reschedule, which triggers a follow-up eval. The follow-up eval creates a plan to (stop 1) (place 1). The replacement alloc has a RescheduleTracker (or gets its RescheduleTracker updated).

But in the case where the follow-up eval can't place all allocs (there aren't enough resources), it can create a partial plan to (stop 1) (place 0). It then creates a blocked eval. The plan applier stops the failed alloc. Then when the blocked eval is processed, the job is missing an allocation, so the scheduler creates a new allocation. This allocation is not a replacement from the perspective of the scheduler, so it's not handed off a RescheduleTracker.

This changeset fixes this by annotating the reschedule tracker whenever the scheduler can't place a replacement allocation. We check this annotation for allocations that have the stop desired status when filtering out allocations to pass to the reschedule tracker. I've also included tests that cover this case and expands coverage of the relevant area of the code.

Fixes: #12147
Fixes: #17072
Fixes: https://hashicorp.atlassian.net/browse/NET-9551

---

Workflow of the typical happy path:

graph TD;
id1(alloc fails)--client status: failed-->id2(eval A);
id2(eval A)-->id3(follow-up eval B);
id3(follow-up eval B)--desired status: stop-->id4(alloc A);
id3(follow up eval B)--create with RescheduleTracker -->id5(alloc B);

Loading

Workflow of the buggy path:

graph TD;
id1(alloc fails)--client status: failed-->id2(eval A);
id2(eval A)-->id3(follow-up eval B);
id3(follow-up eval B)--desired status: stop-->id4(alloc A);
id3(follow-up eval B)--could not place alloc B-->id5(blocked eval C);
id5(blocked eval C)--create without RescheduleTracker -->id6(alloc B);

Loading

[gulducat]

I haven't fully absorbed all the implications of this, but here are my comments so far

[gulducat]

lgtm!

[pkazmierczak]

LGTM!

[tgross]

Waiting on some comments Juanita told me she was working on before merging this.

[Juanadelacuesta]

Great work in this meta bug :)