RFC31: Jobspec V2

[SteVwonder]

Add a new Jobspec version. Mainly designed to handle the flexibility provided by the new flux run and flux submit interfaces in flux-framework/flux-core#2323.

Major changes from V1:

• relaxes restrictions on the types of resources
• relaxes restrictions on the ordering of resources, especially in regards to the slot (i.e., a node can be inside or outside a slot).
• adds exclusive flag to more resources (i.e., outside a slot)
• adds per_resource count key for late-binding of tasks to resources
• adds job dependencies

Still TODO:

• Things that need more discussion (also marked within the text as a TODO warning):

  • should we restrict this jobspec version to just Nodes, CPUs, and GPUs like V1?
  • do we allow resource requests that do not contain core (e.g., slot>node)?
  • do we allow adding exclusive to any resource (i.e., within a slot)?

• Things within the RFC text that I just need to go back and handle (also marked within the text as a TODO warning):

  • do we want to restrict per_resource values to some list of resources
    (maybe only those in the jobspec being validated/written)?
    • In Apparently simple resource request not possible in canonical jobspec? #150, we said that only resources within a slot can be used
  • Use Case 1.3 "Existing Examples"

• More tightening up of the json.schema (not sure ultimately how important this is now that we are using the python Jobspec class for validation in flux-core).

• Add additional compelling use case examples that demonstrate the difference between this version and V1. In particular, how the loosening of the ordering is beneficial.

[SteVwonder]

should we restrict this jobspec version to just Nodes, CPUs, and GPUs like V1?

Based on discussion in #624, if we do restrict the resource set, it should also include hardware threads.

[dongahn]

At the risk of a scope creep, do we want to add memory capacity into V2? Our workflow users already want to specify memory as contrasts.

How about sockets?

How about SSDs? (probably still far fetched?)

[SteVwonder]

Yeah, I'm partial to not restricting the resource set for two reasons:

1. The flux run interface and resource scheduler are both powerful and generic enough to support arbitrary resources types
2. It would generically solve all of the use-cases that you just mentioned.

Downside being that sched-simple wouldn't easily be able to claim complete V2 compliance. It would probably have to have an asterisk next to V2 for a while (e.g., * only supports the following resources: core, node, socket, gpu, slot). I think this logic also applies for any future schedulers that might be developed for Flux; the jump in complexity from Jobspec V1 -> V2 would be quite large.

---

If we do restrict the resource set, I don't think adding sockets would be too much of a burden. AFAIK, sockets can only be children of nodes and parents of cores. This prevents a explosion of the resource ordering space. That being said, the interplay between socket and memory may be complex if we decide to include memory.

If we explicitly add memory, I'm wondering if we should just generically allow it, or be prescriptive about the types of memory layouts V2 would support. For example, do we want to just support node-granularity pool(s) of memory? Or allow socket-level/numa-level as well? Do we allow L1/2/3 cache, or just main memory? If we are restricting the resource set for simplicity and feasibility of implementation, then I suggest we limit the memory that V2 supports to node-granularity main memory (i.e., memory can only be a child of node or slot). This would handle the memory-constrained application use-case (i.e., don't OOM the node) while minimizing the total added complexity. Unfortunately, it would also prevent explicit requests for an application's cores and numa-memory to be co-scheduled

[SteVwonder]

Per the coffee discussion:

• Restricting the resource types to a finite set that we enumerate will make this version more tractable. It also makes the extension of R easier.
• Proposed resource type set: node, socket, core, hardware-thread, gpu, memory, slot
• We should make sure that the slot->(core,memory) use case works. This would be the most portable jobspec option for memory-constrained applications.

[SteVwonder]

We have decided to restrict the resource set to: node, socket, core, hardware-thread, gpu, memory, and slot. Do we also want to impose some sort of limited ordering on those resources? For example, do we want to say that:

• hardware-threads, memory and gpus cannot have children
• valid children for core only include hardware-thread
• valid children for socket, node, and slot include core, memory, hardware-threads, gpus
• node and slot also have socket as a valid child
• core and hardware-thread cannot be siblings

Would it make sense to just say that the ordering is limited to valid orderings produced by hwloc?

[SteVwonder]

During a coffee hour, @eleon requested that slot->node be considered valid jobspec in V2.

[SteVwonder]

During a ☕ hour call, we discussed pushing this forward again. At a September Exaworks meeting, I made a slide that attempts to summarize the V2 changes as follows:

Jobspec V2 opens up supported resource set and orderings:

• Resource set: node, socket, core, hardware-thread, gpu, memory, storage, and slot
• Compute hardware “spine” (intermediate resources optional): 
node->socket->core->hardware-thread
• “Auxillary” resources (i.e., memory, storage, gpu) can ”hang off” of anywhere:
(node->(socket->(core,gpu), memory), storage)

[dongahn]

Yes, it feels like this will be something that can immediately impact our collaboration with 1) HPE, 2) OpenShift and 3) workflow community including ExaWorks.

[SteVwonder]

Need to expand this list of valid types.

[SteVwonder]

Took another pass on this PR today. Flipping this out of draft status since I think the wording is ready to be reviewed. As the OP states, the last big remaining TODOs (that I'm aware of) are to add an additional example or two that is compelling for what V2 can do over V1 and tightening up the jsonschema file a little bit.