Allocating Resources

Resource allocation requests generally come in two forms. First, users include an initial resource request when submitting applications to the scheduler queue. Traditionally, this has been in the form of a static envelope designed to fit the maximum size of the job.

However, interest in dynamic resource management has also been growing as new programming models continue to emerge in the high-performance computing and data analytics areas. Of primary concern is the enablement of application-directed resource changes in partnership with the scheduler. Several broad categories are envisioned, including the ability to:

• request allocation of additional resources, including memory, bandwidth, and compute. This should be accomplished in a non-blocking manner so that the application can continue to progress while waiting for resources to become available. Note that the new allocation will be disjoint from (i.e., not affiliated with) the allocation of the requestor - thus the termination of one allocation will not impact the other.
• extend the reservation on currently allocated resources, subject to scheduling availability and priorities. This includes extending the time limit on current resources, and/or requesting additional resources be allocated to the requesting job. Any additional allocated resources will be considered as part of the current allocation, and thus will be released at the same time.
• release currently allocated resources that are no longer required. This is intended to support partial release of resources since all resources are normally released upon termination of the job. The identified use-cases include resource variations across discrete steps of a workflow, as well as applications that spawn sub-jobs and/or dynamically grow/shrink over time
• "lend" resources back to the scheduler with an expectation of getting them back at some later time in the job. This can be a proactive operation (e.g., to save on computing costs when resources are temporarily not required), or in response to scheduler requests in lieue of preemption. A corresponding ability to "reacquire" resources previously released would be required.

PMIx provides APIs for executing both static and dynamic requests (see, e.g., PMIx_Allocation_request_nb). Regardless of the use-case, the type and amount of resources involved must be specified. Every scheduler, of course, has its own syntax for specifying resource, and the PMIx community has no intention of attempting to standardize such definitions. Instead, the community has focused on providing a few generalized attributes that schedulers and PMIx implementors can easily translate.

General Attributes

#define PMIX_ALLOC_ID "pmix.alloc.id" // (char*) provide a string identifier for this allocation request // which can later be used to query status of the request

Network Resources

Network resource requests headed by PMIX_ALLOC_NETWORK #define PMIX_ALLOC_NETWORK "pmix.alloc.net" // (pmix_data_array_t*) Array of pmix_info_t where the key // represents the key to be used when passing the resources // to client procs, and the value indicates the number of // resources to be allocated. If not given as part of an // info struct that identifies the impacted nodes, then the // description will be applied across all nodes in the // requestor's allocation. #define PMIX_ALLOC_NETWORK_ID "pmix.alloc.netid" // (char*) key to be used when accessing this requested allocation #define PMIX_ALLOC_BANDWIDTH "pmix.alloc.bw" // (float) Mbits/sec #define PMIX_ALLOC_NETWORK_QOS "pmix.alloc.netqos" // (char*) quality of service level #define PMIX_ALLOC_TIME "pmix.alloc.time" // (uint32_t) time in seconds #define PMIX_ALLOC_NETWORK_TYPE "pmix.alloc.nettype" // (char*) type of desired transport (e.g., tcp, udp) #define PMIX_ALLOC_NETWORK_PLANE "pmix.alloc.netplane" // (char*) id string for the NIC to be used for this allocation // (e.g., CIDR for Ethernet) #define PMIX_ALLOC_NETWORK_ENDPTS "pmix.alloc.endpts" // (size_t) number of endpoints to allocate

Compute Resources

#define PMIX_ALLOC_NUM_NODES "pmix.alloc.nnodes" // (uint64_t) number of nodes #define PMIX_ALLOC_NODE_LIST "pmix.alloc.nlist" // (char*) regex of specific nodes #define PMIX_ALLOC_NUM_CPUS "pmix.alloc.ncpus" // (uint64_t) number of cpus #define PMIX_ALLOC_NUM_CPU_LIST "pmix.alloc.ncpulist" // (char*) regex of #cpus for each node #define PMIX_ALLOC_CPU_LIST "pmix.alloc.cpulist" // (char*) regex of specific cpus indicating the cpus involved. #define PMIX_ALLOC_MEM_SIZE "pmix.alloc.msize" // (float) number of Mbytes

#define JSRUN_NRS “jsrun.nrs” #define JSRUN_USE_ALLOCATION “jsrun.use_allocation” #define JSRUN_GPUS_PER_RS "jsrun.gpu_per_rs" #define JSRUN_MEM_PER_RS “jsrun.mem_per_rs” #define JSRUN_CPUS_PER_RS “jsrun.cpu_per_rs" #define JSRUN_LATENCY_PRIORITY "jsrun.latency_priority" #define JSRUN_LAUNCH_ON_ROOT “jsrun.root_scheduling" #define JSRUN_USE_RESOURCE “jsrun.use_resource" #define JSRUN_RESOURCES_PER_HOST "jsrun.nrs_per_host"