Worker plugins are instantiated per worker. So, for N nodes, there will be N workers. The number of threads is dictated by the number of required cores per instance. However, by default a DVM daemon will "see" all of the hardware resources, and now we potentially have N such daemons for N such nodes that can cause resource contention.
One way to resolve this is to specify each DVM daemon is limited to the node on which the worker is running, which requires a little more bookkeeping to ensure that the right daemon is used when starting a task, but this approach would work. Another way is to have a scheduler plugin run a single DVM daemon instead of a worker plugin running multiple daemons. With that approach the scheduler plugin would spin up a single DVM daemon that would automatically connect with slurm to learn the available resources. Submitted tasks would work similarly to the way they do now -- they would refer to the URI of that single daemon, which would, again, know all and see all and ensure the resources are allocated and managed appropriately. Shutting down the scheduler would merely shutdown the daemon.
The approach I'm going with is the latter as that's more in line with how DVM daemons were originally intended to be used--a single daemon managing resources for a job, not multiple such daemons. Moreover, it gives us the possibliity of having IPS tasks use cores that span multiple nodes, which was not possible in the preivous implementation. (Not that anyone would currently do that, but now we'd have that possibility.) Additionally, in edge cases where we have a heterogeneous compute environment, as with the ExCL cluster, DVM will automatically become aware of the available resources, regardless. (I'm not suggesting we use the ExCL cluster, I'm just using that as a tangible example of such a HPC configuration. I imagine this is not that unusual in university settings where it's conceivable to have dissimilar nodes with identical CPU architectures.)
Worker plugins are instantiated per worker. So, for N nodes, there will be N workers. The number of threads is dictated by the number of required cores per instance. However, by default a DVM daemon will "see" all of the hardware resources, and now we potentially have N such daemons for N such nodes that can cause resource contention.
One way to resolve this is to specify each DVM daemon is limited to the node on which the worker is running, which requires a little more bookkeeping to ensure that the right daemon is used when starting a task, but this approach would work. Another way is to have a scheduler plugin run a single DVM daemon instead of a worker plugin running multiple daemons. With that approach the scheduler plugin would spin up a single DVM daemon that would automatically connect with slurm to learn the available resources. Submitted tasks would work similarly to the way they do now -- they would refer to the URI of that single daemon, which would, again, know all and see all and ensure the resources are allocated and managed appropriately. Shutting down the scheduler would merely shutdown the daemon.
The approach I'm going with is the latter as that's more in line with how DVM daemons were originally intended to be used--a single daemon managing resources for a job, not multiple such daemons. Moreover, it gives us the possibliity of having IPS tasks use cores that span multiple nodes, which was not possible in the preivous implementation. (Not that anyone would currently do that, but now we'd have that possibility.) Additionally, in edge cases where we have a heterogeneous compute environment, as with the ExCL cluster, DVM will automatically become aware of the available resources, regardless. (I'm not suggesting we use the ExCL cluster, I'm just using that as a tangible example of such a HPC configuration. I imagine this is not that unusual in university settings where it's conceivable to have dissimilar nodes with identical CPU architectures.)