[openmpi] Add custom resources support

[everpeace]

Motivation

Machine learning tasks sometimes require special hardware. Currently we operate several custom resources on our on-premise Kubernetes cluster to run distributed machine learning tasks (with openmpi package) which requires special hardware other than GPUs.

How

• introduce customResources parameter. This specifies custom resources to assign openmpi-job containers in worker pods.
• The value should be comma separated list of custom-resource-name=amount.

Note

This feature doesn't break any backward compatibility. So I would be very happy if it could be merged to the official repo. But I'm not sure that such general custom resources feature should be supported in this package. I am happy if I could have feedbacks.

Development of device plugin in kubernetes repository seems to be very active now. They would introduce special hardware support officially in the near future. For example, FPGA, Solarflare NICs, Infiniband, etc.

---

This change is 

[everpeace]

/assign @jiezhang @inc0
/cc @alsrgv @jlewi

[everpeace]

/test kubeflow-presubmit

[everpeace]

/test kubeflow-presubmit

[inc0]

/lgtm

[jiezhang]

Review status: 0 of 2 files reviewed at latest revision, all discussions resolved.

---

kubeflow/openmpi/workloads.libsonnet, line 178 at r1 (raw file):

  customResources(params, role)::
    if role == ROLE_WORKER then

Can you move { to its previous line? i.e. if role == ROLE_WORKER then {

BTW, we have a script to autoformat: scripts/autoformat_jsonnet.sh

---

Comments from Reviewable

[jiezhang]

Review status: 0 of 2 files reviewed at latest revision, 1 unresolved discussion.

---

kubeflow/openmpi/workloads.libsonnet, line 102 at r1 (raw file):

      image: params.image,
      imagePullPolicy: params.imagePullPolicy,
      resources: std.mergePatch($.resources(params, role), $.customResources(params, role)),

Is it same as + operator? That seems more straightforward.

See object composition in https://jsonnet.org/learning/tutorial.html

---

Comments from Reviewable

[jiezhang]

@everpeace I'm okay with supporting custom resources in this package.

[everpeace]

Review status: 0 of 2 files reviewed at latest revision, 2 unresolved discussions.

---

kubeflow/openmpi/workloads.libsonnet, line 102 at r1 (raw file):

Previously, jiezhang (Jie Zhang) wrote…

Is it same as + operator? That seems more straightforward.

See object composition in https://jsonnet.org/learning/tutorial.html

actually no. $.resources and $.customResources have the same key limits. So, if we did $.resources + $.customResources then, contents of $.resources will be vanished.

---

kubeflow/openmpi/workloads.libsonnet, line 178 at r1 (raw file):

Previously, jiezhang (Jie Zhang) wrote…

Can you move { to its previous line? i.e. if role == ROLE_WORKER then {

BTW, we have a script to autoformat: scripts/autoformat_jsonnet.sh

Sure. Oh, I noticed the script and remembered I ran. Let me rerun.

---

Comments from Reviewable

[everpeace]

@jiezhang I changed the format and squash it. Could you review again? I hope you'll approve this, Thanks 🙇

[jiezhang]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiezhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• kubeflow/openmpi/OWNERS [jiezhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jiezhang]

/lgtm