Customization for Job containers

[dseapy]

Summary

Be able to set various container settings such as annotations, labels, resources, etc. on buffer-create & buffer-cleanup Jobs.

Use Cases

• When needing to deploy with ResourceQuota or LimitRange in the namespace one must supply & rely on defaults specified in a LimitRange, otherwise jobs will not run. Don't know if leveraging existing standardResources or if it being configurable is best, but being able to have something set would be awesome.
• There are probably various use-cases someone might want to inject labels or annotations. For my current setup, I hit an issue when having istio auto-injection enabled in the namespace, and needing to disable it for the pod using an sidecar.istio.io/inject: "false" annotation to avoid injecting a sidecar which prevents the Job from terminating.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[dseapy]

actually, realizing that standardResources doesn't contain limits, only requests. My current setup has a ResourceQuota in the namespace requiring defining memory limits. I can set defaults in a LimitRange for now as a workaround for the resources bit.

[dseapy]

i can work on this. for my setup things will get cleaner once I can set annotations on all pods and resources on all containers, then I can avoid the resourcequota & istio namespace workarounds I've got atm.

to summarize what specifically i'd like to be able to do:

• be able to add sidecar.istio.io/inject: "false" annotation to pods created by numaflow, needed when deploying in a namespace with auto istio-injection enabled.
• ensure memory limits on all containers, so that if i'm deploying in a namespace with a ResourceQuota requiring memory limits, I can avoid needing to override in a LimitRange which affects non-numaflow managed pods in the same namespace.

for this i believe the tasks left to do in numaflow are:

1. override any container resources (where standardResources is currently used)
2. set annotations on any pods not yet supporting it (buffer-create, buffer-delete)

---

assuming there are no objections to allowing user to configure these things, I have some questions:

1. where is best set init container resources?
2. do we use same resources/annotations/etc for buffer-create and buffer-delete, or allow users to set them differently? where is best set them?

Seems like for (1) and (2) they could be set either in the Pipeline resource or maybe globally in controller configmap. It also seems possible (2) could be set on the isbsvc instead.

For both (1) and (2) how granular controls are desired - ranging one setting per-controller all the way to controlling per-pod/container (i.e. init container specified in vertices).

WDYT @vigith @whynowy