Env var duplication

[s-ducks]

Environment variables DASK_SCHEDULER_ADDRESS and DASK_WORKER_NAME are duplicated on each successive worker pod created.

Minimal Complete Verifiable Example:

from dask_kubernetes.operator import KubeCluster
cluster - KubeCluster(name="test", n_workers=3, namespace="external")

kubectl get pods -n external
NAME                                              READY   STATUS    RESTARTS   AGE
test-default-worker-07336f78ed-6495fb4f86-ltwvb   1/1     Running   0          10m
test-default-worker-babb2f1fed-7c6db6cdfc-lpfmf   1/1     Running   0          10m
test-default-worker-e2e95b0bb4-59f647889d-dpw7t   1/1     Running   0          10m
test-scheduler-89b645d66-hzdgr                    1/1     Running   0          10m

then:

kubectl get pod -n external -o yaml test-default-worker-e2e95b0bb4-59f647889d-dpw7t | grep -e DASK_SCHEDULER_ADDRESS -e DASK_WORKER_NAME -e value
    - $(DASK_WORKER_NAME)
    - name: DASK_WORKER_NAME
      value: test-default-worker-babb2f1fed
    - name: DASK_SCHEDULER_ADDRESS
      value: tcp://test-scheduler.external.svc.cluster.local:8786
    - name: DASK_WORKER_NAME
      value: test-default-worker-e2e95b0bb4
    - name: DASK_SCHEDULER_ADDRESS
      value: tcp://test-scheduler.external.svc.cluster.local:8786

we can see that this has the address and name of one of the other worker pods and a duplicated scheduler address. This does not effect the actual set values in the container however:

kubectl -n external exec test-default-worker-e2e95b0bb4-59f647889d-dpw7t -- env | grep DASK
DASK_VERSION=2023.11.0
DASK_WORKER_NAME=test-default-worker-e2e95b0bb4
DASK_SCHEDULER_ADDRESS=tcp://test-scheduler.external.svc.cluster.local:8786

Environment:

• Dask version: 2023.11.0
• Python version: 3.10.12
• Operating System: Mac
• Install method (conda, pip, source): pip

[efeboyaci]

Something related to this issue that has not been resolved.
If env is provided in the daskworkergroup, the env is duplicated in the deployment config. I believe the last entry applies to the pod.
This problem prohibits us to use cluster with a different FQDN.

Example:

DaskWorkerGroup

env:
- name: DASK_SCHEDULER_ADDRESS
  value: tcp://dev-cluster-scheduler:8786

Worker Pod

env:
- name: DASK_SCHEDULER_ADDRESS
  value: tcp://dev-cluster-scheduler:8786
- name: DASK_WORKER_NAME
  value: dev-cluster-default-worker-5f4e5766de
- name: DASK_SCHEDULER_ADDRESS
  value: tcp://dev-cluster-scheduler.dask-operator.svc.cluster.local:8786

Solution

Duplicate items with same names in container["env"] and env must be removed from env before adding it to configuration

dask-kubernetes/dask_kubernetes/operator/controller/controller.py

Lines 166 to 170 in b668cc6

	for container in deployment_spec["spec"]["template"]["spec"]["containers"]:
	if "env" in container:
	container["env"].extend(env)
	else:
	container["env"] = env

dask-kubernetes/dask_kubernetes/operator/controller/controller.py

Lines 202 to 206 in b668cc6

	for container in pod_spec["spec"]["containers"]:
	if "env" in container:
	container["env"].extend(env)
	else:
	container["env"] = env

[jacobtomlinson]

@efeboyaci please open this in a new issue instead of commenting on an existing closed one.