Client networking from spark executors to kubernetes worker

[julwin]

Hi everyone,

I'm having issues with our setup. We're using:

• Airflow on k8s via the community helm chart
• KubernetesExecutor
• Spark on k8s via the bitnami helm chart
• Pyspark
• SparkSubmitOperator

When using the SparkSubmitOperator with Pyspark scripts, it is required to set deploy-mode = "client". If we do so, the spark executor (for simplicity we are running the Spark cluster on the same k8s cluster) needs to connect to the spark driver (in this case our k8s pod running SparkSubmitOperator) on a dynamic port range, which can be configured by setting

conf={
            "spark.driver.port": "42000",
            "spark.driver.blockManager.port": "42017",
}

The spark executors are now unable to connect to the driver hostname, since it has no DNS name (and the port range is not exposed).

We tried working around the exposed ports doing this:

executor_config={
            "pod_override": k8s.V1Pod(
                metadata=k8s.V1ObjectMeta(labels={"spark": "driver"}),
                spec=k8s.V1PodSpec(
                    containers=[
                        k8s.V1Container(
                            name="base",
                            image="custom-image:spark-0.1.4",
                            ports=[
                                k8s.V1ContainerPort(container_port=port)
                                for port in range(42000, 42033)
                            ],
                        ) 
                    ],
                ),
            )
        }

But that still leaves us unable to connect. What is then suggested by the spark docs is to create a headless service for your deployment to enable the DNS resolution.

When creating a headless service manually, the connection now works. But when using KubernetesExecutors, we get new issues when multiple worker pods are created running SparkSubmitOperators which then either map or don't map to the headless service, depending on how loose the selector is defined.

My question now is: Is it possible to dynamically add a headless service to each worker pod when it is created? I tried modifying the pod_template.yaml by adding a headless service, but it does not accept multiple resources.

I was thinking about creating a custom "watcher" deployment and using it in combination with pod lifecycle policies to dynamically create and tear down headless services, but that seems overly complex.
Am I overlooking a more elegant solution here?

[ska-ibees]

Hi @julwin,

I'm encountering a similar challenge with Spark and Airflow on Kubernetes (Exactly same setup as yours). Could you share how you resolved or worked around the issue with executors connecting to the driver pod in client mode? Any insights or updates would be greatly appreciated.

Thanks!

[bryanbai]

any update ?