Spark jobs in EMR on EKS through Spark Operator and spark-submit

[adityamohan93]

Description

In Airflow, for EMR on EKS virtual cluster, EmrContainerOperator is advised to start a new Spark job. From doc, seems it needs sparkSubmitJobDriver as input[1] where we provide location for script. EMR on EKS also support Spark Operator to submit jobs[2]. Are these supported in Airflow because from documentation seems it doesn't?

1. https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/emr/emr_eks.html#submit-a-job-to-an-amazon-emr-virtual-cluster
2. https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator-gs.html

Use case/motivation

Run Spark job on AWS EMR on EKS. This question is relevant to anyone doing the same.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

The EmrContainerOperator uses boto3 api
EMRContainers.Client.start_job_run
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr-containers/client/start_job_run.html

So according to the API you just need to set configurationOverrides.

emr_container_task = EmrContainerOperator(
    task_id="spark_job_with_custom_image",
    ...
    configuration_overrides={
        "applicationConfiguration": [
            {
                "classification": "spark-defaults",
                "properties": {
                    "spark.kubernetes.container.image": "your-account-number.dkr.ecr.us-west-2.amazonaws.com/your-custom-spark-image:latest",
                    "spark.kubernetes.driver.container.image": "your-account-number.dkr.ecr.us-west-2.amazonaws.com/your-custom-spark-image:latest",
                    "spark.kubernetes.executor.container.image": "your-account-number.dkr.ecr.us-west-2.amazonaws.com/your-custom-spark-image:latest"
                }
            }
        ]
    }
)