Invalid keys in executor_config should raise an error

[karenbraganz]

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

10.0.1

Apache Airflow version

2.10.5

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Astronomer

Deployment details

No response

What happened

The executor_config dictionary used to configure worker pods with the Kubernetes executor does not raise an error if any keys besides "pod_override" or "pod_template_file" are used. Even with invalid keys, the DAG is imported and will run as expected. The invalid executor_config will simply be disregarded.

What you think should happen instead

If any keys besides "pod_override" or "pod_template_file" are used in executor_config, and error (maybe a DAG import error) should be raised.

This should give the user feedback that their executor_config keys are invalid and need to be removed or changed.

How to reproduce

1. Create an Airflow instance that uses the Kubernetes Executor.
2. Create a DAG with the following code including an invalid key in executor_config.

import datetime

from airflow.decorators import dag
from airflow.decorators import task

executor_config = {
    "key": "value"
}

@dag(start_date=datetime.datetime(2024, 10, 1), schedule="@daily", catchup=False)
def dag_1():

    @task(executor_config=executor_config)
    def task_1():
        print("task 1")

    task_1()

dag_1()

3. Observe that the DAG will be imported and run on the Airflow instance without any issues.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[amoghrajesh]

I think we should instead introduce a stricter type to the executor_config and not just let it be a dict which can have any values flowing in?

[insomnes]

I think we should instead introduce a stricter type to the executor_config and not just let it be a dict which can have any values flowing in?

With multiple / hybrid executors it may be tricky to detect on the Operator initialization stage which executor would be used and properly validate the config.

Maybe executors can have some subclass of the common config connected, so at least we can demand it to be of the specific type, and the user can "prevalidate" their config this way. When hybrid executors are gone, this task would be more easy. Because with the deterministic executor from the operator args it should be simple to map it with the proper executor config type and "validate"

[HsiuChuanHsu]

Hi @karenbraganz, this issue is highly relevant to our work, as we frequently use executor_config settings, so i think this would be great if we can detect any invalid keys when the DAG is imported.

[o-nikolas]

@karenbraganz

For the reasons @insomnes stated, we can't do any broad validation at the base class level at this point in time. Groundwork (similar to what @insomnes described) would have to be done first, such as each executor creating a type for their executor_config and validation functions that the Base Operator/Dag processor code could use to validate the input in a transparent way. But that is a large chunk of work that needs to be done delicately in it's own project of sorts.