Skip to content

Not able to create k8s connection using airflow UI with kube config (json) #20626

@shivanitomarzeotap

Description

@shivanitomarzeotap

Apache Airflow version

2.1.4

What happened

in_cluster configuration option is disabled, config path is empty and the kubeconfig(json) is provided to the airflow admin connection. As written here because first two options are not selected, it should create the connection with provided kubeconfig json file.
But it fails saying that - Invalid kube-config file

What you expected to happen

If in_cluster configuration and config path options are disabled, it should create the connection with provided kubeconfig json file.

How to reproduce

Create airflow admin connection via airflow UI where in_cluster option is disabled and config path is empty.
Screenshot 2022-01-03 at 1 42 34 PM
and use this connection for dag

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Composer

Deployment details

No response

Anything else

hypothesis - The connection form includes empty string for un-set values, and in code we only have a check for None and not for empty string.

Traceback - [2022-01-03 07:54:31,547] {spark_kubernetes.py:68} INFO - Creating sparkApplication [2022-01-03 07:54:31,586] {taskinstance.py:1465} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1166, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task result = task_copy.execute(context=context) File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 70, in execute response = hook.create_custom_object( File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 168, in create_custom_object api = client.CustomObjectsApi(self.api_client) File "/opt/python3.8/lib/python3.8/functools.py", line 967, in __get__ val = self.func(instance) File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 149, in api_client return self.get_conn() File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 127, in get_conn config.load_kube_config( File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/config/kube_config.py", line 737, in load_kube_config loader = _get_kube_config_loader_for_yaml_file( File "/opt/python3.8/lib/python3.8/site-packages/kubernetes/config/kube_config.py", line 700, in _get_kube_config_loader_for_yaml_file raise ConfigException( kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions