Airflow Kubernetes Connection Not Accepting JSON Kubeconfig

[RuizChristianJ]

Apache Airflow version

2.7.3

What happened

I am trying to create a kubernetes connection using a JSON for the kubeconfig. I have tried both using an environment variable and an airflow connection to no avail. I have tested this kubeconfig as a yaml; passing the path to kube_config_path works as intended. But when converting that same yaml to a json and passing it to kube_config breaks.

[2023-11-16, 18:34:20 UTC] {base.py:73} INFO - Using connection ID 'deveks' for task execution.
[2023-11-16, 18:34:20 UTC] {kubernetes_pod.py:667} ERROR - string indices must be integers, not 'str'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 440, in execute
    self.pod_request_obj = self.build_pod_request_obj(context)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 630, in build_pod_request_obj
    "airflow_kpo_in_cluster": str(self.hook.is_in_cluster),
                                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 238, in is_in_cluster
    self.api_client  # so we can determine if we are in_cluster or not
    ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 246, in api_client
    return self.get_conn()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 207, in get_conn
    config.load_kube_config(
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 808, in load_kube_config
    loader = _get_kube_config_loader(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 762, in _get_kube_config_loader
    kcfg = KubeConfigMerger(filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 681, in __init__
    self._load_config_from_file_path(paths)
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 706, in _load_config_from_file_path
    self.load_config(path)
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 724, in load_config
    self._merge(item, config.get(item, []) or [], path)
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 730, in _merge
    if exists['name'] == new_item['name']:
       ~~~~~~^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 618, in __getitem__
    v = self.safe_get(key)
        ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 615, in safe_get
    return self.value[key]
           ~~~~~~~~~~^^^^^
TypeError: string indices must be integers, not 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 483, in cleanup
    for event in self.pod_manager.read_pod_events(pod).items:
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 370, in pod_manager
    return PodManager(kube_client=self.client)
                                  ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 388, in client
    return self.hook.core_v1_client
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 250, in core_v1_client
    return client.CoreV1Api(api_client=self.api_client)
                                       ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 246, in api_client
    return self.get_conn()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 207, in get_conn
    config.load_kube_config(
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 808, in load_kube_config
    loader = _get_kube_config_loader(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 762, in _get_kube_config_loader
    kcfg = KubeConfigMerger(filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 681, in __init__
    self._load_config_from_file_path(paths)
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 706, in _load_config_from_file_path
    self.load_config(path)
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 724, in load_config
    self._merge(item, config.get(item, []) or [], path)
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 730, in _merge
    if exists['name'] == new_item['name']:
       ~~~~~~^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 618, in __getitem__
    v = self.safe_get(key)
        ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/config/kube_config.py", line 615, in safe_get
    return self.value[key]
           ~~~~~~~~~~^^^^^
TypeError: string indices must be integers, not 'str'

What you think should happen instead

It should be able to consume the provided json and establish a connection like it does when using the kube_config yaml file.

How to reproduce

Operating System

Linux

Versions of Apache Airflow Providers

I have tried 5.0.0 up to the latest release of 7.9.0 for apache-airflow-providers-cncf-kubernetes

Deployment

Astronomer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.