YAML file supports extra json parameters

[VinayGb665]

---

LocalFileSystemBackend supports YAML files to add connections and the extra parameters ( like in here ) would have to be provided as raw JSON instead of default YAML. The PR is to resolve this issue and provide extra parameters as nested key pairs

Relevant to #9477
Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Target Github ISSUE in description if exists
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[mik-laj]

Is there anything that makes this change exclusive to YAML? It seems to me that this problem also occurs in JSON. What do you think to handle this case in the _create_connection function?

[VinayGb665]

Is there anything that makes this change exclusive to YAML? It seems to me that this problem also occurs in JSON. What do you think to handle this case in the _create_connection function?

I am not really sure if changing _create_connection would help here, because the model in models/connection.py defines the extra field to be of type string and whenever a dict gets passed to it, raises an error

[mik-laj]

What do you think about this change? The error message will now contain extra and extra_dejson keys.

From 16b5a7587f5190829f83fdea81493fd001a63ff2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kamil=20Bregu=C5=82a?= <kamil.bregula@polidea.com>
Date: Mon, 29 Jun 2020 08:21:01 +0200
Subject: Accept extra_json parameter in LocalFilesystemBackend

---
 airflow/secrets/local_filesystem.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/airflow/secrets/local_filesystem.py b/airflow/secrets/local_filesystem.py
index 90f565c86..3a94da1e0 100644
--- a/airflow/secrets/local_filesystem.py
+++ b/airflow/secrets/local_filesystem.py
@@ -180,7 +180,7 @@ def _create_connection(conn_id: str, value: Any):
     if isinstance(value, str):
         return Connection(conn_id=conn_id, uri=value)
     if isinstance(value, dict):
-        connection_parameter_names = get_connection_parameter_names()
+        connection_parameter_names = get_connection_parameter_names() | {"extra_dejson"}
         current_keys = set(value.keys())
         if not current_keys.issubset(connection_parameter_names):
             illegal_keys = current_keys - connection_parameter_names
@@ -189,6 +189,14 @@ def _create_connection(conn_id: str, value: Any):
                 f"The object have illegal keys: {illegal_keys_list}. "
                 f"The dictionary can only contain the following keys: {connection_parameter_names}"
             )
+        if "extra" in value and "extra_dejson" in value:
+            raise AirflowException(
+                "The extra and extra_dejson parameters are mutually exclusive. "
+                "Please provide only one parameter."
+            )
+        if "extra_dejson" in value:
+            value["extra"] = json.dumps(value["extra_dejson"])
+            del value["extra_dejson"]
 
         if "conn_id" in current_keys and conn_id != value["conn_id"]:
             raise AirflowException(
-- 
2.20.1

[VinayGb665]

Would work like a charm 🚀 !!
Should I update the merge request ?

[mik-laj]

Should I update the merge request ?

Yes. Please update PR and complete the documentation and tests.

[VinayGb665]

Updated

[mik-laj]

small nitts in docs

[VinayGb665]

small nitts in docs

Updated