Connection inputs in Extra field are overwritten by custom form widget fields

[josh-fell]

Apache Airflow version:
2.1.0

What happened:
There are several hooks that still use optional parameters from the classic Extra field. However, when creating the connection the Extra field is overwritten with values from the custom fields that are included in the form. Because the Extra field is overwritten, these optional parameters cannot be used by the hook.

For example, in the AzureDataFactoryHook, If resource_group_name or factory_name are not provided when initializing the hook, it defaults to the value specified in the connection extras. Using the Azure Data Factory connection form, here is the initial connection submission:

After saving the connection, the Extra field is overwritten with the custom fields that use "extras" under the hood:

What you expected to happen:

Wavering slightly but I would have initially expected that the Extra field wasn't overwritten but updated with both the custom field "extras" plus the originally configured values in the Extra field. However, a better UX would be that the values used in the Extra field should be separate custom fields for these hooks and the Extra field is hidden. Perhaps it's even both?

How to reproduce it:
Install either the Microsoft Azure or Snowflake providers, attempt to create a connection for either the Snowflake, Azure Data Factory, Azure Container Volume, or Azure types with the Extra field populated prior to saving the form.

Anything else we need to know:
Happy to submit PRs to fix this issue. 🚀

[potiuk]

Yeah. Smart merging of the fields is best approach. There are already quite a few connections where the "extra" field is hidden (when available extra are exhaustive) but I think it would not be too difficult to make it works smart and automatically split/merge both approaches. Feel free to provide PR for that