[WIP] Make TaskSDK conf respect default config from provider metadata

[jason810496]

related: #60000

Why

Resolve the blocker for #60000 (replacing all Core conf usage in providers). As pointed out in #60074 (comment), we have to make the Task SDK conf respect the provider default values.

How

Currently, there are two ways to specify default values for providers

1. provider_config_fallback_defaults.cfg (outdated with default value specified in provider.yaml, but it still works in Core)
2. provider metadata (the get_provider_info.py entry point via metadata import in ProvidersManager)

The current status before this PR is

1. Only Core conf respects provider_config_fallback_defaults.cfg
2. Neither Core conf nor SDK conf respects the default values from provider metadata

Since both Core and SDK conf require access to the provider config default values, we add new shared logic that respects default config from provider metadata.

What

Refactor both Core conf and Task SDK conf by consolidating all lookups of provider default values into a shared library, including:

• Move the existing logic for reading default values from provider_config_fallback_defaults.cfg from Core conf to a shared module.
• Add a provider_configs property in ProvidersManagerTaskRuntime.
• Add _get_option_from_provider_cfg_config_fallbacks and _get_option_from_provider_metadata_config_fallbacks into the shared _lookup_sequence, and ensure that ProvidersManager / ProvidersManagerTaskRuntime are initialized lazily.

Verification

After the patch, all of the following conf usages will show the correct default values.

For Core:

import contextlib

from airflow.configuration import conf


print(conf.get("core", "executor"))
# defined in provider_config_fallback_defaults.cfg, already works before the fix
print(conf.get("kubernetes_executor", "tcp_keep_idle"))

# defined in Amazon provider.yaml, will raise an exception before the fix
with contextlib.suppress(Exception):
    print(conf.get("aws_lambda_executor", "conn_id"))

For TaskSDK:

import contextlib
from airflow.sdk import conf

print(conf.get("core", "executor"))

# define in provider_config_fallback_defaults.cfg, will raise exception before the fix
with contextlib.suppress(Exception):
    print(conf.get("kubernetes_executor", "tcp_keep_idle"))

# define in Amazon provider.yaml, will raise exception before the fix
with contextlib.suppress(Exception):
    print(conf.get("aws_lambda_executor", "conn_id")) # provider.yaml

TODO / Follow-up

• update unit tests
• add prek script to enforce updating provider.yaml when adding new [section/option] in community providers

[jason810496]

I'm not sure whether everyone agrees with the current direction. I would appreciate it if someone could confirm before we go further, since this is more like an architectural decision.

Thanks!