[v3-2-test] Fix conf.has_option not respects default provider metadata (#64209)

[github-actions]

• Fix provider config loading and fallback logic

Simplifies and centralizes provider configuration loading by removing redundant methods and properties, ensuring consistent state management. Updates fallback logic to use a sentinel value instead of None, improving accuracy in config lookups. Improves test coverage for provider metadata fallbacks and clarifies lazy initialization behavior. Streamlines provider manager interfaces to reduce duplication and potential import cycles.

• Cleanup load_providers_configuration and respect with new provider conf mechanism

• Fix configuration_description cached_property error to restore .write
  compatibility

• Try fix unit tests error

• Fix provider config merging and section separation

Ensures provider configuration sections do not overlap with core sections
and gives priority to provider metadata when merging configuration options.
Refactors tests to verify correct toggling of provider configuration and
proper separation of provider and core config sections. Updates fallback
lookup order and cleans up unused properties for clarity and maintainability.

• Fix check-default-configuration check and harden the error case

Root cause: providers_discovery.py used structlog.getLogger() directly. Before structlog.configure() is called (which happens later in settings.py:726), structlog's default PrintLogger writes to stdout with no level filtering. So debug logs during early provider
discovery pollute the stdout of airflow config list --default, corrupting the generated config file.

Fix: Switched to logging.getLogger() (stdlib). stdlib logging defaults to WARNING level and writes to stderr, so debug logs are suppressed and stdout stays clean. This is also the correct pattern for shared library code — structlog configuration is the application's
responsibility.

• Use conf.invalidate_cache() instead of dict.pop

• Harden the provider conf tests

• Address Copilot's review comments

• Use invalidate_cache instead of dict.pop

• Restore dict.pop back instead of invalidate_cache

• Fix lookup_sequence not respect _use_providers_configuration

• Address Wei's review comment for run_check_default_configuration

• Add dedicated _invalidate_provider_flag_caches method

• Create fresh conf instance for tests case that might mutate its internal
  state for thread-safe concern

• Deprecate load_providers_configuration,
  restore_core_default_configuration instead of directly remove

• Address Amogh's comment: Deprecate already_initialized_provider_configs instead of directly removing it

Add deprecation warning to already_initialized_provider_configs property
in both ProvidersManager and ProvidersManagerTaskRuntime, guiding users
to use provider_configs instead.

• Address Amogh's comment: Use VALUE_NOT_FOUND_SENTINEL for provider cfg fallback consistency

Change fallback=None to fallback=VALUE_NOT_FOUND_SENTINEL in
get_from_provider_cfg_config_fallback_defaults and update the
corresponding sentinel check for consistency with the metadata variant.

• Address Amogh's comment: Fix typo in _use_providers_configuration comment

---

(cherry picked from commit 4c63306)

Co-authored-by: Jason(Zhe-You) Liu 68415893+jason810496@users.noreply.github.com
Co-authored-by: Rahul Vats 43964496+vatsrahul1001@users.noreply.github.com

[vatsrahul1001]

Dup of #64469