Skip to content

Decouple remote logging config from core#67056

Open
jason810496 wants to merge 7 commits into
apache:mainfrom
jason810496:refactor/logging/decuple-logging-config-from-core
Open

Decouple remote logging config from core#67056
jason810496 wants to merge 7 commits into
apache:mainfrom
jason810496:refactor/logging/decuple-logging-config-from-core

Conversation

@jason810496
Copy link
Copy Markdown
Member

@jason810496 jason810496 commented May 17, 2026

Why

Decouple remote logging config from hardcoded airflow-core/src/airflow/config_templates/airflow_local_settings.py by introducing new remote-logging field for ProvidersManager. Add shared resolve_remote_task_log it, so core and the Task SDK
share one implementation of the precedence rule.

What

Single shared resolver resolve_remote_task_log with an explicit three-tier precedence:

  1. User-defined [logging] logging_config_class module exporting
    REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID.
  2. ProvidersManager scheme dispatch on [logging] remote_base_log_folder,
    instantiated via the provider class's from_config() classmethod.
  3. Legacy attr-path fallback against
    airflow.config_templates.airflow_local_settings (transitional).

airflow.logging_config.load_logging_config is deprecated and split into

  • _get_logging_config() (logging dict)
  • _load_logging_config() (remote handler, lazy)
  • configure_logging no longer eagerly resolves the remote handler.

Next step to deprecate hardcoded airflow_local_settings.py

  1. Provider migration: Migrate the logic from airflow_local_settings.py as from_config method for each current supported RemoteIO.
class S3RemoteLogIO(LoggingMixin):
    @classmethod
    def from_config(cls) -> "S3RemoteLogIO":
        from airflow.providers.common.compat.sdk import conf

        # provider-specific `conf` validation or custom logic
        return cls(
            base_log_folder=conf.get("logging", "base_log_folder"),
            remote_base=conf.get("logging", "remote_base_log_folder"),
            delete_local_copy=conf.getboolean("logging", "delete_local_logs"),
        )
  1. Raise deprecation warning for each if/else branch after the corresponding RemoteIO.from_config is added.

Was generative AI tooling used to co-author this PR?

@jason810496 jason810496 self-assigned this May 17, 2026
@jason810496 jason810496 marked this pull request as ready for review May 18, 2026 02:15
@jason810496 jason810496 requested review from Lee-W, jscheffl and shahar1 May 18, 2026 05:16
Comment thread airflow-core/newsfragments/67056.significant.rst Outdated
Comment thread airflow-core/src/airflow/providers_manager.py Outdated
Copy link
Copy Markdown
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks clean and good to me - except some small comments. Another pair of eyes would be good prior merge.

…config

Decouple importing the [logging] logging_config_class dict from
remote-handler resolution so each can run independently. The deprecated
load_logging_config wrapper delegates to both and keeps the old return
shape. Wire the lazy remote-handler load through the shared
resolve_remote_task_log factory.
@jason810496 jason810496 force-pushed the refactor/logging/decuple-logging-config-from-core branch from aa513bb to 37939dc Compare May 19, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants