Skip to content

Add OSSRemoteLogIO.from_config and register oss remote logging scheme #70269

Description

@jason810496

Part of #70265 (related: #67056).

Why

#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py: core and
the Task SDK now resolve the handler via ProvidersManager dispatch on the
[logging] remote_base_log_folder URL scheme, instantiating the provider class through a no-arg
from_config() classmethod. This issue migrates the oss scheme.

What

  • Add OSSRemoteLogIO.from_config() in
    providers/alibaba/src/airflow/providers/alibaba/cloud/log/oss_task_handler.py, mirroring the
    legacy branch:
    elif remote_base_log_folder.startswith("oss://"):
    from airflow.providers.alibaba.cloud.log.oss_task_handler import OSSRemoteLogIO
    _default_conn_name_from("airflow.providers.alibaba.cloud.hooks.oss", "OSSHook")
    REMOTE_TASK_LOG = OSSRemoteLogIO(
    **cast(
    "dict[str, Any]",
    {
    "base_log_folder": BASE_LOG_FOLDER,
    "remote_base": remote_base_log_folder,
    "delete_local_copy": delete_local_copy,
    }
    | _io_kwargs,
    )
    )

    — including the [logging] remote_task_handler_kwargs IO-kwargs merge and expanduser on
    base_log_folder, so behavior is unchanged for existing configs.
  • Register the oss scheme under a remote-logging: section in
    providers/alibaba/provider.yaml and mirror it in
    providers/alibaba/src/airflow/providers/alibaba/get_provider_info.py.
  • Add tests mirroring TestS3RemoteLogIOFromConfig in
    providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.
  • Verify end to end with a real system test: set up the actual backend yourself (a real
    service or a local equivalent), run a task with remote logging enabled, and confirm logs are
    uploaded and read back through the new dispatch path. Include the setup and verification
    results in the PR description.

Reference

Merged examples to follow: #69817 (s3), #69816 (cloudwatch). This is the most mechanical of the
remaining migrations — the legacy branch is a straight mirror of the s3 one. If from_config
raises on a bad config, the shared factory falls back to the legacy path, so this is not a breaking
change.

The legacy branch derives a default connection id from OSSHook.default_conn_name; handle the
connection id the same way the merged amazon PRs do (the hook reads
[logging] remote_log_conn_id itself).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions