Skip to content

Add deferrable mode to SFTPOperator #65475

@sunildataengineer

Description

@sunildataengineer

Description

The SFTPOperator currently runs in synchronous mode only, blocking
a worker slot for the entire duration of every file transfer.
For large file transfers, this wastes worker resources unnecessarily.

Proposed Solution
Add deferrable=True parameter to SFTPOperator following the same
pattern used in other Airflow providers. When deferrable=True,
the operator will defer execution to a trigger, freeing the worker
slot immediately and allowing the Triggerer to poll for completion.

Implementation Plan

  • Add SFTPOperatorTrigger class in triggers/sftp.py
  • Add deferrable param to SFTPOperator.init()
  • Add defer() call in execute() when deferrable=True
  • Add execute_complete() callback method
  • Add unit tests

Related
Similar to deferrable mode already implemented in SFTPSensor.

I am willing to submit a PR for this.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions