Description
The SFTPOperator currently runs in synchronous mode only, blocking
a worker slot for the entire duration of every file transfer.
For large file transfers, this wastes worker resources unnecessarily.
Proposed Solution
Add deferrable=True parameter to SFTPOperator following the same
pattern used in other Airflow providers. When deferrable=True,
the operator will defer execution to a trigger, freeing the worker
slot immediately and allowing the Triggerer to poll for completion.
Implementation Plan
- Add SFTPOperatorTrigger class in triggers/sftp.py
- Add deferrable param to SFTPOperator.init()
- Add defer() call in execute() when deferrable=True
- Add execute_complete() callback method
- Add unit tests
Related
Similar to deferrable mode already implemented in SFTPSensor.
I am willing to submit a PR for this.
Use case/motivation
No response
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Description
The SFTPOperator currently runs in synchronous mode only, blocking
a worker slot for the entire duration of every file transfer.
For large file transfers, this wastes worker resources unnecessarily.
Proposed Solution
Add deferrable=True parameter to SFTPOperator following the same
pattern used in other Airflow providers. When deferrable=True,
the operator will defer execution to a trigger, freeing the worker
slot immediately and allowing the Triggerer to poll for completion.
Implementation Plan
Related
Similar to deferrable mode already implemented in SFTPSensor.
I am willing to submit a PR for this.
Use case/motivation
No response
Related issues
No response
Are you willing to submit a PR?
Code of Conduct