Skip to content

issue-37213: Standardizing FTPToS3Operator#63082

Open
jroachgolf84 wants to merge 1 commit intoapache:mainfrom
jroachgolf84:issue-37213
Open

issue-37213: Standardizing FTPToS3Operator#63082
jroachgolf84 wants to merge 1 commit intoapache:mainfrom
jroachgolf84:issue-37213

Conversation

@jroachgolf84
Copy link
Collaborator

@jroachgolf84 jroachgolf84 commented Mar 7, 2026

Description

Per #37213:

There seems to be substantial discrepancies between transfer operators, especially those using S3 as a source or destination. For example, in the S3ToSFTPOperator, only a single file can be transferred from the SFTP to the S3 bucket. However, in the FTPToS3Operator an arbitrary number of files can be transferred from S3 to an FTP site. To add to this, the S3ToFTPOperator only supports a single file being transferred.

The PR refactors the FTPToS3Operator to mirror the functionality of the S3ToFTPOperator (it's "inverse" counterpart) and SFTPToS3Operator (it's sibling). This meant removing the ability to move multiple files and sticking with a single-file approach. This matches the behavior of its close relatives and helps to maintain Airflow DAG-authoring best practices.

Testing

These changes were tested E2E, as well as with updated unit tests. These unit tests can be run as follows:

git fetch origin
git checkout origin/issue-37213
git pull

breeze testing providers-tests providers/amazon/tests/unit/amazon/aws/transfers/test_ftp_to_s3.py

closes: #37213

@jroachgolf84 jroachgolf84 requested a review from o-nikolas as a code owner March 7, 2026 23:26
@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Mar 7, 2026
@jroachgolf84 jroachgolf84 changed the title issue-37213: Standardizing FTPToS3Operator issue-37213: Standardizing FTPToS3Operator Mar 8, 2026
@jroachgolf84
Copy link
Collaborator Author

@vincbeck, do you mind taking a peek at this one?

Copy link
Contributor

@vincbeck vincbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those are breaking changes without any deprecation path. I do not think we can move forward with that. We usually first deprecate things, and then remove them

@jroachgolf84
Copy link
Collaborator Author

Would it be okay to go back and add deprecation for those parameters?

@vincbeck
Copy link
Contributor

vincbeck commented Mar 9, 2026

Would it be okay to go back and add deprecation for those parameters?

Definitely

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inconsistency in S3 Transfer Operators

2 participants