-
Notifications
You must be signed in to change notification settings - Fork 13.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support regex pattern in SFTPHOOK #15409
Conversation
@mik-laj The author of this issue prefers to support |
I think, we should use fnmatchr because other operators also uses this function. This allows you to standardize the experience of using different operators so that you don't have to think about the parameter format. |
It would be great if your methods were similar to the SFTP for GCS operator. |
def retrieve_file(self, remote_full_path: str, local_full_path: str) -> None: | ||
def retrieve_file( | ||
self, remote_full_path: str, local_full_path: str, regex_pattern: Optional[str] = None | ||
) -> None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing I really dislike about this implementation is it subtly changes semantics depending on the arguments. In this function, if regex_pattern
is None, remote_full_path
should point to a file, but when pattern matching is performed, it should point to a directory instead (and the file name matched with the pattern). This will be very unintuitive in practice and likely cause headaches to both users and maintainers.
Pattern matching is definitely a useful feature here, but I really don’t think this is how it should be done.
@saveriogzz Do you want to implement it? |
yes, I will work on it! |
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
In many cases, users are interested in some cases of files or types, not the whole thing. for example, users want to retrieve only CSV files in the
/tmp
directory and etc. Unfortunately,Paramiko
does not support wirelated: #15332
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.