Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making SFTPHook's constructor consistent with its superclass SSHHook #20164

Merged
merged 2 commits into from
Dec 14, 2021

Conversation

saveriogzz
Copy link
Contributor

This is just a first dummy solution that probably scores low for elegance since I'm making both the parameters Optional.
What do you guys think? I will adjust according to your suggestions :)

related issue: #20144


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@potiuk
Copy link
Member

potiuk commented Dec 9, 2021

I think adding a test for that would be nice too.

@potiuk
Copy link
Member

potiuk commented Dec 9, 2021

(and fixing static checks - I heartily recommend installing pre-commit. It helps with the iterations a LOT.

@github-actions
Copy link

github-actions bot commented Dec 9, 2021

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Dec 9, 2021
@potiuk
Copy link
Member

potiuk commented Dec 13, 2021

Looks good. I thin what it needs is a bit more docstring description on which parameter is preferred but looks good otherwise.

@potiuk
Copy link
Member

potiuk commented Dec 13, 2021

You can also take a look at examples and see which connection is used there (and align with the preferred one)

@saveriogzz
Copy link
Contributor Author

I am writing some tests and I will send a complete PR as soon as I'm done!

You can also take a look at examples and see which connection is used there (and align with the preferred one)

Do you mean example dags?

it should be possible to create tests for the two versions of the hook constructor using 'parametrized'
@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

Do you mean example dags?

Yep.

@saveriogzz saveriogzz marked this pull request as ready for review December 14, 2021 12:57
@saveriogzz
Copy link
Contributor Author

I didn't find any example dag including the SftpHook or Operator!

@potiuk
Copy link
Member

potiuk commented Dec 14, 2021

I didn't find any example dag including the SftpHook or Operator!

😱 How about adding one ?

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah.. I think example for that one can be added later :)

@potiuk potiuk merged commit f35ad27 into apache:main Dec 14, 2021
@saveriogzz
Copy link
Contributor Author

😱 How about adding one ?

Will do!

@qgallet
Copy link
Contributor

qgallet commented Jan 7, 2022

I believe this introduces a regression on any use of the SFTPHook that doesn't set a specific sftp_conn_id. Since the parameter has a default value, this is then used to override ssh_conn_id with it.

This happens for instance on the SFTPToGCSOperator. It worked on version 2.3.0 of the provider, but now on 2.4.0, no matter what connection is passed in its constructor, the hook will use "sftp_default".

One simple fix would be to remove the default value on sftp_conn_id but I'm not sure it's completely safe and backwards compatible. Or maybe switch the order of the two parameters so that sftp_conn_id is first. What do you think ?

@saveriogzz
Copy link
Contributor Author

saveriogzz commented Jan 8, 2022

I believe this introduces a regression on any use of the SFTPHook that doesn't set a specific sftp_conn_id. Since the parameter has a default value, this is then used to override ssh_conn_id with it.

This should be fixed by #20756

@potiuk
Copy link
Member

potiuk commented Jan 8, 2022

Yeah. I yanked the 2.4.0 release and 2.4.1rc1 with the fix included is up for voting. We will release it likely on Wednesday

@saveriogzz
Copy link
Contributor Author

Yeah. I yanked the 2.4.0 release and 2.4.1rc1 with the fix included is up for voting. We will release it likely on Wednesday

Thanks Jarek! Sorry for the mistake in my PR

@potiuk
Copy link
Member

potiuk commented Jan 8, 2022

Thanks Jarek! Sorry for the mistake in my PR

No worries :) Happens. That's why we have so flexible mechanims and split the release to separate providers that we can easily handle those kind of situations :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers okay to merge It's ok to merge this PR as it does not require more tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants