SFTPOperator does not finish downloading

[ReneTC]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

I have an issue with the SFTPOperator from airflow.providers.sftp.operators.sftp.. Mostly it works perfectly.
However, sometimes it refuses to download the complete file and is just stuck "downloading". A job has been running for 7 days to download a 38 mb file. I can see in the output file there is 37.5 mb out of the 38 mb. Last output of the log is Starting to transfer from ftp_server_file.csv to local_gcs_file.csv

But most of the time it works perfectly - It has downloaded a 39 mb file before without issues.

I'm downloading from a sftp server to google cloud storage in Airflow 2.3.3.

What you think should happen instead

Somehow this operator still think it is downloading. Which makes sense since the file is not 100% downloaded. But the download speed must be close to 0 byte per second. Job should terminate.

How to reproduce

No response

Operating System

Google Cloud Composer

Versions of Apache Airflow Providers

No response

Deployment

Composer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Taragolis]

@ReneTC Could you provide a bit more information about which version of providers are you use?

Did you tried latest version of SFTP provider? Because I see in provider changelog that we in 4.0.0 switch from pysftp to paramico and without actual version it is quite difficult to determine is it problem in old version of package or this error still persist in new one.

[ReneTC]

I see. I am using

Composer version:
2.0.28

Airflow version:
2.3.3

My providers settings are set to:
apache-airflow-providers-sftp = "~=2.5.0"

I will convince admin to update and report back.

[potiuk]

One of the reasons for paramico change was that pysftp did not allow for the kind of configuration that direct paramiko would. You should likely look at keepalive_interval and possibly other parameters there.

The problem that you might experience is because some networking equipment between you and the server could have been restarted (usual thing in the cloud) and if you do not enable keep-alives, you might as a client not even be aware that the other end has stopped responding.

The nature of TCP connection is such that there is no "physical" connection set up - those are just networking packets sent over the network to certain ports. So if you have no "keepalive" enabled, the client might simply not realize the other end is disconnected because if the other end did not have time to signal that it has died or the CLOSE message might not have been delivered.

Only TCP keepalives can give you the certainty that you detect this situation. And they are disabled by default - you need to enable them - in case of SSHook (and SFTPHook derives from it in the new implementation) via the keepalive_interval.

[potiuk]

I am closing it provisionally - because I believe keepalive is the solution. I believe you might also not need to upgrade the provider - because keepalive seems to be also available in previous version of the hook (the SFTP provider before also used SSHook as a base and I believe you have 3.0.0 version of ssh provider: https://airflow.apache.org/docs/apache-airflow-providers-ssh/3.0.0/_api/airflow/providers/ssh/hooks/ssh/index.html So likely you can test keepalive without migration (though of course it is advised).

[potiuk]

Of course, we can re-open it if you test both keepalive and migration to newer version and the issue is still there (though I think some more debugging info would be needed for that).