Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow loading of directory content via SFTP when timeout is set #3506

Closed
DiceMaster opened this issue Jan 30, 2019 · 5 comments

Comments

Projects
None yet
4 participants
@DiceMaster
Copy link

commented Jan 30, 2019

Directory listing is wery slow when connecting to a SFTP server with CURLOPT_TIMEOUT or -m cmd option. Without a timeout everything works fast.

Tested on my Mac (macos 10.14.2) when connecting to other Mac with curl v7.64.0 (or with daily snapshot 20190129) with following command:
curl sftp://<host>/ -m 20 -u <user>

There is some delay for loading of each subdirectory/file.
In curl v7.61.0 the same command performs almost instantly.

After some investigation I found that the bug was introduced with changes to lib/ssh.c in commit 1a89099

@bagder bagder added the SCP/SFTP label Jan 30, 2019

@koskokos2

This comment has been minimized.

Copy link

commented Jan 31, 2019

Noticed the very same problem and for me it works very slow even without any additional arguments just:
curl sftp://<auth>@<host>/

In version 7.61.0 that command takes (for a folder with 4 entries) 0.7s and in 7.61.1 (first affected version) it takes 6.7s for a localhost sftp server.

@manfredsc

This comment has been minimized.

Copy link

commented Mar 4, 2019

Same for me, 7.61.1 is way slower than version 7.61.0 when doing a simple sftp listing.
The trace-ascii output is exactly the same for both versions.

It seems curl-7.61.1 does exactly one line of print-out per second as one can see with --trace-time:

14:45:01.186936 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    3 0        root         4096 Jun 12  2014 dev.
14:45:02.187983 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Jun 12  2014 lib.
14:45:03.189041 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Oct  7  2014 bin.
14:45:04.190077 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Jun 12  2014 tmp.

So there is somewhere an artificial 1 second delay per printed line! Hope this rings some bells...
Please tell how to further narrow this annoying bug. Thanks.

@bagder

This comment has been minimized.

Copy link
Member

commented Mar 4, 2019

I can reproduce. Fix pending.

bagder added a commit that referenced this issue Mar 4, 2019

ssh: loop the state machine if not done and not blocking
If the state machine isn't complete, didn't fail and it didn't return
due to blocking it can just as well loop again.

This addresses the problem with SFTP directory listings where we would
otherwise return back to the parent and as the multi state machine
doesn't have any code for using CURLM_CALL_MULTI_PERFORM for as long the
doing phase isn't complete, it would return out when in reality there
was more data to deal with.

Fixes #3506
@manfredsc

This comment has been minimized.

Copy link

commented Mar 4, 2019

Works for me, version 7.64.0 plus this patch.
Thanks!

@bagder

This comment has been minimized.

Copy link
Member

commented Mar 4, 2019

Thanks for confirming!

@bagder bagder closed this in d6f73af Mar 5, 2019

@lock lock bot locked as resolved and limited conversation to collaborators Jun 3, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.