Skip to content

Slow loading of directory content via SFTP when timeout is set #3506

Closed
Closed
Slow loading of directory content via SFTP when timeout is set#3506
Labels
SCP/SFTP

Description

@DiceMaster
DiceMaster
opened on Jan 30, 2019

Directory listing is wery slow when connecting to a SFTP server with CURLOPT_TIMEOUT or -m cmd option. Without a timeout everything works fast.

Tested on my Mac (macos 10.14.2) when connecting to other Mac with curl v7.64.0 (or with daily snapshot 20190129) with following command:
curl sftp://<host>/ -m 20 -u <user>

There is some delay for loading of each subdirectory/file.
In curl v7.61.0 the same command performs almost instantly.

After some investigation I found that the bug was introduced with changes to lib/ssh.c in commit 1a89099

Activity

  1. bagder
    added
    SCP/SFTP
    on Jan 30, 2019

  2. koskokos2 commented on Jan 31, 2019

    @koskokos2
    koskokos2
    on Jan 31, 2019

    Noticed the very same problem and for me it works very slow even without any additional arguments just:
    curl sftp://<auth>@<host>/

    In version 7.61.0 that command takes (for a folder with 4 entries) 0.7s and in 7.61.1 (first affected version) it takes 6.7s for a localhost sftp server.

  3. manfredsc commented on Mar 4, 2019

    @manfredsc
    manfredsc
    on Mar 4, 2019

    Same for me, 7.61.1 is way slower than version 7.61.0 when doing a simple sftp listing.
    The trace-ascii output is exactly the same for both versions.

    It seems curl-7.61.1 does exactly one line of print-out per second as one can see with --trace-time:

    14:45:01.186936 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    3 0        root         4096 Jun 12  2014 dev.
14:45:02.187983 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Jun 12  2014 lib.
14:45:03.189041 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Oct  7  2014 bin.
14:45:04.190077 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Jun 12  2014 tmp.

    So there is somewhere an artificial 1 second delay per printed line! Hope this rings some bells...
    Please tell how to further narrow this annoying bug. Thanks.

  4. bagder commented on Mar 4, 2019

    @bagder
    bagder
    on Mar 4, 2019
    Member

    I can reproduce. Fix pending.

  5. bagder
    added a commit that references this issue on Mar 4, 2019

    ssh: loop the state machine if not done and not blocking

    d9af8f2

  6. manfredsc commented on Mar 4, 2019

    @manfredsc
    manfredsc
    on Mar 4, 2019

    Works for me, version 7.64.0 plus this patch.
    Thanks!

  7. bagder commented on Mar 4, 2019

    @bagder
    bagder
    on Mar 4, 2019
    Member

    Thanks for confirming!

  8. bagder
    closed this as completedin d6f73afon Mar 5, 2019
  9. lock
    locked as resolved and limited conversation to collaborators on Jun 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    SCP/SFTP

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions