use libssh2_scp_recv2 to support > 2GB files on windows over SFTP #451

wants to merge 1 commit into


None yet

6 participants

dbyron0 commented Sep 22, 2015

Now that libssh2/libssh2#31 is merged to master of libssh2, here is the little bit left to use it in curl. I imagine we need to release a version of libssh2 that contains libssh2_scp_recv2 before it's safe to merge this, but I figured I'd get the ball rolling a bit.

Tested by receiving a > 2GB file on OS X 10.10.5 using a scp:// url. I could use a hand testing this on windows as I'm still struggling to build curl against a hand-built libssh2 there.

Also, these results from make check on OS X 10.10.5:

TESTDONE: 864 tests out of 867 reported OK: 99%
TESTFAIL: These test cases failed: 20 507 534 
TESTDONE: 994 tests were considered during 409 seconds.

though note the same failures occurred on master.

I used this test config:

********* System characteristics ******** 
* curl 7.45.0-DEV (x86_64-apple-darwin14.5.0) 
* libcurl/7.45.0-DEV OpenSSL/0.9.8zd zlib/1.2.5 libssh2/1.6.1_DEV
* Features: Debug TrackMemory IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets 
* Host: David-Byrons-MacBook-Pro.local
* System: Darwin David-Byrons-MacBook-Pro.local 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64
* Server SSL:        OFF  libcurl SSL:  ON 
* debug build:       ON   track memory: ON 
* valgrind:          OFF  HTTP IPv6     ON 
* HTTP Unix     ON 
* FTP IPv6           ON   Libtool lib:  OFF
* Shared build:      yes  Resolver:     stock
* SSL library:       OpenSSL
* Ports:
*   HTTP/8990 FTP/8992 FTP2/8995 RTSP/9007 
*   TFTP/8997 HTTP-IPv6/8994 RTSP-IPv6/9008 FTP-IPv6/8996 
*   GOPHER/9009 GOPHER-IPv6/9009
*   SSH/8999 SOCKS/9000 POP3/9001 IMAP/9003 SMTP/9005
*   POP3-IPv6/9002 IMAP-IPv6/9004 SMTP-IPv6/9006
*   HTTP-PIPE/9014 
* Unix socket paths:
*   HTTP-Unix:http.sock
bagder commented Sep 23, 2015

I would prefer to have a check for the libssh2 version in use and use some #ifdef logic to make the code still work as before with the older libssh2 and use the new function only when available!

@bagder bagder self-assigned this Sep 23, 2015

I updated my other comment, too but figured just in case I'd put it in both places:

I was able to build and test tonight using the following setup:
VS 2012 x86 on 2008 R2
openssl 1.0.1j

The only real special thing I did was to rename libssh2.lib to libssh2_a.lib so libcurl would be happy with linking it. It's a minor thing and, hey, I was warned about the creakiness of the curl build system. The builds went smoothly otherwise.

Everything seems to be working as expected when I use the resulting binary, I can do something like 'curl -o c:\huge-frigging.file scp://user:pass@host/file/to/transfer' and it gets me the whole file. I tested a few 4g transfers and one 7.5g transfer from linux to the aforementioned windows system and the hashes matched. I see the unit tests failed, but If you'd like something more rigorous done by hand I'll try to squeeze it in.

dbyron0 commented Sep 26, 2015

@bagder, like this: dbyron0@076a696 ?

bagder commented Sep 26, 2015

@dbyron0 Exactly!

@bagder bagder changed the title from use libssh2_scp_recv2 to support > 2GB files on windows to use libssh2_scp_recv2 to support > 2GB files on windows over SFTP Jan 21, 2016
@bagder bagder added a commit that closed this pull request Feb 14, 2016
@dbyron0 @bagder dbyron0 + bagder SCP: use libssh2_scp_recv2 to support > 2GB files on windows
libssh2_scp_recv2 is introduced in libssh2 1.7.0 - to be released "any
day now.

Closes #451
@bagder bagder closed this in cae21ff Feb 14, 2016
bagder commented Feb 14, 2016

Thanks a lot for your work on this. Landed the curl side of this fix now, so let's get the libssh2 release out soon and then we can soon celebrate the end of this long journey!

dbyron0 commented Feb 14, 2016

Thank you. Glad it finally made it in :)

rivy commented May 29, 2016 edited

@bagder @dbyron0 ,
Unfortunately, I'm still seeing the same problem with the most current curl version (7.49.0). The binary was downloaded from (from the "Win64 - Generic" section of the "Download" page, built by Viktor Szakats).

C:> curl --version
curl 7.49.0 (x86_64-pc-win32) libcurl/7.49.0 OpenSSL/1.0.2h zlib/1.2.8 WinIDN libssh2/1.7.0 nghttp2/1.10.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2

As in my original issue (#329), any file greater than 2GB in size is still seen and downloaded as only 2047MB.

jay commented May 29, 2016 edited

I can't reproduce, although I did not try that exact version. I downloaded 4,414,115,840 Win10_1511_1_English_x64.iso via sftp and the hash matched.

SSH-2.0-5.39 FlowSsh: Bitvise SSH Server (WinSSHD) 6.47: free only for personal non-commercial use

curl 7.49.0-DEV (i686-w64-mingw32) libcurl/7.49.0-DEV mbedTLS/2.2.1 zlib/1.2.8 libidn/1.32 libssh2/1.7.0 nghttp2/1.9.2
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz HTTP2

update: tried scp as well, also fine

rivy commented Jun 1, 2016

I tried SFTP, and it is working correctly. That actually solves my use case!

Thanks for the suggestion!

Unfortunately, SCP is still bugged for me (seeing and transferring only 2047MB of >2GB files). Is there any configuration information I can give you to help track it down?

jay commented Jun 1, 2016

I tried that build and I'm having the same problem. IIRC there's a compile-time define that turns on the support for large files via scp only if the libssh2 version is recent enough. The client does identify as SSH-2.0-libssh2_1.7.0 but is it possible the include is for an older version? I don't know what's up here. @vszakats

vszakats commented Jun 1, 2016

I've made built-time trace tests, generated and verified .map files and they didn't show problems in headers used at build-time or libraries at runtime (though possibly it might be safer to use #include "libssh2*.h" over the current #include <libssh2*.h>). It means libssh2 1.7.0 is correctly detected and used in these binaries.

Built-time trace patch:

Not much idea what could cause large-file support to be broken with SSH/SCP. AFAICS there is no other public build for Windows, supporting that combination, to double-check it.

vszakats commented Jun 1, 2016

If there is any useful patch or existing trace option that might help locating the problem, I'm ready to create further test builds with those included/enabled.

vszakats commented Jun 3, 2016

According to a newly run test, large-file support is correctly enabled in libssh2 component, too:

jay commented Jun 4, 2016

Thanks for checking Viktor. I still have no idea. When I have some free time I'll build for x64 to see if there's any difference.

jay commented Jun 4, 2016

Moved to #858, please take any further correspondence there.

@jay jay locked and limited conversation to collaborators Jun 4, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.