use libssh2_scp_recv2 to support > 2GB files on windows over SFTP

[dbyron0]

Now that libssh2/libssh2#31 is merged to master of libssh2, here is the little bit left to use it in curl. I imagine we need to release a version of libssh2 that contains libssh2_scp_recv2 before it's safe to merge this, but I figured I'd get the ball rolling a bit.

Tested by receiving a > 2GB file on OS X 10.10.5 using a scp:// url. I could use a hand testing this on windows as I'm still struggling to build curl against a hand-built libssh2 there.

Also, these results from make check on OS X 10.10.5:

TESTDONE: 864 tests out of 867 reported OK: 99%
TESTFAIL: These test cases failed: 20 507 534 
TESTDONE: 994 tests were considered during 409 seconds.

though note the same failures occurred on master.

I used this test config:

********* System characteristics ******** 
* curl 7.45.0-DEV (x86_64-apple-darwin14.5.0) 
* libcurl/7.45.0-DEV OpenSSL/0.9.8zd zlib/1.2.5 libssh2/1.6.1_DEV
* Features: Debug TrackMemory IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets 
* Host: David-Byrons-MacBook-Pro.local
* System: Darwin David-Byrons-MacBook-Pro.local 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64
* Server SSL:        OFF  libcurl SSL:  ON 
* debug build:       ON   track memory: ON 
* valgrind:          OFF  HTTP IPv6     ON 
* HTTP Unix     ON 
* FTP IPv6           ON   Libtool lib:  OFF
* Shared build:      yes  Resolver:     stock
* SSL library:       OpenSSL
* Ports:
*   HTTP/8990 FTP/8992 FTP2/8995 RTSP/9007 
*   TFTP/8997 HTTP-IPv6/8994 RTSP-IPv6/9008 FTP-IPv6/8996 
*   GOPHER/9009 GOPHER-IPv6/9009
*   SSH/8999 SOCKS/9000 POP3/9001 IMAP/9003 SMTP/9005
*   POP3-IPv6/9002 IMAP-IPv6/9004 SMTP-IPv6/9006
*   HTTP-PIPE/9014 
* Unix socket paths:
*   HTTP-Unix:http.sock
***************************************** 

[bagder]

I would prefer to have a check for the libssh2 version in use and use some #ifdef logic to make the code still work as before with the older libssh2 and use the new function only when available!

[keith-daigle]

I updated my other comment, too but figured just in case I'd put it in both places:

I was able to build and test tonight using the following setup:
VS 2012 x86 on 2008 R2
Libs:
openssl 1.0.1j
libssh2/libssh2:master
dbyron0/curl:large_file_windows

The only real special thing I did was to rename libssh2.lib to libssh2_a.lib so libcurl would be happy with linking it. It's a minor thing and, hey, I was warned about the creakiness of the curl build system. The builds went smoothly otherwise.

Everything seems to be working as expected when I use the resulting binary, I can do something like 'curl -o c:\huge-frigging.file scp://user:pass@host/file/to/transfer' and it gets me the whole file. I tested a few 4g transfers and one 7.5g transfer from linux to the aforementioned windows system and the hashes matched. I see the unit tests failed, but If you'd like something more rigorous done by hand I'll try to squeeze it in.

[dbyron0]

@bagder, like this: dbyron0@076a696 ?

[bagder]

@dbyron0 Exactly!

[bagder]

Thanks a lot for your work on this. Landed the curl side of this fix now, so let's get the libssh2 release out soon and then we can soon celebrate the end of this long journey!

[dbyron0]

Thank you. Glad it finally made it in :)

[rivy]

@bagder @dbyron0 ,
Unfortunately, I'm still seeing the same problem with the most current curl version (7.49.0). The binary was downloaded from https://bintray.com/artifact/download/vszakats/generic/curl-7.49.0-win64-mingw.7z (from the "Win64 - Generic" section of the "Download" page, built by Viktor Szakats).

C:> curl --version
curl 7.49.0 (x86_64-pc-win32) libcurl/7.49.0 OpenSSL/1.0.2h zlib/1.2.8 WinIDN libssh2/1.7.0 nghttp2/1.10.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2

As in my original issue (#329), any file greater than 2GB in size is still seen and downloaded as only 2047MB.

[jay]

I can't reproduce, although I did not try that exact version. I downloaded 4,414,115,840 Win10_1511_1_English_x64.iso via sftp and the hash matched.

SSH-2.0-5.39 FlowSsh: Bitvise SSH Server (WinSSHD) 6.47: free only for personal non-commercial use

curl 7.49.0-DEV (i686-w64-mingw32) libcurl/7.49.0-DEV mbedTLS/2.2.1 zlib/1.2.8 libidn/1.32 libssh2/1.7.0 nghttp2/1.9.2
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz HTTP2

update: tried scp as well, also fine

[rivy]

I tried SFTP, and it is working correctly. That actually solves my use case!

Thanks for the suggestion!

Unfortunately, SCP is still bugged for me (seeing and transferring only 2047MB of >2GB files). Is there any configuration information I can give you to help track it down?

[jay]

I tried that build and I'm having the same problem. IIRC there's a compile-time define that turns on the support for large files via scp only if the libssh2 version is recent enough. The client does identify as SSH-2.0-libssh2_1.7.0 but is it possible the include is for an older version? I don't know what's up here. @vszakats

[vszakats]

I've made built-time trace tests, generated and verified .map files and they didn't show problems in headers used at build-time or libraries at runtime (though possibly it might be safer to use #include "libssh2*.h" over the current #include <libssh2*.h>). It means libssh2 1.7.0 is correctly detected and used in these binaries.

Built-time trace patch: https://github.com/vszakats/harbour-deps/blob/7f3116ef7eded8aeabedec3583369de102216c36/curl.diff
Log: https://ci.appveyor.com/project/vszakats/harbour-deps/build/1.0.832#L5716
Results: https://bintray.com/vszakats/generic/curl-test/7.49.1/view#files

Not much idea what could cause large-file support to be broken with SSH/SCP. AFAICS there is no other public build for Windows, supporting that combination, to double-check it.

[vszakats]

If there is any useful patch or existing trace option that might help locating the problem, I'm ready to create further test builds with those included/enabled.

[vszakats]

According to a newly run test, large-file support is correctly enabled in libssh2 component, too:
Log: https://ci.appveyor.com/project/vszakats/harbour-deps/build/1.0.834#L5970
Patch: https://github.com/vszakats/harbour-deps/blob/d037fbc151fd03cfb1d87c882327d71e13661830/libssh2.diff

[jay]

Thanks for checking Viktor. I still have no idea. When I have some free time I'll build for x64 to see if there's any difference.

[jay]

Moved to #858, please take any further correspondence there.