ssh session error with win32 build of curl.exe and libcurl connecting to sftp

[consulion]

Hi,
we are unable to establish an sftp connection with public key authentication using the current version of curl/libcurl (8.2.1).
curl.exe from the curl website works, but the issue occurs when we build it ourselves with VS2022 (when using the solution in projects\Windows\VC14.30, using vcpkg and when building with nmake as described in winbuild/Readme.md) or using curl 8.2.1 (x86_64-w64-mingw32) which comes with mingw64.

Output of curl.exe looks like this:

.\curl.exe --verbose -u test: --key key.pem sftp://192.168.100.16

processing: sftp://192.168.100.16
Trying 192.168.100.16:22...
Connected to 192.168.100.16 (192.168.100.16) port 22
Found host 192.168.100.16 in C:\Users\test.ssh/known_hosts
Set "rsa-sha2-256,rsa-sha2-512,ssh-rsa" as SSH hostkey type
SSH MD5 public key: NULL
SSH SHA256 public key: NULL
SSH host check: 0, key: AAAA[redacted]==
SSH authentication methods available: publickey,password
Using SSH private key file 'key.pem'
SSH public key authentication failed: Would block requesting userauth list
Failure connecting to agent
Authentication failure
Closing connection
curl: (67) Authentication failure

interestingly, it doesn't matter if the key file exists at all, output stays the same.
When opening the same connection with libcurl (which is what we are trying to achieve), simply CURLE_LOGIN_DENIED is returned, the error buffer just says "Authentication failure"; i think the important bit here is SSH public key authentication failed: Would block requesting userauth list

-V output of working curl.exe from website:

curl 8.2.1 (i686-w64-mingw32) libcurl/8.2.1 OpenSSL/3.1.2 (Schannel) zlib/1.3 brotli/1.1.0 zstd/1.5.5 WinIDN libssh2/1.11.0 nghttp2/1.56.0 ngtcp2/0.19.1 nghttp3/0.15.0
Release-Date: 2023-07-26
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe UnixSockets zstd

-V output of a curl.exe that shows this behaviour:

curl 8.2.1 (i386-pc-win32) libcurl/8.2.1 OpenSSL/3.1.2 (Schannel) zlib/1.2.13 WinIDN libssh2/1.11.0
Release-Date: 2023-07-26
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe UnixSockets

Is there any way to get a version of libcurl for Windows32 that supports sftp correctly?
Could we help solving this problem somehow? Do you need any more information?

Thanks,
Ole

curl/libcurl version

curl 8.2.1

operating system

Windows 10

[jay]

Is this a timing issue? If you run each build 10x do you get the same results? Could it have something to do with the way you built libssh2? If you are using the libssh2 dll use curl --dump-module-paths to confirm the right one is being loaded.

[consulion]

Is this a timing issue? If you run each build 10x do you get the same results?

I just ran it 100 times through a script, and the error appears every time.

Could it have something to do with the way you built libssh2?

Possibly, but it happens when I link against a static libssh2 built with
cmake -DCRYPTO_BACKEND=WinCNG -DENABLE_ZLIB_COMPRESSION=ON -DZLIB_LIBRARY=C:\dev\zlib\lib\zlib.lib -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Release -A Win32 ..
as well as with curl built with vcpkg, which uses
cmake "-DCMAKE_BUILD_TYPE=Release" "-DFETCHCONTENT_FULLY_DISCONNECTED=ON" "-DBUILD_EXAMPLES=OFF" "-DBUILD_TESTING=OFF" "-DENABLE_ZLIB_COMPRESSION=ON" "-DCMAKE_DISABLE_FIND_PACKAGE_ZLIB=OFF" "-DCRYPTO_BACKEND=OpenSSL" "-DBUILD_STATIC_LIBS:BOOL=OFF" "-DBUILD_SHARED_LIBS=ON"
and with the mingw64 version as well

If you are using the libssh2 dll use curl --dump-module-paths to confirm the right one is being loaded.

Since it also happens with static linking this doesn't seem to be the issue - i checked anyway for the vcpkg version and it shows the correct dll.

Is there any pre built version of libcurl (dll or static lib) that we could use? Or can we provide any other information to facilitate finding the cause of this problem

[jay]

Thanks for that. Based on the verbose output the state when the error occurs is SSH_AUTH_PKEY:

curl/lib/vssh/libssh2.c

Lines 1158 to 1187 in a77a4a3

	case SSH_AUTH_PKEY:
	/* The function below checks if the files exists, no need to stat() here.
	*/
	rc = libssh2_userauth_publickey_fromfile_ex(sshc->ssh_session,
	conn->user,
	curlx_uztoui(
	strlen(conn->user)),
	sshc->rsa_pub,
	sshc->rsa, sshc->passphrase);
	if(rc == LIBSSH2_ERROR_EAGAIN) {
	break;
	}
	Curl_safefree(sshc->rsa_pub);
	Curl_safefree(sshc->rsa);
	if(rc == 0) {
	sshc->authed = TRUE;
	infof(data, "Initialized SSH public key authentication");
	state(data, SSH_AUTH_DONE);
	}
	else {
	char *err_msg = NULL;
	(void)libssh2_session_last_error(sshc->ssh_session,
	&err_msg, NULL, 0);
	infof(data, "SSH public key authentication failed: %s", err_msg);
	state(data, SSH_AUTH_PASS_INIT);
	rc = 0; /* clear rc and continue */
	}
	break;

Note LIBSSH2_ERROR_EAGAIN causes a break rather than show the error message SSH public key authentication failed: Would block requesting userauth list. libssh2 code has several places it sets that error message but it always then returns LIBSSH2_ERROR_EAGAIN. Either another libssh2 function is changing the return code somewhere or libcurl is pulling an old error message from the queue (in other words, a different libssh2 function failed but didn't set an error message to overwrite that one). Please try this patch so we can see the return code

diff --git a/lib/vssh/libssh2.c b/lib/vssh/libssh2.c
index 37040b4..1488739 100644
--- a/lib/vssh/libssh2.c
+++ b/lib/vssh/libssh2.c
@@ -1180,6 +1180,7 @@ static CURLcode ssh_statemach_act(struct Curl_easy *data, bool *block)
         char *err_msg = NULL;
         (void)libssh2_session_last_error(sshc->ssh_session,
                                          &err_msg, NULL, 0);
+        infof(data, "libssh2_userauth_publickey_fromfile_ex error: %d\n", rc);
         infof(data, "SSH public key authentication failed: %s", err_msg);
         state(data, SSH_AUTH_PASS_INIT);
         rc = 0; /* clear rc and continue */

I also suspect this function because you wrote earlier it doesn't matter if the key file exists or not.

[consulion]

thanks for the patch, output is:
* libssh2_userauth_publickey_fromfile_ex error: -1

[consulion]

I debugged it a little (had to remove the \n from your patch or it would throw an assert in log), and found the reason this is happening at least with libssh2 using wincng crypto backend:
in libssh2 wincng.c _libssh2_wincng_load_private, the key is expected to start with
-----BEGIN RSA PRIVATE KEY-----
or
-----BEGIN DSA PRIVATE KEY-----
and if it doesn't, it returns -1, which is the same as if the key file doesn't exist at all.
My key began with -----BEGIN OPENSSH PRIVATE KEY-----, which works nicely with curl.exe downloaded from the website, however, after converting it using ssh-keygen -m pem -f key.pem , neither my build nor the version from the website can authenticate successfully (curl: (67) Authentication failure), libssh2_userauth_publickey_fromfile_ex error: -18.

As this seems to be a libssh2 problem, should i open an issue there?

[jay]

and if it doesn't, it returns -1, which is the same as if the key file doesn't exist at all.

what is the return code of libssh2_userauth_publickey_fromfile_ex in that case?

libssh2_userauth_publickey_fromfile_ex error: -18.

what is the error message curl shows you in that case? aside from your certificate issue I'm trying to determine if we're doing something wrong in curl processing the error messages or this is a libssh2 issue with not updating the error messages

As this seems to be a libssh2 problem, should i open an issue there?

Yes, please file at libssh2 and link to this issue and the two issues below which may be related:
#4568
libssh2/libssh2#431 (since determined those crossed out issues are likely not related)

[consulion]

Sorry for the delay,

what is the return code of libssh2_userauth_publickey_fromfile_ex in that case?

if the path supplied via curl --key parameter points to an OpenSSH private key (or any file not containing -----BEGIN RSA PRIVATE KEY----- / -----BEGIN DSA PRIVATE KEY-----) or even a nonexistent path, libssh2_userauth_publickey_fromfile_ex returns -1,

what is the error message curl shows you in that case [...]

SSH public key authentication failed: Username/PublicKey combination invalid
and
curl: (67) Authentication failure

Yes, please file at libssh2 and link to this issue

first I want to try linking curl against a libssh2 with the OpenSSL crypto backend and try how that behaves, but I had linking problems, I will dig into it tomorrow - also didn't have a chance trying with curl 8.3.0 yet.

[consulion]

Update: I managed to link curl 8.3.0 against libssh2 with OpenSSL crypto backend, and this solves the problem, i can connect to sftp hosts using a private key with -----BEGIN OPENSSH PRIVATE KEY----- format; however, when supplying an RSA private key, the error message is just (curl: (67) Authentication failure), and the verbose log message says SSH public key authentication failed: Username/PublicKey combination invalid.
So as far as I understand it's a combination of two problems:

• No check if the supplied certificate exists and if it's in the correct format
• libssh2 private key authentication doesn't work using wincng crypto backend

Since it took quite some time getting past these hurdles, i don't know when I'll have time to analyze this in detail (project deadlines approaching), but if you need any more information i'd be glad to help.

[jay]

if the path supplied via curl --key parameter points to an OpenSSH private key (or any file not containing -----BEGIN RSA PRIVATE KEY----- / -----BEGIN DSA PRIVATE KEY-----) or even a nonexistent path, libssh2_userauth_publickey_fromfile_ex returns -1,

I've filed #11881 to address the incorrect error messages when libssh2_userauth_publickey_fromfile_ex returns -1.

So as far as I understand it's a combination of two problems:

• No check if the supplied certificate exists and if it's in the correct format
• libssh2 private key authentication doesn't work using wincng crypto backend

Ok. In my opinion these are libssh2 issues and you would have to file with them. #11881 is as far as I think we should take it in libcurl.

[jay]

@consulion Did you report this to libssh2?