curl: (7) Failure initializing sftp session: sftp_new: Out of memory

[linuxchris12]

I did this

Upgraded from 28 to Fedora 29
then running
curl sftp://someuser:somepw@url/

I get the error:
curl: (7) Failure initializing sftp session: sftp_new: Out of memory
and the trace:

== Info:   Trying xxx.xxx.xxx.xxx...
== Info: TCP_NODELAY set
== Info: Connected to xxx port 22 (#0)
== Info: User: someuser
== Info: Authentication using SSH public key file
== Info: completed keyboard interactive authentication
== Info: Authentication complete
== Info: Failure initializing sftp session: sftp_new: Out of memory
== Info: Closing connection 0

this was working fine under Fedora 28 and earlier for some years for me.

I expected the following

get the contents of the sftp directory

curl/libcurl version

curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1 zlib/1.2.11 brotli/1.0.5 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.8.5/openssl/zlib nghttp2/1.34.0
Release-Date: 2018-09-05
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink

operating system

Fedora 29

I also tried

use sftp client to directly connect to the site, which is working
use curl to connect to a ftp site, which is working as well

[bagder]

That seems to be libssh's sftp_new() function that returns an error here:

curl/lib/ssh-libssh.c

Line 809 in 27a5e1f

sshc->sftp_session = sftp_new(sshc->ssh_session);

I can't tell why this would suddenly fail for you! Does SFTP work for you if you try different URLs or does it always fail?

/cc @nmav

[linuxchris12]

I tried this public host:
curl sftp://demo:password@test.rebex.net:22
and its working here:

`* Rebuilt URL to: sftp://demo:password@test.rebex.net:22/

• Trying 195.144.107.198...
• TCP_NODELAY set
• Connected to test.rebex.net (195.144.107.198) port 22 (#0)
• User: demo
• Authentication using SSH public key file
• completed keyboard interactive authentication
• Authentication complete
  drwx------ 2 demo users 0 Jul 26 2017 .
  drwx------ 2 demo users 0 Jul 26 2017 ..
  drwx------ 2 demo users 0 Dec 03 2015 aspnet_client
  drwx------ 2 demo users 0 Oct 27 2015 pub
  -rw------- 1 demo users 403 Apr 08 2014 readme.txt
• Connection #0 to host test.rebex.net left intact`

I compared the encryption methods:
the failing server is:
Algorithm: ECDH, Curve25519, Hash: SHA-256
ssh-rsa 2048
Client-to-server cipher AES-256 GCM
Client-to-server MAC: AES-256 GCM
Server-to-client cipher: AES-256 GCM
Server to client MAC: AES-256 GCM

the working server is:
Algorithm: ECDH, Curve25519, Hash: SHA-256
ssh-rsa 2048
Client-to-server cipher AES-256 SDCTR
Client-to-server MAC: HMAC-SHA-256
Server-to-client cipher: AES-256 SDCTR
Server to client MAC: HMAC-SHA-256

that is just a difference I noticed when using filezilla to connect to the servers (which is working fine for both) I don't know enough about that to see if that can be the reason.

[nmav]

@ansasaki any idea?

[linuxchris12]

I have taken the git repository and made a couple of builds, found the following:

with libssh, not working:
curl 7.63.0-DEV (x86_64-pc-linux-gnu) libcurl/7.63.0-DEV libssh/0.8.5/openssl/zlib
Release-Date: [unreleased]
Protocols: dict file ftp gopher http imap pop3 rtsp scp sftp smtp telnet tftp
Features: AsynchDNS IPv6 Largefile UnixSockets

with libssh again, not working:
curl 7.58.0-DEV (x86_64-pc-linux-gnu) libcurl/7.58.0-DEV libssh/0.8.5/openssl/zlib
Release-Date: [unreleased]
Protocols: dict file ftp gopher http imap pop3 rtsp scp sftp smtp telnet tftp
Features: AsynchDNS IPv6 Largefile UnixSockets

with libssh2, working:
curl 7.54.1-DEV (x86_64-pc-linux-gnu) libcurl/7.54.1-DEV OpenSSL/1.1.1 zlib/1.2.11 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy

So could be related to the change from libss2 to libssh.
I will try with a custom build recent libssh version later on.

[ansasaki]

Hi @linuxchris12,

I am investigating this in libssh side, but I can't reproduce the issue.

Can you try applying this commit to make libssh verbose and run again?

This would help me a lot to find the root of the problem.

[linuxchris12]

Hi @ansasaki

sorry for my lag. Here is the log output using your commit:

with_verbosity.log

[ansasaki]

@linuxchris12, thanks for the log!
Looking for the root of your problem, we found out that errors set by channel_* functions were being overwritten in sftp_new() with an OOM error. This was fixed in this commit:

https://git.libssh.org/projects/libssh.git/commit/?id=3784226fd85bc1256ef927640f4d400348da038f

This still don't explain your problem, but at least clarify the received message. Some error is returned by the server after the "sftp" subsystem is requested (in channel_request()). I'm still investigating this and would appreciate any help in understanding better your scenario.

[linuxchris12]

@ansasaki thanks for the info and the good work.
If I can be of any help, let me know.

I can setup a test-account for you at the server for which I am getting this error, if that would help.
Just let me know how I can send you the credentials.

[ansasaki]

@ansasaki thanks for the info and the good work.
If I can be of any help, let me know.

I can setup a test-account for you at the server for which I am getting this error, if that would help.
Just let me know how I can send you the credentials.

That would be great, thanks! You can send me the credentials by email to ansasaki@redhat.com

[linuxchris12]

@ansasaki thanks for the info and the good work.
If I can be of any help, let me know.
I can setup a test-account for you at the server for which I am getting this error, if that would help.
Just let me know how I can send you the credentials.

That would be great, thanks! You can send me the credentials by email to ansasaki@redhat.com

I have sent you the login.

[ansasaki]

That would be great, thanks! You can send me the credentials by email to ansasaki@redhat.com

I have sent you the login.

Thanks for setting up the server. I tested yesterday the current libssh master and I was able to successfully authenticate, although nothing was present in the directory (same behaviour with sftp; perhaps wrong permissions?)

Can you check if it works for you?

[linuxchris12]

that is strange. Did you try the curl command that I have sent you? That is definitely failing with the above error for the server for me. I have changed the permissions and put a file there now.

If I try to use the sftp client, it is working for me as well:

sftp sftp://xxx@yyy/ Password: Connected to yyy. sftp> ls with_verbosity.log sftp>

[ansasaki]

@linuxchris12 Thanks for your help!

I believe I (re-)found an old issue: when a hostname is provided and it resolves to both IPv6 address and IPv4 address, it tries to connect only to the first one. If it fails, the connection is aborted.

There was a comment in the libssh code warning that it could be a problem, and it really is. I tried to connect to the IPv4 address the hostname you provided resolves to and it was successful. But the IPv6 address it resolves to gives me the EHOSTUNREACH error. I tried to ping to the address and it gave me the same error.

This combined to the fact that all sftp errors were being overwritten with an OOM error gives you the behaviour you described.

Other thing is that curl correctly handles this (tries the IPv6 address first and, if it fails, tries the IPv4 address) and successfully opens a socket, but this socket is not provided to libssh. Then libssh tries to open a new socket and fails with the above behaviour. So, the messages curls logs in verbose mode:

* Connected to <hostname> (<IPv4 address>) port 22 (#<connection #>)

... is about this first socket curl opens.

I've opened an issue for libssh (https://bugs.libssh.org/T127)

[ansasaki]

@linuxchris12 There was one more issue, and probably the most important (and I guess that it is the real problem): a regression introduced by CVE-2018-10933 fix.

Basically in this fix, the messages are filtered depending on the state of the ssh session. The filter was filtering out the SSH_MSG_EXT_INFO message, if it arrived after the client is already authenticated.

This message is used when negotiating the keys (key exchange). The problem is that the server can request a re-exchange to set up new keys for the session, and then send this message after the client is authenticated, which would lead to an error.

I fixed this in this commit: https://gitlab.com/ansasaki/libssh-mirror/commit/8efb7ae348dd9f82440867b86c90f40959c6d738

It will be reviewed and hopefully merged soon. The issue I mentioned in the previous comment is still there, but apparently it only happens to me, due to a custom DNS configuration.

[linuxchris12]

@ansasaki Great!

I can confirm it works for me now. I have taken this commit:
e4e51ccc1340e313c203842d0180a1c4e33c95cc
of libssh and just added your fix
https://gitlab.com/ansasaki/libssh-mirror/commit/8efb7ae348dd9f82440867b86c90f40959c6d738

it is now working with both the self-built curl configured with --with-libssh=/...
and the curl just at is is shipped by fedora 29 if I put the fixed libssh.so.4.7.2 into /usr/lib64 as it should

Good job.
I did not check the other fix though.
Will close this now. I will let the test-server running for some time and then shut it down if the patch was reviewed.