s_client seems to hang with -starttls xmpp

[bowlofeggs]

When I run s_client against my xmpp server (ejabberd), it seems to hang and does not print out the certificate info that I see when I connect to other types of servers (like http):

$ openssl s_client -servername chat.example.com -connect chat.example.com:5222 -starttls xmpp -verify 100
verify depth is 100
CONNECTED(00000003)

Other TLS checkers do seem to be OK with the servers I've tried, and clients and other servers also seem to approve of the connections so I think there might be something going on in s_client's xmpp code.

I also found someone talking about this in Debian's bug tracker:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=747469#5

[richsalz]

Can you see if s_client is spinning while waiting for the server response?
There isn't an XML parser, it's looking for fixed strings and I'm guessing your server doesn't send those exact bytes.

[levitte]

Looking at the code, it seems that the client is waiting for a success response, indefinitely. Possible error responses don't seem to be considered, at all.

(side note: for the s2s protocol, the argument -starttls xmpp-server should be used)

[levitte]

Errrr... I just tested a build of master, and it worked without hickups, so errrr, going back to analysis and will open my big mouth later

[levitte]

Ok, analysis done, it would seem that this could happen if the server didn't give any response at all, even to the first XML line openssl throws at it. May I suggest, as a test, to add the command line options -debug -state? That should show you the exact conversation, like this (in the error case):

$ openssl s_client -connect xmpp-server.example.org:5269 -starttls xmpp -debug -state
CONNECTED(00000003)
write to 0xb089d0 [0x7ffd3e8f9550] (121 bytes => 121 (0x79))
0000 - 3c 73 74 72 65 61 6d 3a-73 74 72 65 61 6d 20 78   <stream:stream x
0010 - 6d 6c 6e 73 3a 73 74 72-65 61 6d 3d 27 68 74 74   mlns:stream='htt
0020 - 70 3a 2f 2f 65 74 68 65-72 78 2e 6a 61 62 62 65   p://etherx.jabbe
0030 - 72 2e 6f 72 67 2f 73 74-72 65 61 6d 73 27 20 78   r.org/streams' x
0040 - 6d 6c 6e 73 3d 27 6a 61-62 62 65 72 3a 63 6c 69   mlns='jabber:cli
0050 - 65 6e 74 27 20 74 6f 3d-27 78 6d 70 70 2e 6f 70   ent' to='xmpp.op
0060 - 65 6e 73 73 6c 2e 6f 72-67 27 20 76 65 72 73 69   enssl.org' versi
0070 - 6f 6e 3d 27 31 2e 30 27-3e                        on='1.0'>
read from 0xb089d0 [0xaf7dc0] (8192 bytes => 93 (0x5D))
0000 - 3c 73 74 72 65 61 6d 3a-65 72 72 6f 72 3e 3c 69   <stream:error><i
0010 - 6e 76 61 6c 69 64 2d 6e-61 6d 65 73 70 61 63 65   nvalid-namespace
0020 - 20 78 6d 6c 6e 73 3d 27-75 72 6e 3a 69 65 74 66    xmlns='urn:ietf
0030 - 3a 70 61 72 61 6d 73 3a-78 6d 6c 3a 6e 73 3a 78   :params:xml:ns:x
0040 - 6d 70 70 2d 73 74 72 65-61 6d 73 27 2f 3e 3c 2f   mpp-streams'/></
0050 - 73 74 72 65 61 6d 3a 65-72 72 6f 72 3e            stream:error>
read from 0xb089d0 [0xaf7dc0] (8192 bytes => 0 (0x0))
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 93 bytes and written 121 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
---
$ 

[bowlofeggs]

On Thu, 2017-07-20 at 15:22 +0000, Rich Salz wrote: Can you see if s_client is spinning while waiting for the server response?

If I add a -debug on there, I see a bunch of continuous output like this: ``` read from 0x20f9170 [0x1ff2150] (8192 bytes => 0 (0x0)) read from 0x20f9170 [0x1ff2150] (8192 bytes => 0 (0x0)) read from 0x20f9170 [0x1ff2150] (8192 bytes => 0 (0x0)) ``` So yeah, I'd describe that as "spinning".

[bowlofeggs]

Oh I should have mentioned this - this is openssl-1.0.2k-1.fc24.x86_64 (from Fedora 24).

[richsalz]

Yes, that’s spinning ☺ The server sounds borked, sorry technical term.

[levitte]

Ah, 1.0.2 doesn't break out of the loop on a zero read, 1.1.0 and on does. Ok, that should be easy to fix.

[bowlofeggs]

@richsalz The server does seem to function correctly - I am able to s2s with many other servers and a variety of users are connecting with a variety of clients. However, I maaaaay be using s_client incorrectly. With that -state flag, I can see an XML error from the server:

<?xml version='1.0'?>
<stream:stream id='12007209944305194275' version='1.0' xml:lang='en' xmlns:stream='http://etherx.jabber.org/streams' from='example.com' xmlns='jabber:client'>
    <stream:error>
        <host-unknown xmlns='urn:ietf:params:xml:ns:xmpp-streams'/>
    </stream:error>
</stream:stream>

I tried to use the -servername example.com flag to invoke SNI, but maybe XMPP works some other way?

[bowlofeggs]

Oh here's another interesting thing - does s_client look up the SRV records? Because if I use -connect example.com:5222 instead of -connect chat.example.com:5222 everything works just fine. It looks like s_client is doing more than I thought it would do - I expected to just point it straight at the server that runs my XMPP stuff (which is chat. in my case). I have an SRV record that says to use chat.example.com.

[bowlofeggs]

I do also happen to have a NAT loopback on my router that would make example.com port 5222 reflect to chat.example.com, so that could explain it too.

[levitte]

The following patch fixes the issue:

diff --git a/apps/s_client.c b/apps/s_client.c
index 85c1b6b579..dc467994f8 100644
--- a/apps/s_client.c
+++ b/apps/s_client.c
@@ -1667,6 +1667,8 @@ int MAIN(int argc, char **argv)
             if (strstr(mbuf, "/stream:features>"))
                 goto shut;
             seen = BIO_read(sbio, mbuf, BUFSIZZ);
+            if (seen <= 0)
+                goto shut;
             mbuf[seen] = 0;
         }
         BIO_printf(sbio,

The reason you're successful on port 5222 is that it's the client port (i.e. c2s), which the 1.0.2 s_client speaks (and therefore doesn't get a zero read in the starttls handshake)

[bowlofeggs]

I think there could be more going on than just the spinning - it's not so much the port 5222 thing as the servername bit - should the -servername flag work for the xmpp handshake? I tried to connect to my server from a host outside my LAN and I experienced the same issue (so it's not my NAT loopback making it work). In other words, I would expect this to work but it doesn't:

$ openssl s_client -servername example.com -connect chat.example.com:5222 -starttls xmpp

This does work, but possibly only because of port forwarding on port 5222 (I don't have access to an external IPv6 host to test with at the moment):

$ openssl s_client -connect example.com:5222 -starttls xmpp

Are my expectations around the -servername flag incorrect?

[levitte]

Nope.

It's like this; with -starttls xmpp, s_client presents itself as a client, which implies c2s protocol. So, what happens when you talk to the s2s port is this:

client: <stream:stream xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client' to='chat.example.com' version='1.0'>
server: <stream:error><invalid-namespace xmlns='urn:ietf:params:xml:ns:xmpp-streams'/></stream:error>

The 1.0.2 s_client reads from the server and tries to find <starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls', and doesn't stop reading if it receives anything else. The server, on the other hand, won't send anything back after the error tag, so s_client ends up spinning around reading zero bytes. That's what my patch fixes.

-servername doesn't come into play at this stage. That's a TLS feature which comes in after TLS has been established.

[levitte]

Nowm that was for port 5269. You haven't told us what problems you're getting with port 5222.

[levitte]

As far as I can tell, the issue is fixed and the fix will appear in the next 1.0.2 letter release (1.0.2m). How that will affect your Fedora installation, I cannot tell.

[bowlofeggs]

@levitte As you noted, I was actually doing c2s on the cs2 port. The issue I think that might remain is that the client isn't telling the server what host name it wants to connect to. xmpp is like http in that a single port can handle many domain names, and the client needs to request a domain name. So if the user's name is me@example.com, but the server that handles example.com is named chat.example.com, the client has to say that they are trying to communicate with the "vhost" example.com.

This is why my server is upset - the -servername example.com flag isn't telling my server that it wants the example.com "vhost", so the server thinks that chat.example.com is being requested (which isn't a valid vhost on my server).

Fixing the spinning issue will help with the client hanging, but it still doesn't make it possible to check my server with the client since the client won't use the correct "vhost" upon connecting and the server will say that the requested host isn't valid.

What do you think?

I'd be happy to open a new issue about this if you like.

[levitte]

Like I said earlier, -servername sets the Server Name Indication at the TLS level, nothing else. OpenSSL 1.0.2 has no other way to indicate a server name. OpenSSL does have the added option -xmpphost

[bowlofeggs]

Ah, the -xmpphost option is what I've been missing. The host I've been using s_client on has an older version of openssl, but it does work on my Fedora Rawhide box with openssl-1.1.0f-7.fc27.x86_64. Thus, I am satisfied that there are no remaining problems - thanks for sharing that option with me!

[levitte]

You're welcome!