Lost messages when sender's server does not support s2s stream management

[haansn08]

I am hosting an XMPP server (Prosody) from my home since February 2022.

I have noticed that every now and then some of my friends' messages get lost without any error indication, which makes using XMPP infeasible for our everyday communications, since the sender cannot be sure that messages will actually be delivered. Looking at my server logs and also having talked to the other server's admins, no suspicious log entries can be found. However seemingly it only happens for contacts on ejabberd servers (like conversations.im) that do not support S2S Stream Management (XEP-0198).

I'm don't really understand why SM is needed if XMPP is based on TCP (if there's no TCP ACK then the server should know that a message wasn't delivered, right?), but looking at the ejabberd FAQ message loss still can happen "in wifi or 3G networks" and this problem has been solved by XEP-0198. My router connects to the internet over LTE, and I suspect if this is a problem for c2s connections, this can happen for s2s connections too.

Are there any plans to implement SM for s2s connections? Prosody already implements SM, but just a few contacts being on ejabberd servers, makes hosting XMPP from your home (a very important use case for privacy) infeasible, if you agree that lost messages are terrible for communication.

[licaon-kter]

Says it's supported here:

ejabberd/src/mod_stream_mgmt.erl

Line 26 in f20e9e9

-protocol({xep, 198, '1.5.2'}).

https://docs.ejabberd.im/admin/configuration/modules/#mod-stream-mgmt

92% of servers do: https://compliance.conversations.im/test/xep0198/

Your only reason to ask for this is a FAQ from 2015? Well...

[haansn08]

You are talking about c2s stream management support. My question is about s2s stream management, which is equally important if the server is hosted from a home internet connection.

Your only reason to ask for this is a FAQ from 2015?

The reason I ask is because I didn't receive some important messages from some of my contacts and I want to find out why.

[licaon-kter]

My question is about s2s stream management,

Which XEP is that exactly? Link to the 0198 section?

The reason I ask is because I didn't receive some important messages from some of my contacts and I want to find out why.

Great, so the client logs say what? Their server logs? More info pls

[haansn08]

Note: Stream Management can be used for server-to-server streams as well as for client-to-server streams. However, for convenience this specification discusses client-to-server streams only. The same principles apply to server-to-server streams.

XEP-0198: Stream Management -- Introduction

Great, so the client logs say what? Their server logs?

As already mentioned, no warnings, errors, or suspicious log entries were found around the timestamp of the messages. We know for certain, that the message was received by the other server, since carbons were send to my contact's other devices.

[licaon-kter]

XEP-0198: Stream Management -- Introduction

Link to the server TO server stream management section, please.

We know for certain, that the message was received by the other server, since carbons were send to my contact's other devices.

Wait, what? So it's not a server TO server issue but a CLIENT to server one?

Which client software get the message and which not?

Does their user account have MAM enabled?

Were both clients ONLINE at that moment?

[haansn08]

Link to the server TO server stream management section, please.

Please read the Introduction to XEP-0198 carefully. It mentions server-to-server streams in the last paragraph.

Wait, what? So it's not a server TO server issue but a CLIENT to server one?

No. Because carbons were sent to my friend's other devices we can rule out a c2s problem on his side.

[licaon-kter]

Carbons are a proof that the message has reached THEIR server, if A CLIENT did nor receive the message then it's a CLIENT issue.

Messages are not sent from YOUR server separately for EACH client your contact has. It's one message ONLY, and THEIR server should distribute it.

To me this sounds like a misconfigured server for your contact. Like it has no MAM support, or that user did not enable MAM or some client error.

Again. not a server to server issue.

Please answer my questions:

Which client software got the message and which not?
Does their user account have MAM enabled?
Were both clients of your contact ONLINE at that moment?

https://prosody.im/doc/modules/mod_smacks

Right, experimental. Interesting. @mwild1 does this sound like a SM issue to you?

[haansn08]

To me this sounds like a misconfigured server for your contact.

• One server in question is conversations.im and my contact's client is the Play Store version of Conversations.

• Another contact where this keeps happening is on magicbroccoli.de, and he uses F-Droid Conversations and Gajim.

[weiss]

There's a PR, but we unfortunately never got round to bringing it into a mergeable state. It would probably be nice to have. However, I'm not sure it's the proper solution to achieve reliability. If the sender receives no acknowledgement, the sender won't know whether the message or just the acknowledgement got lost. So if the sender now returns an error message, that might be misleading; if the sender resends the message, it might be a duplicate.

In my book, the real stream management feature is to allow for efficient resumption of c2s sessions (as opposed to reliabilty: for c2s, this is achieved with MAM support). Efficient session resumption is a non-issue with s2s.

Long story short: Right now, XMPP basically relies on stable s2s links. As properly hosted servers usually have stable links, this issue doesn't pop up much in practice.

[haansn08]

Is there a better way to achieve reliabilty for unstable s2s links? I think many people getting into XMPP right now use it for it's privacy properties and high degree of autonomy. Hosting an XMPP server from your home internet connection (as opposed to VPS, where somebody else has access to your system) should be supported I think. Also knowing the protocol can handle unstable s2s links without having to fear message loss would be nice...

[weiss]

Clients such as Conversations or Gajim will show check marks if the peer's client returned a message delivery receipt. So in the absense of such a mark, you cannot be sure your contact received the message. That's about as far as you get with the current specs/implementations, I'm afraid.

Hosting an XMPP server from your home internet connection (as opposed to VPS, where somebody else has access to your system) should be supported I think.

I'm not sure home Internet connections are necessarily unreliable. Anyway, I agree it would be nice to support that case as well, but the two of us agreeing on nice things won't magically make them happen I'm afraid.

So yes, for use cases where protecting against an attacker with access to your VPS (but not to the network between your server and your peer's server/client) is a requirement and E2E encryption won't do the trick, I'd not necessarily recommend XMPP.

[licaon-kter]

@haansn08 step by step, now the other questions :)

[haansn08]

To me this sounds like a misconfigured server for your contact. Like it has no MAM support, or that user did not enable MAM or some client error.

This is unlikely to be the problem if you download Conversations from Google Play and register on conversations.im.

[licaon-kter]

Ok, server side no, that's fine. Now client side, ask them to check, accounts - account - upper right menu - Archiving. Is it Always?

[haansn08]

Without asking, yes. They did not change any default settings for archiving.

[licaon-kter]

@haansn08 The default is: Never afaik

[haansn08]

My friends just confirmed to me it is set to 'Always'.