OMEMO - older messages are lost #2241
Comments
licaon-kter
changed the title
|
I had a look at this, but didn't find the fix yet. It's not necessary to log out in order to reproduce this, you can just leave and rejoin the MUC. The problem appears to be related to browser storage. When messages are fetched after rejoining the MUC, the models array is empty (as if there weren't any messages stored), although I can see them in localStorage via the developer console. Looks like somehow (probably when the MUC gets closed) the messages collection gets persisted to localStorage in such a way that it wipes the references to the saved messages. Upon rejoin, messages are recreated from MAM, (which AFAIK doesn't provide enough info to decrypt them, if they were sent by us). |
|
Reopen please, some messages are still mangled after a while. |
|
Sorry, but that's too vague. |
|
Which part is vague? "a while"? It's because I wasn't paying attention, can't specify how much time has passed. "mangled"? One message has transformed into |
|
I can repro at will, it appears that the last MUC message gets axed, some sort of off-by-one? then the last one gets mangled. Console says: and and These tend to keep repeating for some reason albeit there's no MUC activity Hope these help. |
|
Thank you, that helps a lot. I think it's an off-by-one error, like you suspected, which causes the last cached message to be returned from MAM and the MAM entry then overwrites the decrypted plaintext. |
|
It looks like the bug fix for #1481 has some relation to this as well. If I close and then re-enter a group now, I'm seeing the error message from EDIT: Never mind, that is resolved if I set |
|
@afriedmanGlacier: Can you paste the traceback of the error here? |
|
Sure...do you need debug included? I set log.js:64 ERROR: Error Could not find JID to decrypt OMEMO message for |
|
You are only able to decrypt OMEMO encrypted messages once. The double ratchet prevents subsequent decryptions. So if you set This is mentioned in the documentation for that setting: https://conversejs.org/docs/html/configuration.html#muc-clear-messages-on-leave That's the one issue. The other one: This can happen if a user's nickname has changed. The archived messages are fetched, and they still have the old nickname, so Converse can't find the JID based on the nickname. XEP-0421 provides a way to fix this, by adding persistent occupant IDs to MUC message and presence stanzas. Do you know whether your server supports it? Newer versions of Prosody do. I'm in the process of adding support for XEP-0421. |
This let's us populate the `from_real_jid` attribute for messages in cases where the user's nickname has changed. Updates #2241
This let's us populate the `from_real_jid` attribute for messages in cases where the user's nickname has changed. Only save the occupant-id if the MUC supports it Store all advertised features on the `chatbox.features` model. This allows us to look up a feature without using the async `disco.supports` API. Updates #2241
|
@afriedmanGlacier: I've now merged the branch which adds support for XEP-0421. If your XMPP server supports it, then Converse should now be able to decrypt messages whose nicknames don't match any current users (as long as those messages weren't decrypted already, as explained above). |
|
We use ejabberd, and it does not appear to support that XEP yet: processone/ejabberd#3397 @weiss do you know if there are plans for this or if it's been implemented yet? |
I'm not aware of such plans. I agree it would be nice to have, if that helps! 😃 (Wondering about this use case though: Converse.js supports OMEMO in anonymous groups? Isn't the value of end-to-end encryption questionable if you don't know who the other end is?) |
Oh, or maybe it's just using the room JID rather than the real JID even in non-anonymous groups? |
|
@weiss: It's not that the group is anonymous, it's that If a user's nickname changes, then MAM messages might be received from the old nickname and without XEP-0421 the client doesn't have a way to look up the real JID of the sender and so the message can't be decrypted. |
|
Right, totally makes sense to me now. Thanks (and sorry for side tracking the topic)! |
|
Adding on to @licaon-kter @afriedmanGlacier; getting these random messages that are not able to be decrypted in 1:1 messages as well. In the screenshot below, the 3 messages I sent at 6:21 PM were all within a minute. One of these messages was unable to be decrypted. The contents of this particular message was "Sure".
I'm not showing any relevant errors in debug that I can tell. It also seems that these only appear when I scroll back up in history. I also think they only appear when you "reopen" the app. My setup is one Android user and one Converse user (MAM). |
|
Using HEAD I don't see the exact same error, but the whole thing is unstable is general. Eg. Messages sent while Converse was not connected are almost surely not decrypted. Note: no key change, no IndexDB clean up. The old messages (already decrypted) are shown fine though. |
|
@licaon-kter you said "Messages sent while Converse was not connected are almost surely not decrypted." Is that only in MUC or in 1:1 messages also? We see that same issue in MUC, but it generally doesn't seem to happen if the sending device/client is connected when we re-open Converse. One cause seems to be that the MUC occupant nickname that is linked to the jid is not populated until the user is online or some other event, so when MAM/offline messages come in, it didn't have a way to identify the contact or real_jid. I know "presence" is one thing that populates the nickname/jid (which is why it works in this particular case when the other device is online), but that shouldn't be necessary. In testing with other contacts, sometimes the nickname was stored and we didn't have this issue. When entering a group, there is a "fetchMembers" function that queries for group affiliations/occupants. According to the xmpp specs, the server CAN return group nickname with this query, but in our case it doesn't. |
|
1:1 testing for me |
Ok, so I studied the code, and I think what you're describing might be an issue. When a MAM message is received, we don't get the MUC occupant's real JID in the MAM message stanza, only the nickname (and the XEP-0421 occupant id if the XMPP supports it, but your's doesn't). In order to decrypt a message, we need the sender's real JID, which means we need a way to look up the real JID from the MUC nickname. That's why, when Converse joins a MUC, it first calls Then, as a MAM messages come in, it gets parsed and it's crucial that the However, if the members list that's returned by the XMPP server doesn't return nicks, then @afriedmanGlacier Can you provide an example of a members list IQ response from your XMPP server? I'm not sure what can be done if the members list that's returned doesn't include nicknames or XEP-0421 occupant ids. I wonder what other XMPP clients do about this issue. I think the problem can be mitigated/solved by storing occupant data indefinitely. @afriedmanGlacier Are you using localStorage or IndexedDB as your store? Converse does keep the occupant data between logins, as long as you don't keep the MUC open. If you leave it via the UI (by clicking the dropdown and then This last point seems to me to also be causing a lot of confusion among people. One potential solution there is to not delete the data when you leave a MUC. With localStorage this isn't really an option, due to the storage limit, but IndexedDB doesn't have a limit. |
|
@iNPUTmice informed me that for a non-anon MUC, you do get real JIDs in MAM results. So I have a fix for the above problem in the works. |
That solves the problem of not being able to look up OMEMO session data from incoming MAM messages. See here: #2241 (comment) Updates #2241
|
@licaon-kter Could you please check if things are better now? |
|
@jcbrand sorry, was offline over the weekend. We are using IndexedDB. Do you still need me to provide an example of a members list IQ response? Looks like you may have figured out the solution without that, so we'll try what you've done first. Thanks! |
|
I updated from your commit and I'm still having the decryption issue with groups when I reopen Converse if messages came in groups while I was disconnected. I'll try to verify what's happening and send a log later today or tomorrow. Also, while the fix you made in theory should help what I'm seeing with group messages, I'm guessing it's not the same issue for @licaon-kter since that's 1:1? |
|
After looking at this further, it seems like the issue I'm referencing is probably different than what @licaon-kter and some of the guys that I work with have seen. I'm guessing I should create a separate issue for what I'm seeing? In the case of this original ticket, the messages are decrypted upon arrival but later on after reconnecting a message can be "lost" and changed to not decrypted. What I'm seeing is that messages that come in to a MUC (and sometimes 1:1) while the client is closed don't get decrypted upon opening the client and doing a MAM fetch. Just to be clear, we are using a version of converse-desktop that has been updated to 8.0.1 plus most of the latest commits. So it's possible that the web client doesn't have the issue I'm seeing because it seems like it could be a timing issue. After the update for getting sender real JID, we are correctly receiving the JID, but the decryption still often fails when opening the app for messages that arrived while closed. In looking at the message attributes during the decryption process, I noticed that it fails with no warning in In the attrs: However, the correct |
|
@licaon-kter Can this ticket be closed? |
|
Closing in favour of #2733 |
HEAD (0a7dff4), Firefox 80, using IndexDB
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Older messages are available
Actual behavior
All past messages (both from me and my contacts) are replaced with
I sent you an OMEMO encrypted message but your client doesn’t seem to support that. Find more information on https://conversations.im/omemoMoreover in 1:1, all the messages are deleted, only the last one is kept, but it's actually the string above.
Sometimes some messages (eg. pulled from MAM if Converse was offline) are not corrupted, but on the next login they'll be.
Note: key is NOT regenerated on login.
Related to: #2235
The text was updated successfully, but these errors were encountered: