Older OMEMO messages become null on login

[licaon-kter]

HEAD (4dedfdb), Firefox 80, using IndexDB

To Reproduce
Steps to reproduce the behavior:

1. login
2. send or receive messages in 1:1 or MUC, with OMEMO
3. logout
4. login

Expected behavior
Older messages are available

Actual behavior
All past messages (both from me and my contacts) are replaced with null
Moreover in 1:1, all the messages are deleted, only the last one is kept, but it's actually the string null
Sometimes some messages are not corrupted, but on the next login they'll be.

OMEMO MUC, 2 console messages: first, second
OMEMO 1:1, 2 console messages: first, second

[licaon-kter]

In OMEMO MUC, after several logins/outs some messages disappear fully, a bunch of:
"Sorry, could not decrypt a received OMEMO because we don't have the JID for that user."
...now stay in their place

But, this is a private chat created with Conversations so there's a clear message on join: This groupchat is not anonymous which contradicts it.

Then again my testing users are online but not in the muc at the moment., yet the participants list shows them.

[licaon-kter]

Please reopen.

Now they change to: I sent you an OMEMO encrypted message but your client doesn’t seem to support that. Find more information on https://conversations.im/omemo so not really fixed :(

[jcbrand]

It is fixed.

The fix is a bit technical to explain, but you have duplicate encrypted messages in your IndexedDB storage.

The bugfix is to properly identify a (possibly existing) locally cached and decrypted message for any incoming encrypted MAM message. This wasn't happening before, so an encrypted duplicate was created and the fact that null was shown was a side-effect bug.

You'll notice that the fallback message doesn't show any more with newer messages, only for the older ones that had null before them.

[licaon-kter]

I don't understand, so replacing my old normal messages with the placeholder is a feature?

I'll retry...

[jcbrand]

There weren't any messages replaced. The messages that showed null and now the placeholder were duplicates of messages that you already had.

[licaon-kter]

Are we lost in translation?

I'll try again: on re-login the chat had ONLY placeholder messages at the correct time for the correct contacts. There are no duplicate messages, nor did I say there where.

[jcbrand]

I tried to explain the underlying issue I fixed to you. I believe it's the same bug that you noticed originally, because it fixed messages appearing as null after logging out and logging back in again.

The fact that you now only see placeholder messages might because you no longer had the decrypted local copies or the keys to decrypt the ones coming from MAM. I'm not sure, but AFAIK this bug is fixed.

[licaon-kter]

I see, so i'll have to open a new issue with "old messages" get corrupted then.

Fun fact, MAM ones (eg. Leave muc, write from other members, open MUC) show up fine, just the already existing ones get lost.