Unable To Decrypt meta issue

[BillCarsonFr]

Unable to decrypt Epic Issue

Meta issue relating to all "Unable to Decrypt" problems, a.k.a "Waiting for this message, this may take a while".

Needed information to resolve a reported issue

In order to properly debug an Unable To Decrypt error, we need logs from the receiver of the message (the one seeing the issue) and those from the sender. We can't debug issues without logs from both sides.

In your bug report, please identify which event you can't read. Either include the event ID, or something like "message from Dave at 10:10".

How to send rageshake from Element Web:

• Click on your avatar in top left part of the screen
• Select "All Settings"
• Select "Help & About"
• Then click on the submit debug logs button

How to send rageshake from Element Android:

• Tap on the top right button on the home screen
• Select Report Bug

How to send rageshake from element iOS:

• Open the top left drawer
• Select the Feedback button at the bottom

---

Causes of Unable to Decrypt errors

🟢: We believe that this will be fixed with Element R
🕛: Affects a feature which is not yet supported in Element R

We categorize the main sources of UISI errors as follow:

Client Issues

Sender Side

• 🟢 The sender failed to send the keys. There’s no consistent retry logic on network error, so key sending is fragile. Queue and retry to-device msgs, so E2EE works in bad networks or with unreliable HS #673. Test: TestClientRetriesSendToDevice
• Memberlist out of sync in fresh DM element-web#20962
• 🟢 The keys may be sent slowly:
  • batching of m.room_key to-device messages is suboptimal element-web#24680 Test: complement-crypto#34
  • m.room_key_request responses are prioritised above regular m.room_key messages element-web#24681
• 🕛 Sharing historical room keys on invite is slow and should prioritise recent messages Keyshare-on-invite should start with most-recently-used sessions element-web#23778
• We abandon claiming OTKs from remote servers after 10s, and never retry, permenantly breaking E2EE to users on those servers. element-web#24138
• Failures when sharing outbound session keys are swallowed element-web#23792
• Fallback keys are cycled too quickly matrix-org/matrix-rust-sdk#3127
• NSE process can encrypt events which can cause UTDs due to key reuse element-ios#7751
• OneTime key already exists error matrix-org/matrix-rust-sdk#1415
• Element-web makes invalid assumptions about room membership changes with lazy-loading element-web#27285
• After reloading, my own failed-to-send message is UTD element-web#27334
• Olm sessions are vulnerable to wedging on OS crash or power failure, leading to UTDs matrix-org/matrix-rust-sdk#3354
• Message encrypted with different session from what is sent to recipients #2425
• Delayed membership responses in /sync cause UTDs:
  • Delayed membership responses in /sync cause UTDs matrix-org/matrix-rust-sdk#3622
  • Delayed membership responses in /sync cause UTDs  matrix-org/matrix-js-sdk#4291

Receiver Side

• 🟢 incoming encrypted to-device messages can be lost when the application is restarted #762 Test: complement-crypto#37
  • If you restart (e.g. upgrade) Element while it's waiting to process a m.room_key, it'll drop it and you'll get UISIs element-web#23113
• 🟢 Rolling "Waiting for that message" bug (Last message always in UTD) element-android#5905 (No test since no specific testcase is known - closed because Rust is sufficiently different)
• Olm session seems to change in storage element-web#24450
• 🟢 To-device messages may be lost if /keys/query fails element-web#24682 Test: complement-crypto#38
• Device not receiving to-device messages element-x-android#2520
• Broken olm: Olm sessions sometimes get out of sync, resulting in undecryptable messages.
  • 🟢 Failed to decrypt Olm message with InvalidMAC(MacError) element-ios#7479 (Cannot reproduce, so no test)
  • Decryption failure caused one-time-key to be discarded element-ios#7480
  • 🟢 to-device messages can be processed out-of-order, causing dropped keys and decryption errors element-web#25723 Test: complement-crypto#35
  • Element X iOS: Olm sessions can wedge when using multiple Clients (e.g with NSE process) matrix-org/matrix-rust-sdk#3110
  • We can throw away one-time keys which are still published on the server, or have messages in flight #2356
  • Olm sessions getting wedged #2310
• Decrypting messages shows IndexedDB transaction failure element-web#14174
• Unable to decrypt cause: new device, and there were no active devices to receive keys #2421
• Unwedging doesn't work if wedged session is new, and first one created for device matrix-org/matrix-rust-sdk#3427
• Expected UTDs: Threaded replies to messages sent before you joined an encrypted room are incorrectly shown as UTD element-web#27577

Server Issues

• To-device messages can take a long time to get sent over federation
  • To-device messages delayed by _DestinationWakeupQueue matrix-org/synapse#15161
  • Synapse does not attempt to send events in the device outbox at startup synapse#16680
• To-device messages may get lost on the server
  • to_device message didn't get delivered matrix-org/synapse#9533
  • to-device message went missing matrix-org/synapse#15335
  • Running under sqlite, Synapse incorrectly populates the to-device messages current stream ID matrix-org/synapse#16681
• bugfix: correctly tell clients when the fallback key has been used matrix-org/sliding-sync#390
• Synapse drops received federated to-device messages if it cannot talk to worker processes synapse#17117
• E2E Device lists can get out of sync with the devices actually present in a room. #2411: E2E Device lists can get out of sync with the devices actually present in a room, causing keys not to be sent. This can happen for various reasons: see the linked issue.
• Rolling a homeserver's database back via backup could cause duplicate OTKs and hence UISIs #2155
• /sync and /members do not return "current state" synapse#16940
• /sync incorrectly calculates state changes for non-gappy syncs with lazy-loading synapse#17050

Key Backups

• Successful download from key backup, but 0 keys returned matrix-org/matrix-rust-sdk#3197
• Key backups: some backups have incorrect MAC causing UTDs on login #2338
• Desynchronized Secret Storage #2322

Protocol Issues

• A client has been offline for too long and the senders have run out of one time keys. This will be addressed by the proposal to maintain fallback keys.
• /sync API does not tell clients when the server's view of state changes outside the timeline matrix-org/matrix-spec#1209
• Room events arrive faster than to-device messages during federation connectivity problems, causing decryption failures matrix-org/matrix-spec#1123
• One-time-key upload/claim is racy matrix-org/matrix-spec#1124
• Important/urgent to-device messages (eg room keys) can get blocked behind big queues of less important ones matrix-org/matrix-spec#1659
• Users whose servers were unreachable will receive undecryptable messages due to failed OTK claim #2154
• Users who join an encrypted room at the same time as a message is sent may receive a UTD (join/send race) #2268
• Rotating the current device keys will break encryption and cause UTDs #2374

Missing features

• Users joining a private room within a space via "restricted" join rule have no way of accessing (encrypted) history #646
• If all devices are logged out there will be no end to encrypt to. This will be addressed by the proposal for dehydrated devices. Device dehydration v2 #922

UX

Expected UTD

• Some room history (pre-invite for example), should never be decryptable by you. This kind of history should probably be hidden or displayed differently.
• Expected UTDs: Hide device-relative historical UTDs #2313: New devices cannot decrypt existing history until they have access to key backup

User Config

• The UX when "Never send encrypted messages to unverified sessions from this session" is enabled is bad and can cause unexpected UTDs #2450

[BillCarsonFr]

Interesting related blog post https://blog.neko.dev/posts/unable-to-decrypt-matrix.html

[jittygitty]

@BillCarsonFr Wow what a great write-up you found at https://blog.neko.dev/posts/unable-to-decrypt-matrix.html

Please correct me if I'm wrong, but having looked at that, it seems that just about all of those errors could be resolved via wizard/options provided to the user by the client which could try to resend encryption key with help of server or to even create a new one for the channel/room. I don't see any 'technical' issue/blocker other than currently missing functionality client-side to prompt user for permission to, with the help of in-between server, do what's necessary to fix room encryption/decryption.

If there's a 'security' issue with properly re-identifying the correct user, for example to re-provide keys to the other user while they've gone offline so they can pick them up from server when they are online again, can't we have a simple "challenge/response" ie question/answer to "pick-up" the keys? (I'm trying to avoid creation of a "new" room/channel and instead fix one already created but giving errors of unable to decrypt. The sender's device has not sent us the keys for this message.)

It seems the reason certain steps aren't by default taken to automatically fix such issues, is that it could be a security risk in certain situations. But if the "creator" of a room gives needed permission, it seems in a sense trivial to resolve such decryption/key errors etc. Or am I wrong and missed something?

And to the author of:
https://blog.neko.dev/posts/unable-to-decrypt-matrix.html
Many thanks!

[BillCarsonFr]

Please correct me if I'm wrong, but having looked at that, it seems that just about all of those errors could be resolved via wizard/options provided to the user by the client which could try to resend encryption key

FTR, there used to be such UI, but it was prone to social attack and annoying in the UI.
It's something explored, but it's a bit hard to have a fix all encryption problems button. We are trying to go to the bottom of the root cause for distribution failure by moving to the rust sdk

[jittygitty]

@BillCarsonFr Ah ok guess I'm relatively new so didn't know of the older UI. Personally, I'd be ok with the "social attack" risks and UI annoyance versus the embarrassment of inviting users to my new chat and getting decryption errors.

My concern was that it may be impossible to fix the root for all cases of decryption issues, especially if some are due to security measures which may need to be over-ridden by user consent in order to fix, which brings us back to that UI annoyance.

But regardless, it's great to hear the underlying sdk is being improved to hopefully eliminate or greatly reduce these issues. I had heard of rust in conduit (I run go-dendrite) but didn't know Kotlin SDK is being all redone in rust, is that right?

Anyway, thanks again to everyone working on this! (I look forward to some beta-testing with the new sdk when its ready for that.)

[anon8675309]

The link to send debug logs does not appear on Element Web on my server. Did this UI change, or is there some option that an admin needs to enable to get that to show up?

[t3chguy]

@anon8675309 your config.json must have the URL to send debug logs to, like the example https://github.com/vector-im/element-web/blob/develop/config.sample.json#L25

[anon8675309]

Can you confirm that this is still the case? I have the bug_report_endpoint_url entry from the sample and the link to send debug logs does not appear. If it's working as expected for you, I'll set up a new server and open a new ticket with the minimal steps to reproduce. (I searched for a report of this issue and didn't find anything, but I'll do so again before opening a new issue).

[hieronymousch]

Was able to send feedback from Element desktop... and have this issue with one single user on my home server, both users on the same server. Upgraded the room to version 10 and got the error again after not even 10 messages

[Ezwen]

Hi there, I have been encountering this problem a lot recently. I've already sent logs from element-web and element-android as the person who received encrypted messages that cannot be decrypted, but I could not yet send logs as somehow sending such problematic messages.

Question: when this situation happens in a matrix room, where a given user ends up only sending messages that cannot be decrypted by other room members, is there any (even intricate) known workaround? The only "workaround" I used so far was to upgrade the room to a newer room version, which solves the issue by creating a new room, but I can't really call this a proper solution… and right now I have this problem on a room that is already using version 10…

[lousando]

@Ezwen I've found that asking sender of messages to run /discardsession usually fixes any messages moving forward. Though it does not solve the messages with the encryption issue.

[BillCarsonFr]

Created a new issue that could cause UTDs #2374

[richvdh]

2024-03-18: Element Desktop still doesn't have rust crypto enabled by default yet, but I will update this issue when it is.
2024-04-11: We are beginning to roll out to Element Desktop incrementally.

Element Desktop received Rust crypto for new sessions back in February, for what it's worth. Rollout of migration for existing sessions is tracked at element-hq/element-web#27001.

[richvdh]

Updates on recent progress here:

• We found and fixed a cause of broken Olm sessions on Element X iOS. You don't have to be using Element X iOS yourself to notice UTDs in this scenario: it causes problems on the sender and recipient side alike.
• We found and fixed a bug which caused Element X to upload broken key backups.
• We also found a bug in the Dart SDK which caused clients based on that SDK to upload broken key backups. Thanks to @krille-chan and team for fixing it.
• We rolled out an update to Element X and Element Web R which identifies events that were sent before you logged in, and shows the resultant UTD with a clearer error message.
• We found and fixed a bug which could cause Synapse to drop to-device messages (potentially containing message keys) during periods of high load.
• We found and fixed a bug in the Rust matrix-sdk-crypto (so affecting Element Web R and all mobile Element clients) which could cause it to fail to download key backups (thus preventing access to messages sent before you logged in).
• We found and fixed a bug in Synapse which could cause it to give incorrect membership state to clients, which would mean senders not sharing keys with all members of the room. (It is known, however, that there are more bugs in this area).

[penyuan]

Updates on recent progress here:

...

Just want to say a bit thank you to @richvdh for this progress update. ❤️ There is still MUCH that needs to be fixed in terms of bugs and UI/UX when it comes to encryption/decryption weirdness, and so many people I introduce to Element still get bounced off because of these problems. But at least getting these progress updates is helpful!

[kegsay]

Recently I've been giving updates for this on This Week in Matrix. If you fail to decrypt a message please:

• send a bug report to us and mention:
  • the event ID which failed to decrypt,
  • whether it was 1 or many events which failed to decrypt,
  • if many events, are they all from different people?
• ask the sender to also send a bug report mentioning the event ID

We often need both sides of the conversation to fix the issue.

It would also be helpful for us if you can opt-in to analytics, as that feeds into our graphs which plot UTDs in aggregate. The general trend of the past few months has thankfully been fewer UTDs across clients that opt-in, but there is more work to be done here.

[yennor]

I usually got that kind of problem when me or the peer beeing in an area with bad mobile phone connection. With bad I mean really bad. You can get disconnected from the network for several minutes all the time, randomly get connected again for a few seconds, or beeing connected, but almost no data gets through.
I haven't been there (rural area of Colombia) since last year and won't for a few months. So So I can't tell if the situation improved with the new clients.
But maybe for your test-suite, simulate random tcp package drops (very high percentage) with high RTT (Several seconds, sometimes I measured up to 20 seconds. around 5-8 seconds is normal). And sent a few thousand messages there and back.

[jittygitty]

Does matrix-org/matrix-js-sdk#454 already "implement" solution for: #647 ?
And if so how would my friend request keys from me (on Desktop Element) on a new fresh install of his?

Are we able now to send decryption keys to whomever we wish?

[t3chguy]

@jittygitty yes but Rust Crypto does not support that and that is what Element uses, so the js-sdk PR is unrelated

[jittygitty]

@t3chguy That's good news, so I guess if I wanted to make use of this feature, I would have to install the "Web" version client of element chat on my webserver? (I currently run Dendrite already, but with mobile and Desktop clients only.)

Or is there another client app that uses js-sdk? (Element Desktop does not use it?) Or will rust crypto gain it soon?

I use another account on 'matrix.org' using desktop+mobile, for that acct. we can simply login at chat.element.io and if I'm logged in via my other devices, I can send keys to my web chat.element.io and resend to friend who lost his? thx

[t3chguy]

Element Desktop = Element Web + Electron. Element Web only supports Rust Crypto at this time. As for other matrix-js-sdk consumers I suggest finding a place discussing Matrix rather than Element specifically.

[jittygitty]

Apologies, indeed, seems I'm confused as to what matrix SDK is used by what "client", and I had not noticed the matrix-org/matrix-js-sdk#454 was outside of element-hq repositories. So, I guess you are saying that likely none of the clients under https://github.com/element-hq are able to leverage that matrix-js-sdk pull 454?

Is there a place I can find all such pertinent information on the various clients available? If anyone can point me in the right direction with a link would be appreciated. Otherwise, I'll try some search engine lookups to dig for such info. thx

[mpeter50]

It seems Cinny uses the matrix-js-sdk, and maybe there are more, but I havent found another client that does.
Then, its an other question if Cinny makes use of this feature of the SDK. You could ask about that in Cinny's support room.

[richvdh]

Another round-up of recent updates. (Most of these have already been reported in TWIM, but I think it's handy to have a record here too.)

• We found and fixed a cause of broken Olm sessions on Element iOS. As with the previous fix to EX iOS: you don't have to be using Element iOS yourself to notice UTDs in this scenario: it causes problems on the sender and recipient side alike.
• We also have more fixes to both Element X iOS (#2944) and Element X Android (#3050) which could cause broken Olm Sessions.
• We found and fixed a bug in matrix-rust-sdk affecting Element X.
• We added a workaround to the sliding-sync proxy which would guard against messages being lost after the database was dropped, potentially causing UTDs for Element X clients.
• We rolled out a change to Synapse and all Element clients, which will identify messages sent before you joined a room, and let you know you shouldn't expect to decrypt them.

A reminder that we're still at war with this issue, and it's incredibly helpful for people to send debug logs when they come across UTD errors.

That said, our analytics show that we are starting to make progress here: