Riot fails in strange ways when indexeddb is full

[turt2live]

Description

This has lead to all kinds of fun problems:

• Syncs not being saved (When reloading develop, the last saved sync can be from 5+ days ago #6118)
• Not being able to send bug reports/logs
• Encryption being reported as disabled in rooms (see screenshot below)
• Cannot send encrypted events (likely because Riot thinks encryption is disabled)

Riot should probably fail a bit more gracefully. I didn't realize my storage was full until I went to start a report about encryption being disabled and happened to consider the case of indexeddb being full. The app runs fine otherwise - messages can be sent and received, features work, etc. The only problems are those described above.

Version information

• Platform: web (in-browser)
• Browser: Chrome 64
• OS: Windows 10
• URL: riot.im/develop

[turt2live]

For those that may be running into this, clean up your primary drive. Chrome, for example, sets the maximum quota to ~6% of your available free space.

[lampholder]

As per #6125 (which I'm going to close as a dup) we need to warn loudly when this happens as a bare minimum. It would be nice also to tell people what they can do to address the problem (and ultimately even nicer make it so they don't encounter this problem by effectively managing the size of indexeddb.

[lukebarnard1]

A useful resource is Offline Storage for Progressive Web Apps, where it compares browsers and their quotas:

Browser	Limit
Chrome	<6% of free space
Firefox	<10% of free space
Safari	<50MB
IE10	<250MB
Edge	Dependent on volume size

Browsers do different things in response to the limit being exceeded:

Firefox

In Firefox, apps can use up to 10% of free disk space, but will prompt the user for further storage requests after 50MB data stored.

If the user denies expansion above 50MB, QuotaExceededError is thrown and could be used to show a dialog that says something like "Riot has exceeded your browser's quota limits".

When quotas are exceeded (or the user denies expansion), data is evicted:

Browser	Eviction Policy
Chrome	LRU once Chrome runs out of space
Firefox	LRU if the whole disk gets full
Safari	No eviction
Edge	No eviction

Chrome

In @turt2live's case, Chrome probably evicted the riot.im origin, removing all data at once from under the app.

Solution

We will need to move over to a different storage API: Storage. This will allow us to:

1. request "persistent" instead of "best-effort" storage, that won't be removed without the user's consent;
2. request an estimate of the usage and quota for the app.

We can use 1. to prevent the data disappearing underneath the app without the user's consent. We can use 2. to periodically indicate to the user that storage is low.

[turt2live]

Today's failure mode was silently failing to sync. Took nearly 3 hours before I realized that my client wasn't syncing.

[grahamperrin]

Thanks,

… We will need to move over to a different storage API: Storage. …

Is a future move – to Storage Standard – certain?

[hrj]

E2E has suddenly stopped working for me since yesterday.

Unable to decrypt: The sender's device has not sent us the keys for this message. 

There was no change in browser versions or anything else on my system, but the free space on my disk did change drastically yesterday. So it is probably related to this issue.

To reclaim space, which tables of the Local storage / Indexed DB can we delete safely? I don't want to lose my device keys for sure. I assume log entries can be deleted without affecting functionality.

[ara4n]

can you rageshake so we can see whether the issue actually is indexeddb filling up? the error you've quoted there could be related to other issues (e.g. failing to upload one-time keys to your server or something).

the safest bet is to export your E2E keys via settings, and then either relogin to entirely flush the storage, or you could experiment with manually deleting the logs and matrix-js-sdk:riot-web-sync tables from indexeddb. Not sure how gracefully the app would recover from the tables from disappearing under it, though, and the chances are relatively high that it will just panic and re-login.

[jryans]

#9220 tracks this problem as a meta-issue of several separate tasks. We have now implemented some of the ideas here and tracked the remaining ones in new issues, so I think we can close this.