Local messages store

[vladimiry]

This is considered as a priority one missed major feature:

• Supported by app email providers get DDoSed quite often especially in the recent time. So you can't send or get new messages, but it would be nice to at least have access to your already locally stored messages.
• You don't want to be vendor locked. You want to at any time be able to at least process your messages in your own, like for example exporting all your messages into the unencrypted .eml format.
• Having email messages stored locally we could enhance email provider's features by adding not yet supported things like for example full-text search for ProtonMail or batch export.

There potentially might be many email messages content getting options, like at least:

• Using Service Workers. Considered as a too specific and limited approach.
• Listening for email view actions happening in the email provider's UI, also very limited option. The good part is that it would work in a fully passive mode, not producing addition requests to the email provider's backend. But in JavaScript it's not always possible to listen/intercept all the needed actions happening, especially if the code is implemented with an intention to prevent interception happening. Besides going with such option we can only get content of the email message that has ever been explicitly opened/viewed by a user. The simplest in implementation option though.
• Direct backend endpoints calling (Rest, WebSocket, etc). The most flexible option. Email provider's endpoints API is not yet well documented, but I believe it can be researched to a sufficient extent. The good side benefit of going with this option is that over a time getting messages logic can be moved from app code to an individual module. Like has been done with modules published here, so developers could used that module building their own programs.

At the initial stage it's not going to be a comprehensive bridge-like thing, but more like email provider's web UI supporting thing. The initial implementation is not going to keep locally stored messages in sync with the server/actual state. Means it would be a one-off putting to local cache action, with no further message state updating (message got unread state, got removed, got changed folder/label, etc).

A brief workflow steps description:

• Local messages cache can be enabled per account. Disabled by default.
• Having it enabled app starts a background messages fetching job, which starts working with some interval, let's say each 60 minutes. Besides if there is a rate limiter put on top of the endpoints by a backend, then app delays individual requests accordingly.
• Producing a fetching request to the backend, job adds at least lastFetchedEmailItemCreateTime-like, portion size and sort order parameters, depending on the email provider's API endpoints format. That lastFetchedEmailItemCreateTime parameter would be added as a portion start request parameter. So app doesn't fetch the same messages twice. If we go with fetching all the messages during each job iteration, we can keep locally cached messages in sync with the server/actual state, but a more optimal approach would be in going with message state changing queue. So app as a client gets notified about some message state change action happened and patches locally saved message accordingly, but this can't be implemented having no control on the backend, but looks like at least Tutanota has it implemented already (see EntityEventBatchTypeRef entities fetching).
• The local store would be built using some encrypted embedded SQLite-like database. Encryption key would be automatically generated and stored in the already used encrypted settings.bin file.
• Email providers differ in supported features, but at this stage I'd prefer to keep a single set of local messages store table columns. So it would be a single table with a compound primary key=(id, emailProviderType = "protonmail | tutanota", login). App would also store in a raw form an original message blob provided by email provider backend.
• If needed, getting and storing attachments can be added later, as an individual background job.

[vladimiry]

I was exploring existing Node.js compatible embedded databases with built-in encryption support and have not really found solutions that would keep metadata encrypted as well. Means, it's possible to encrypt values of specific columns/fields, but metadata remains unencrypted. Metadata is for example information like how many rows in the database you have, what is the columns set, empty/filled cells, etc. But such information in some case can also be considered as sensitive.

So I'm considering the following approach. App keeps all the messages in memory. App flushes these messages to the encrypted file with some interval and probably on some triggers. This file would be a brick of bytes, fully encrypted, including metadata, similar to how settings.bin is being currently saved (it's a very small file though). When app starts, it loads the saved file into a memory and process continues as described above. At this stage, attachments are not going to be saved, and so it's not going to be a very memory consuming thing. Later it would be possible to introduce an encrypted binary file storage and fetch files into it in a scheduled manner.

Database encryption key would be generated once and stored in settings.bin. A feature of changing this key can be added later in along with re-encrypting the database with a new key. The encryption algorithm is going to be AES 256 CBC with randomly generated IV on every file saving.

[vladimiry]

I was exploring Tutanota's entity event batch stream and have realized how to efficiently utilize it in app. So it's going to be a complete offline emails storage, including the state syncing part. Messages fetching and syncing is already pushed into the master branch.

[vladimiry]

Encrypted database persistence has just landed into the master branch. So backend logic is completely done for Tutanota and it works efficiently enough. Which means building minimal UI around the local messages store is only remaining blocker for releasing the feature. Alternatively, I'm considering releasing the feature only after the Protonmail also gets backend logic done. I've not even yet started researching Protonmail's Rest API though.

[vladimiry]

I think before jumping into the UI building I will add contacts syncing. So emails, folders, and contacts will be stored in the local database.

[vladimiry]

Contacts storing is now in the master branch, in addition to the mails and folders.

[vladimiry]

Here is an animated demo of the work in progress "local cache" feature:

• You can see that app is reactive. Means it reacts to the mails/folders/contacts changing and applies the respective patch to the local cache (local database), and it does it efficiently.
• Enabling "local store" in the account setting adds the database/online toggle button near to the account handle. The button has two states, database view, and online view:
  
• It's hard to notice on a small amount of data, but when the actual syncing is happening the view toggle button is blinking in green color - an indication of the data fetching and database patch forming activity.
  

This is a WIP demo, a feature is not yet ready for release, even in the beta channel, but the app version shown on the gif animation is in master branch already. Initially, this feature is going to be available only for the Tutanota account types.

[vladimiry]

I was exploring the possibility to enable conversation view mode for local store viewer, which is in general a demanded and not yet supported feature by the official Web Client, ref1 / ref2. And it looks like a quite possible thing to do with still remaining issues to sort out though (see the weird, 2nd-level, grayed out removed message):

Conversation shown on the example screenshot is 3 level hierarchy conversation. Hierarchical tree-like displaying is possible because Tutanota keeps the conversation structure using nodes with parent/previous reference model.

[vladimiry]

v2.0.0-beta.1 release has been published.

[vladimiry]

Feature got enabled for protonmail accounts with v2.0.0-beta.2 release.

[vladimiry]

Emails export feature has been enabled with v2.0.0-beta.4 release.

[vladimiry]

Closing the issue as resolved. The final v2 release is going to be published soon. There will be new issues open if I decide to start working on the email attachments syncing/exporting or search in database view mode features.

[vladimiry]

If needed, getting and storing attachments can be added later, as an individual background job.

At this stage, attachments are not going to be saved, and so it's not going to be a very memory consuming thing. Later it would be possible to introduce an encrypted binary file storage and fetch files into it in a scheduled manner.

v4.7.0 is going to optionally enable the live attachments loading during the export process. So full batch export will be happening, including the attachments content.