Support sharing encrypted Realms across processes

[jpsim]

This was previously impossible when encryption used mach signaling, but should now be feasible.

[jpsim]

Cocoa issue: realm/realm-swift#1693

[jpsim]

This is marked as P1. Is that an accurate reflection of its prioritization?

[ianyh]

So is this, in fact, feasible?

[ianyh]

To clarify, I am willing to put development effort into this, but would rather not spend the time if it's going to turn out not to be feasible.

[bmunkholm]

@finnschiermer Can you comment on what's required to support this?

[tgoyne]

The problem for multiprocess is knowing when to discard the in-memory decrypted data for each page. Currently a page which is read by one process will not be reread by that process following a change to it from a different process and then bad things happen.

One idea I have to make this work would be to adjust the allocator to never reuse space on a page until that entire page is freed up, and then make advancing the read transaction mark all pages which contain a freelist entry as out of data. There may be problems with this beyond that it would increase disk usage.

[ianyh]

What information does a process have about changes from another process? Is there a technical limitation that prevents the process from rereading relevant pages? Is it a performance limitation? Is there just not enough granularity in the information to be able to selectively read pages?

[finnschiermer]

@ianyh @bmunkholm We have not designed a mechanism specifically for informing a reader of a change made in another process. I think @tgoyne is right in principle that such a mechanism can be based on monitoring our free-list. We could compare the new free-list against the previous versions free-list to detect allocations, then invalidate (to later trigger decryption of) all touched pages. Since the free-lists of earlier versions are needed to do this, it runs counter to a desire to cut older versions free-lists to minimize file-size growth problems. Also, since the free-list itself is encrypted some care is needed when walking it, but it should be doable. We'll also have to be careful not to duplicate the work, when multiple readers grab new versions - some synchronization will be needed there.

[finnschiermer]

It think it is easier to decrypt all pages involved in walking from the top-ref to all leafs, but not the leafs themselves. The leafs should only be marked invalid in order to trigger later decryption as needed. If we added version numbers to interior nodes we could prune the walk even more.

[finnschiermer]

@tgoyne Hmm. I have a feeling I did not understand your suggestion completely. Would you care to elaborate?

[ianyh]

@finnschiermer If I understand correctly, I believe @tgoyne's idea is to prevent the reuse of allocated storage if doing so would change the data underlying existing in-memory decrypted data. Any newly written data would have no existing in-memory decrypted representation in any process, and thus get decrypted correctly by every process. Presumably, the mechanism for encryption already drops its in-memory decrypted data when a page is entirely reclaimed?

That certainly does seem like it would increase disk usage. I can't think of first-order effects that would impact performance. Either way this would only impact encrypted databases. Would it be too much to require a database to be manually configured as encrypted and shareable between processes before this mechanism kicks in? It would isolate the effect on disk usage, but would increase the complexity of configuration.

[ianyh]

Also, thinking more about constraints, are things significantly simplified if we assume only one of the connections to the db can write?

[ianyh]

@finnschiermer @tgoyne Okay, thinking through more after having familiarized myself with more of the code. Would changes in the free-list be enough to determine out of date pages? The thing I am still grappling with is what happens when a page with data is freed and then allocated again from one process. How would another process distinguish that from the page being unchanged? That could theoretically result in no change to the free-list, right?

[beseder42]

Can you estimate in which time horizon this will be implemented? We need it strongly and would have to make lots of work arounds otherwise. Thanks!

[beseder42]

Would really appreciate any progress on this!

Addition
This is such a needed feature for security reasons!! How come this is not brought to production for so long and now on p2 instead of p1? How can you are promote Realm to be first choice for secure apps but there is no way to work with extensions on an encrypted Realm?
Too sad Realm is only focusing on the paid server features lately and has no more focus on local database features 🙁 Will look for other database for future apps!!

[joshuawright11]

Any update on this? This would be super important to our company.

[ianyh]

I haven’t had the bandwidth to work on this, unfortunately. We came up with a workaround for our specific situation. I don’t know what the status is otherwise.

[finnschiermer]

The status is unchanged. We don't have the bandwidth to work on it at the moment.

[Sarubbaru]

any news? it's very important

[genamendola17]

has anyone found a workaround?

[Cesare85]

Any updates in this regard? This would be extremely important for our company.

[bmunkholm]

Sorry to say - but this is not something we are currently prioritizing. When we do we will surely update here.

[Sarubbaru]

@bmunkholm do you have some workarounds? because I have thousands of crash and I don't know what to do, could u help me please?

[bmunkholm]

Don't use encryption is the best I can say. Consider what you really need to encrypt. Encrypt it yourself and store it in a binary field (but it won't be searchable).

[XabierGoros]

Thanks a lot for the help @bmunkholm.

We've been using Realm with a great result in our production apps for a while. However, this issue comes from 2015 and since we need to share information between different processes we find this issue a big stopper.

Don't use encryption is the best I can say. Consider what you really need to encrypt. Encrypt it yourself and store it in a binary field (but it won't be searchable).

I wouldn't say this is an acceptable workaround or a neat solution for every developer who is claiming this change. Encrypting and decrypting specific data on each access by ourselves would add an extra overhead in time and business complexity when the database itself should already support it as its API states. Once mach signalling wasn't used any more, would it be possible to add this feature to Realm?

[bmunkholm]

@XabierGoros Totally understood. This is a limitation we would love to get rid of. Unfortunately we continue to have other features we judge are more important for more people. But it's surely a feature that's hovering close in the top list of things we want to get to whenever we get the capacity. I'm sorry that's not so useful for you right now though.

[andersfredlund]

And now two years later, still no bandwidth to look into this?

[mohachouch]

Hello,

We want to use realm as part of a large project for a large company in France.

Unfortunately, due to security constraints, we have the obligation to encrypt the database.

I had the sad news to learn that realm does not support accessing an Encrypted Realm from Multiple Processes.

These two features are essential for us. I have seen that this feautre has been made multiple times for 5 years but has not yet developed. realm/realm-swift#1693 realm/realm-java#7667

It's too late for our project because we start it in January but I think you should prioritize this task because a lot of users are moving away from Realm because of it.

Thank you for the work done by your teams.

[bmunkholm]

@andersfredlund @mohachouch We highly appreciate your feedback and can tell you that we are now actively working on this (#5786) and hope to be able to release this soon.

[mohachouch]

@bmunkholm Thank you very much for your quick response. I hope the functionality will arrive in time. (I don't want to use sqllite)

[RomainPct]

Thanks for this important update for the whole community @bmunkholm. Is there an expected release date ?

[bmunkholm]

We would love to be able to tell you, but we can't really promise anything as you know things can happen, bugs needs fixed, people get sick and the list goes on. I would have hoped this was done already, so I'm hopeful we can finish in Jan. But can't promise anything, so don't depend on it.