Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sharing encrypted Realms across processes #1845

Open
jpsim opened this issue May 27, 2016 · 26 comments
Open

Support sharing encrypted Realms across processes #1845

jpsim opened this issue May 27, 2016 · 26 comments

Comments

@jpsim
Copy link
Contributor

@jpsim jpsim commented May 27, 2016

This was previously impossible when encryption used mach signaling, but should now be feasible.

@jpsim
Copy link
Contributor Author

@jpsim jpsim commented May 27, 2016

Cocoa issue: realm/realm-cocoa#1693

Loading

@jpsim
Copy link
Contributor Author

@jpsim jpsim commented Jan 5, 2017

This is marked as P1. Is that an accurate reflection of its prioritization?

Loading

@ianyh
Copy link

@ianyh ianyh commented Mar 24, 2017

So is this, in fact, feasible?

Loading

@ianyh
Copy link

@ianyh ianyh commented Mar 26, 2017

To clarify, I am willing to put development effort into this, but would rather not spend the time if it's going to turn out not to be feasible.

Loading

@bmunkholm
Copy link
Contributor

@bmunkholm bmunkholm commented Mar 27, 2017

@finnschiermer Can you comment on what's required to support this?

Loading

@tgoyne
Copy link
Member

@tgoyne tgoyne commented Mar 27, 2017

The problem for multiprocess is knowing when to discard the in-memory decrypted data for each page. Currently a page which is read by one process will not be reread by that process following a change to it from a different process and then bad things happen.

One idea I have to make this work would be to adjust the allocator to never reuse space on a page until that entire page is freed up, and then make advancing the read transaction mark all pages which contain a freelist entry as out of data. There may be problems with this beyond that it would increase disk usage.

Loading

@ianyh
Copy link

@ianyh ianyh commented Mar 27, 2017

What information does a process have about changes from another process? Is there a technical limitation that prevents the process from rereading relevant pages? Is it a performance limitation? Is there just not enough granularity in the information to be able to selectively read pages?

Loading

@finnschiermer
Copy link
Contributor

@finnschiermer finnschiermer commented Mar 28, 2017

@ianyh @bmunkholm We have not designed a mechanism specifically for informing a reader of a change made in another process. I think @tgoyne is right in principle that such a mechanism can be based on monitoring our free-list. We could compare the new free-list against the previous versions free-list to detect allocations, then invalidate (to later trigger decryption of) all touched pages. Since the free-lists of earlier versions are needed to do this, it runs counter to a desire to cut older versions free-lists to minimize file-size growth problems. Also, since the free-list itself is encrypted some care is needed when walking it, but it should be doable. We'll also have to be careful not to duplicate the work, when multiple readers grab new versions - some synchronization will be needed there.

Loading

@finnschiermer
Copy link
Contributor

@finnschiermer finnschiermer commented Mar 28, 2017

It think it is easier to decrypt all pages involved in walking from the top-ref to all leafs, but not the leafs themselves. The leafs should only be marked invalid in order to trigger later decryption as needed. If we added version numbers to interior nodes we could prune the walk even more.

Loading

@finnschiermer
Copy link
Contributor

@finnschiermer finnschiermer commented Mar 28, 2017

@tgoyne Hmm. I have a feeling I did not understand your suggestion completely. Would you care to elaborate?

Loading

@ianyh
Copy link

@ianyh ianyh commented Mar 29, 2017

@finnschiermer If I understand correctly, I believe @tgoyne's idea is to prevent the reuse of allocated storage if doing so would change the data underlying existing in-memory decrypted data. Any newly written data would have no existing in-memory decrypted representation in any process, and thus get decrypted correctly by every process. Presumably, the mechanism for encryption already drops its in-memory decrypted data when a page is entirely reclaimed?

That certainly does seem like it would increase disk usage. I can't think of first-order effects that would impact performance. Either way this would only impact encrypted databases. Would it be too much to require a database to be manually configured as encrypted and shareable between processes before this mechanism kicks in? It would isolate the effect on disk usage, but would increase the complexity of configuration.

Loading

@ianyh
Copy link

@ianyh ianyh commented Apr 4, 2017

Also, thinking more about constraints, are things significantly simplified if we assume only one of the connections to the db can write?

Loading

@ianyh
Copy link

@ianyh ianyh commented Apr 24, 2017

@finnschiermer @tgoyne Okay, thinking through more after having familiarized myself with more of the code. Would changes in the free-list be enough to determine out of date pages? The thing I am still grappling with is what happens when a page with data is freed and then allocated again from one process. How would another process distinguish that from the page being unchanged? That could theoretically result in no change to the free-list, right?

Loading

@beseder42
Copy link

@beseder42 beseder42 commented May 8, 2017

Can you estimate in which time horizon this will be implemented? We need it strongly and would have to make lots of work arounds otherwise. Thanks!

Loading

@beseder42
Copy link

@beseder42 beseder42 commented May 24, 2017

Would really appreciate any progress on this!

Addition
This is such a needed feature for security reasons!! How come this is not brought to production for so long and now on p2 instead of p1? How can you are promote Realm to be first choice for secure apps but there is no way to work with extensions on an encrypted Realm?
Too sad Realm is only focusing on the paid server features lately and has no more focus on local database features 🙁 Will look for other database for future apps!!

Loading

beeender added a commit to realm/realm-java that referenced this issue Oct 26, 2017
All Realm public APIs should be process-safe now.

Encryption is not supported for multi processes due to core
restriction realm/realm-core#1845

Accessing same Realm file in different processes from different apks
is safe, but the notifications won't work.
beeender added a commit to realm/realm-java that referenced this issue Nov 1, 2017
All Realm public APIs should be process-safe now.

Encryption is not supported for multi processes due to core
restriction realm/realm-core#1845

Accessing same Realm file in different processes from different apks
is safe, but the notifications won't work.
@roberhofer roberhofer self-assigned this Jun 13, 2018
@joshuawright11
Copy link

@joshuawright11 joshuawright11 commented Jul 24, 2018

Any update on this? This would be super important to our company.

Loading

@ianyh
Copy link

@ianyh ianyh commented Aug 3, 2018

I haven’t had the bandwidth to work on this, unfortunately. We came up with a workaround for our specific situation. I don’t know what the status is otherwise.

Loading

@finnschiermer
Copy link
Contributor

@finnschiermer finnschiermer commented Aug 3, 2018

The status is unchanged. We don't have the bandwidth to work on it at the moment.

Loading

@Sarubbaru
Copy link

@Sarubbaru Sarubbaru commented Nov 25, 2019

any news? it's very important

Loading

@genamendola17
Copy link

@genamendola17 genamendola17 commented Nov 25, 2019

has anyone found a workaround?

Loading

@Cesare85
Copy link

@Cesare85 Cesare85 commented Nov 25, 2019

Any updates in this regard? This would be extremely important for our company.

Loading

@bmunkholm
Copy link
Contributor

@bmunkholm bmunkholm commented Nov 25, 2019

Sorry to say - but this is not something we are currently prioritizing. When we do we will surely update here.

Loading

@Sarubbaru
Copy link

@Sarubbaru Sarubbaru commented Nov 25, 2019

@bmunkholm do you have some workarounds? because I have thousands of crash and I don't know what to do, could u help me please?

Loading

@bmunkholm
Copy link
Contributor

@bmunkholm bmunkholm commented Nov 25, 2019

Don't use encryption is the best I can say. Consider what you really need to encrypt. Encrypt it yourself and store it in a binary field (but it won't be searchable).

Loading

@XabierGoros
Copy link

@XabierGoros XabierGoros commented Oct 20, 2020

Thanks a lot for the help @bmunkholm.

We've been using Realm with a great result in our production apps for a while. However, this issue comes from 2015 and since we need to share information between different processes we find this issue a big stopper.

Don't use encryption is the best I can say. Consider what you really need to encrypt. Encrypt it yourself and store it in a binary field (but it won't be searchable).

I wouldn't say this is an acceptable workaround or a neat solution for every developer who is claiming this change. Encrypting and decrypting specific data on each access by ourselves would add an extra overhead in time and business complexity when the database itself should already support it as its API states. Once mach signalling wasn't used any more, would it be possible to add this feature to Realm?

Loading

@bmunkholm
Copy link
Contributor

@bmunkholm bmunkholm commented Oct 22, 2020

@XabierGoros Totally understood. This is a limitation we would love to get rid of. Unfortunately we continue to have other features we judge are more important for more people. But it's surely a feature that's hovering close in the top list of things we want to get to whenever we get the capacity. I'm sorry that's not so useful for you right now though.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet