Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sharing encrypted Realms across processes #1845

Open
jpsim opened this Issue May 27, 2016 · 18 comments

Comments

Projects
None yet
10 participants
@jpsim
Copy link
Contributor

jpsim commented May 27, 2016

This was previously impossible when encryption used mach signaling, but should now be feasible.

@jpsim

This comment has been minimized.

Copy link
Contributor Author

jpsim commented May 27, 2016

Cocoa issue: realm/realm-cocoa#1693

@jpsim

This comment has been minimized.

Copy link
Contributor Author

jpsim commented Jan 5, 2017

This is marked as P1. Is that an accurate reflection of its prioritization?

@ianyh

This comment has been minimized.

Copy link

ianyh commented Mar 24, 2017

So is this, in fact, feasible?

@ianyh

This comment has been minimized.

Copy link

ianyh commented Mar 26, 2017

To clarify, I am willing to put development effort into this, but would rather not spend the time if it's going to turn out not to be feasible.

@bmunkholm

This comment has been minimized.

Copy link
Contributor

bmunkholm commented Mar 27, 2017

@finnschiermer Can you comment on what's required to support this?

@tgoyne

This comment has been minimized.

Copy link
Member

tgoyne commented Mar 27, 2017

The problem for multiprocess is knowing when to discard the in-memory decrypted data for each page. Currently a page which is read by one process will not be reread by that process following a change to it from a different process and then bad things happen.

One idea I have to make this work would be to adjust the allocator to never reuse space on a page until that entire page is freed up, and then make advancing the read transaction mark all pages which contain a freelist entry as out of data. There may be problems with this beyond that it would increase disk usage.

@ianyh

This comment has been minimized.

Copy link

ianyh commented Mar 27, 2017

What information does a process have about changes from another process? Is there a technical limitation that prevents the process from rereading relevant pages? Is it a performance limitation? Is there just not enough granularity in the information to be able to selectively read pages?

@finnschiermer

This comment has been minimized.

Copy link
Contributor

finnschiermer commented Mar 28, 2017

@ianyh @bmunkholm We have not designed a mechanism specifically for informing a reader of a change made in another process. I think @tgoyne is right in principle that such a mechanism can be based on monitoring our free-list. We could compare the new free-list against the previous versions free-list to detect allocations, then invalidate (to later trigger decryption of) all touched pages. Since the free-lists of earlier versions are needed to do this, it runs counter to a desire to cut older versions free-lists to minimize file-size growth problems. Also, since the free-list itself is encrypted some care is needed when walking it, but it should be doable. We'll also have to be careful not to duplicate the work, when multiple readers grab new versions - some synchronization will be needed there.

@finnschiermer

This comment has been minimized.

Copy link
Contributor

finnschiermer commented Mar 28, 2017

It think it is easier to decrypt all pages involved in walking from the top-ref to all leafs, but not the leafs themselves. The leafs should only be marked invalid in order to trigger later decryption as needed. If we added version numbers to interior nodes we could prune the walk even more.

@finnschiermer

This comment has been minimized.

Copy link
Contributor

finnschiermer commented Mar 28, 2017

@tgoyne Hmm. I have a feeling I did not understand your suggestion completely. Would you care to elaborate?

@ianyh

This comment has been minimized.

Copy link

ianyh commented Mar 29, 2017

@finnschiermer If I understand correctly, I believe @tgoyne's idea is to prevent the reuse of allocated storage if doing so would change the data underlying existing in-memory decrypted data. Any newly written data would have no existing in-memory decrypted representation in any process, and thus get decrypted correctly by every process. Presumably, the mechanism for encryption already drops its in-memory decrypted data when a page is entirely reclaimed?

That certainly does seem like it would increase disk usage. I can't think of first-order effects that would impact performance. Either way this would only impact encrypted databases. Would it be too much to require a database to be manually configured as encrypted and shareable between processes before this mechanism kicks in? It would isolate the effect on disk usage, but would increase the complexity of configuration.

@ianyh

This comment has been minimized.

Copy link

ianyh commented Apr 4, 2017

Also, thinking more about constraints, are things significantly simplified if we assume only one of the connections to the db can write?

@ianyh

This comment has been minimized.

Copy link

ianyh commented Apr 24, 2017

@finnschiermer @tgoyne Okay, thinking through more after having familiarized myself with more of the code. Would changes in the free-list be enough to determine out of date pages? The thing I am still grappling with is what happens when a page with data is freed and then allocated again from one process. How would another process distinguish that from the page being unchanged? That could theoretically result in no change to the free-list, right?

@beseder42

This comment has been minimized.

Copy link

beseder42 commented May 8, 2017

Can you estimate in which time horizon this will be implemented? We need it strongly and would have to make lots of work arounds otherwise. Thanks!

@beseder42

This comment has been minimized.

Copy link

beseder42 commented May 24, 2017

Would really appreciate any progress on this!

Addition
This is such a needed feature for security reasons!! How come this is not brought to production for so long and now on p2 instead of p1? How can you are promote Realm to be first choice for secure apps but there is no way to work with extensions on an encrypted Realm?
Too sad Realm is only focusing on the paid server features lately and has no more focus on local database features 🙁 Will look for other database for future apps!!

@bmunkholm bmunkholm removed the S:Backlog label Aug 21, 2017

beeender added a commit to realm/realm-java that referenced this issue Oct 26, 2017

Add multi-process example
All Realm public APIs should be process-safe now.

Encryption is not supported for multi processes due to core
restriction realm/realm-core#1845

Accessing same Realm file in different processes from different apks
is safe, but the notifications won't work.

beeender added a commit to realm/realm-java that referenced this issue Nov 1, 2017

Add multi-process example (#5473)
All Realm public APIs should be process-safe now.

Encryption is not supported for multi processes due to core
restriction realm/realm-core#1845

Accessing same Realm file in different processes from different apks
is safe, but the notifications won't work.

@bmunkholm bmunkholm added the T:Feature label Jun 13, 2018

@roberhofer roberhofer self-assigned this Jun 13, 2018

@joshuawright11

This comment has been minimized.

Copy link

joshuawright11 commented Jul 24, 2018

Any update on this? This would be super important to our company.

@ianyh

This comment has been minimized.

Copy link

ianyh commented Aug 3, 2018

I haven’t had the bandwidth to work on this, unfortunately. We came up with a workaround for our specific situation. I don’t know what the status is otherwise.

@finnschiermer

This comment has been minimized.

Copy link
Contributor

finnschiermer commented Aug 3, 2018

The status is unchanged. We don't have the bandwidth to work on it at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.