Interprocess support for encrypted Realms

[KynoYang]

Realm version is 0.8.2, Android version is 4.4.2

Encrypted interprocess sharing is currently unsupported in io_realm_internal_SharedGroup.cpp line 115
at io.realm.internal.SharedGroup.createNativeWithImplicitTransactions(SharedGroup.java)
at io.realm.internal.SharedGroup.(SharedGroup.java:60)
at io.realm.Realm.(Realm.java:207)
at io.realm.Realm.createAndValidate(Realm.java:597)
at io.realm.Realm.create(Realm.java:567)
at io.realm.Realm.getInstance(Realm.java:540)

Anyone know what it is?

[beeender]

Are you using same Realm in different processes? @KynoYang Multi-processes for java is not supported yet...

[kneth]

As @beeender points out, multiprocess access isn't yet supported - see #1091.

[beeender]

@KynoYang is still seeing this issue http://crashes.to/s/afa05a92642
on smartisan S701
This should be the dup issue to #2762

[phamducgiam]

Hi.

I encountered this issue too cause of using firebase. It causes by adding plugin 'com.google.gms.google-services' for automatically integrating firebase sdk. If I removed that plugin or remove realm encryption, everything is ok. Any suggesstion? Thanks.

[beeender]

@phamducgiam Do you have something like process:remote in your manifest?

[phamducgiam]

@beeender: I just found out. It's because firebase crash automatically create a background process for reporting. I just disable firebase crash for now and will fix it later.

[mvojtkovszky]

Here's my observation, reposted here as instructed in #3203

io.realm.exceptions.RealmError: Unrecoverable error. /data/data/<package>/files/database.realm: Encrypted interprocess sharing is currently unsupported.SharedGroup has been opened by pid: 22257. Current pid is 17958. in io_realm_internal_SharedGroup.cpp line 113
    at io.realm.internal.SharedGroup.createNativeWithImplicitTransactions(Native Method)
    at io.realm.internal.SharedGroup.a(SourceFile:95)
    at io.realm.internal.SharedGroup.<init>(SourceFile:74)
    at io.realm.internal.SharedGroupManager.<init>(SourceFile:49)
    at io.realm.BaseRealm.<init>(SourceFile:81)
    at io.realm.Realm.<init>(SourceFile:140)
    at io.realm.Realm.b(SourceFile:240)
    at io.realm.Realm.a(SourceFile:220)
    at bbq.a(SourceFile:126)
    at io.realm.Realm.getDefaultInstance(SourceFile:166)

Crashlytics reports this kind of crash for around 1,5% of about 50k users so far and I can't figure out what causes it, neither reproduce it on any of our test devices or emulator.

Things to consider:

• getDefaultInstance() that causes the crash is always called inside onCreate() method of our Application class. That's where Realm's default configuration gets set and default instance is obtained.
• Our manifest file doesn't define any remote processes, like process:remote
• If a third party library created a remote process, we would expect the crash to always happen? (I also try to catch it with RealmIOException, but that doesn't seem to do anything for any reason)
• Crash always spikes on the first days following the update and drops significantly after, indicating that only users who updated get affected.
• Considering the sheer amount of crashes and users affected, we get no complaint emails reporting this.

Version of Realm and tooling

Realm version(s): 1.1.0
Android Studio version: 2.1.2
Which Android version and device: all supported (4.0.3+)

[beeender]

@biftekman A hypothesis is there is a bug with some devices's package manager. When installing and running the new version of APK, the old process has not been killed yet -- This would trigger your issue and #2928 .

Crash always spikes on the first days following the update and drops significantly after, indicating that only users who updated get affected.

And I think above fact supports our hypothesis well.

We made some workaround try to solve #2928 -- delay some time if we see there is another process alive and opening the Realm file, but from feedback, it doesn't help too much :(

[mvojtkovszky]

@beeender does the crash then kill the old process as well? Meaning that reopening the app afterwards would occur normally?

[beeender]

@biftekman It will only kill the process which throws this exception. But from the exception message:

Encrypted interprocess sharing is currently unsupported.SharedGroup has been opened by pid: 22257. Current pid is 17958. in io_realm_internal_SharedGroup.cpp

Interesting thing is we can see it is actually the old process throws the exception ( 17958, since the pid number is not big enough, i guess we can assume 22257 is started later than 17958).

Can you see in the crashlytics backend that the exception is reported from the old version or the new version? (Although I am not quite sure how does crashlytics get the version information. If it gets the information based on the apk installed on the system, then it will always reports the new version i think?)

[mvojtkovszky]

@beeender
Looking through our crash logs, current pid value is not always less than opened one, for example

Encrypted interprocess sharing is currently unsupported.SharedGroup has been opened by pid: 4543. Current pid is 22644. in io_realm_internal_SharedGroup.cpp line 113

But that should not indicate that current process id is either "newer" or "older". Although PIDs are allocated on a sequential basis, when max value is reached, allocations start from 0 again.

Regarding Crashlytics reporting: as it is in our case initialised inside Application class which is started by a process under the same package, it will report a crash from whatever process causes it. Since we always see crash results matching the newer versions, I'd guess the new process throws the exception.

[beeender]

4543 is a quite small number for a pid, unless it is just powered on, i'd rather to assume it is "older" than 22644. But anyway, it is all guessing ...

Do you have any logic to restart a service in the apk? I tried something like START_STICKY, but cannot reproduce it :(

[mvojtkovszky]

The only time the app starts a service is when I bind IInAppBillingService to Activity.

But in any case, creating another process during runtime should crash the app in our situation anyway (since Realm instance is obtained in application class), which never happens. So that shouldn't prove anything.

Regarding your original hypothesis, did you manage to reproduce the "not-killed process on update" before?
If so, it seems very plausible.

[beeender]

We tried a lot, but never reproduce "not-killed process on update" :(

But from the crash log, there are two processes for a same app are running in the system which should not ever happen ... We tried a lot but cannot reproduce it on any of our devices ...

[mvojtkovszky]

Is a following workaround currently possible for our situation:
If there is a process that holds an open instance of Realm, is there a way to detect that and close that instance from second process that is about to obtain a new instance?

If not, solution to this is still pretty straightforward:
Never obtain an instance of Realm in Application class, because that will keep it open for as long as that process is alive.
Instead, always obtain instance when needed and close when done, without exceptions.

[09ae9c]

Hi,

Realm Version: 6.0.1, Still crash...

 Unrecoverable error. /data/data/***/files/YUc5eVkzSjFlRjluYkc5aVlXeGZjbVZoYkcwPQ==.realm: Encrypted interprocess sharing is currently unsupported.SharedGroup has been opened by pid: 22164. Current pid is 1009. in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101

[tfkci]

After update have this issue for a while. But can't reproduce (v6.0.2)

[strooooke]

Issue is still ongoing with v6.1.0:

Unrecoverable error. /data/data/****/files/default.realm: Encrypted interprocess sharing is currently unsupported.SharedGroup has been opened by pid: 8814. Current pid is 1216. in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101

Occurs repeatedly (on subsequent app starts) for the same user (but several users are impacted).
Cannot reproduce locally - just seeing the crash reports.

Might be interesting: I have subsequent reports (that may very well belong to the same user), where the "opened by pid" stays constant, and the "current pid" changes.

[Myk-da-Beast]

I see some users recommend moving Realm.init(context) out of Application.onCreate(), but where is the recommended initialization location in that case?

[Zhuinden]

Don't move Realm.init(context) out of Application.onCreate(), but you can move Realm.getDefaultInstance() out of Application.onCreate().

[Myk-da-Beast]

Thanks for the clarification. I don't have Realm.getDefaultInstance() in Application.onCreate() though, and I still get this problem. I suppose it has not yet been fixed?

[Myk-da-Beast]

Does Realm.setDefaultConfiguration() cause the same issue that Realm.getDefaultInstance() does?

[Zhuinden]

Not that I know of 🤔

[0xMineGo800m]

I am seeing the same issue.
Realm version: 6.0.2

I too cannot reproduce this on my current devices. But Crashlytics indicates this happens on Samsung A20s and I just got it now. So I will attempt to update the application that crashes in a couple of ways. Perhaps it will reproduce.

Any way, in Application onCreate() I init Realm with Realm.init(). Which as mentioned here is fine.
You also mentioned not to call Realm.getDefaultInstance() in onCreate() and I think we do call it.
We are checking if there is an old unencrypted Realm file and if there is, we copy it to an encrypted file and delete the old file.
Here is the code that does this:

First I call this in Application onCreate():

public void buildDatabase(){
        Realm.init(this);
        RealmConfiguration.Builder tempConfig = new RealmConfiguration.Builder()
                .schemaVersion(AppRealmMigration.CURRENT_VERSION)
                .migration(new AppRealmMigration())
                .modules(new RealmAsimovCoreModule(), new RealmAsimovProtocolV5Module(), new RealmAsimov());

        RealmConfiguration encryptedConfig = RealmUtils.get().getEncryptedInstance(tempConfig, "default.realm");
        Realm.setDefaultConfiguration(encryptedConfig);
        Realm.compactRealm(encryptedConfig);
    }

And here is RealmUtils.get().getEncryptedInstance():

public RealmConfiguration getEncryptedInstance(RealmConfiguration.Builder realmConfiguration, String unencryptedFileName){

        RealmConfiguration oldConfig = realmConfiguration.build();

        String encryptedFileName = encryptedFileName(unencryptedFileName);
        realmConfiguration.name(encryptedFileName);
        realmConfiguration.encryptionKey(realmEncryptionKey);
        RealmConfiguration newConfig = realmConfiguration.build();

        File newRealmFile = new File(newConfig.getPath());
        if (newRealmFile.exists()) {
            Realm.deleteRealm(oldConfig);
            return newConfig;
        } else {
            try (Realm realm = Realm.getInstance(oldConfig)) {
                // Migrate old Realm and delete old
                realm.writeEncryptedCopyTo(newRealmFile, realmEncryptionKey);

                Realm.deleteRealm(oldConfig);
                return newConfig;
            } catch (Exception e) {
                Logger.log("RealmUtils", "Failed to create encrypted realm (with key: " + realmEncryptionKey + ") with exception: " + e.getMessage(), Logger.FILE_ONLY);
                return newConfig;
            }
        }
    } 

I suspect Realm.getInstance(oldConfig) is like Realm.getDefaultInstance ?

Could this be the issue?

Thank you

Edit 1:
I tried a couple of ways to update the application. It did not reproduce :(

Edit 2:
I am not sure if this is related but there is another type of exception:

Fatal Exception: java.lang.IllegalStateException: History type (as specified by the Replication implementation passed to the SharedGroup constructor) was not consistent across the session
Exception backtrace:
<backtrace not supported on this platform>

There is a stack trace that starts from a certain method that returns an instance of a Realm DB and then:

at io.realm.internal.OsSharedRealm.nativeGetSharedRealm(OsSharedRealm.java)
       at io.realm.internal.OsSharedRealm.<init>(OsSharedRealm.java:171)
       at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:241)
       at io.realm.BaseRealm.<init>(BaseRealm.java:136)
       at io.realm.BaseRealm.<init>(BaseRealm.java:103)
       at io.realm.Realm.<init>(Realm.java:163)
       at io.realm.Realm.createInstance(Realm.java:499)
       at io.realm.RealmCache.doCreateRealmOrGetFromCache(RealmCache.java:360)
       at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:290)
       at io.realm.Realm.getInstance(Realm.java:428)

Edit 3:
And another one in the same area:

Caused by io.realm.exceptions.RealmFileException: Unable to open a realm at path '/data/data/app.package.com/files/defaultEncrypted.realm': Invalid streaming format cookie (18420000062399578111) Path:Exception backtrace:
<backtrace not supported on this platform>. (Invalid streaming format cookie (18420000062399578111) Path: /data/data/app.package.com/files/defaultEncrypted.realm
Exception backtrace:
<backtrace not supported on this platform>) (/data/data/app.package.com/files/defaultEncrypted.realm) in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101

[0xMineGo800m]

Could calling Realm.compactRealm() during Realm init cause this issue? I am calling that during Application onCreate(). I'll try to change it to compactOnLaunch() in the RealmConfiguration builder.

[prettyClouds]

To sum up this thread:

1. You can't access an encrypted Realm from two different processes, so if you are doing that on purpose, one of the processes will crash.

2. It looks like when Android updates, the old APK can still be running when the new one starts up. If the new process tries to access an encrypted Realm while the old one still has an instance, the new one crashes.

What I can't seem to find is any recommendation on how to minimize this issue. One that comes to mind is to minimize the amount of time between when a process calls Realm.getDefaultInstance() and Realm.close(). Patterns like getting a Realm everytime and Activity is created and closing it when it's destroyed will probably exacerbate this problem. Is there anything I am missing? Is there a better way to solve this?
@beeender @kneth

[clementetb]

Unfortunately, we don't have a solution for when the OS instantiates twice the app because of an update. To minimize the impact you could catch the RealmError thrown in such cases and then exit gracefully:

try {
    realm = Realm.getDefaultInstance()
} catch (error: RealmError){
    // report what happened
    // exit gracefully
}

[Zhuinden]

I hear what can help is checking in a BaseActivity if applicationContext is your custom application class, if it is not, then just don't initialize anything.

[volo-droid]

Occurs repeatedly (on subsequent app starts) for the same user (but several users are impacted).
Cannot reproduce locally - just seeing the crash reports.

Might be interesting: I have subsequent reports (that may very well belong to the same user), where the "opened by pid" stays constant, and the "current pid" changes.

We're observing exactly the same in our project. Most of the issues are coming from Samsung S10+, on that device we had a spike of 286 crashes affecting 4 users within 4 days. For example, there're more than 70 crashes with the same opened pid 20925:

Time	PIDs
12 Nov 2020, 17:27:00	opened by pid: 20925. Current pid is 14901.
12 Nov 2020, 16:26:45	opened by pid: 20925. Current pid is 6094.
12 Nov 2020, 13:26:43	opened by pid: 20925. Current pid is 25312.
12 Nov 2020, 12:40:46	opened by pid: 20925. Current pid is 6122.
12 Nov 2020, 10:25:49	opened by pid: 20925. Current pid is 21103.
12 Nov 2020, 10:11:48	opened by pid: 20925. Current pid is 17393.
12 Nov 2020, 08:22:05	opened by pid: 20925. Current pid is 15637.
12 Nov 2020, 08:22:00	opened by pid: 20925. Current pid is 15398.
12 Nov 2020, 07:21:26	opened by pid: 20925. Current pid is 11062.
12 Nov 2020, 07:21:22	opened by pid: 20925. Current pid is 10834.
…	…

One question, would it be possible to include in the exception message not only a PID but also a process name?

[Zhuinden]

@volo-droid what is your Realm version, and what Realm version created the Realm file?

[volo-droid]

@Zhuinden The current version is 7.0.2, but I'm not sure what version was used to create a Realm file. The whole Realm thing is encapsulated in an in-house SDK we use, I'd have to check with its maintainers. In the git blame I can see the following Realm versions 5.4.1 -> 6.1.0 -> 7.0.2

[andersfredlund]

I also see this issue popping up when it is time to upgrade our app and it is mostly Galaxy S10+

[johnsabilla]

We also experience this exception on v7.0.8

[retyui]

Had this error on realm: 6.1.5

/data/data/my.app/files/default.realm: Encrypted interprocess sharing is currently unsupported.SharedGroup has been opened by pid: 10296. Current pid is 14470.

After an app update through Play Market

[KimiChiu]

Is there any workaround? You said it's better to put compactRealm anywhere but Application.onCreate. What if I put it before Realm.getInstance? Is it safe to call it on worker thread?

[vonox7]

@volo-droid you can get the other processName via:

  private fun getProcessName(pid: Int): RunningAppProcessInfo? {
    val manager = App.instance.getSystemService(Context.ACTIVITY_SERVICE) as ActivityManager
    return manager.runningAppProcesses.find { it.pid == pid }
  }

val otherPid = Regex("DB has been opened by pid: ([\\d]+)").find(exception.message!!)!!.groups[1]!!.value.toInt()
getProcessName(otherPid)?.processName

Maybe this issue might happen not only during app-upgrade, but also when your app has a Worker do to some background work?

[luispereira]

I'm using realm 10.4.0 and this still happens. Always when the user migrates from a previous realm version

[cmelchior]

Fixed by #7795