Invalid streaming format cookie

[charleshberlin]

How frequently does the bug occur?

Sometimes

Description

We have two users who are currently experiencing a crash [RLM_ERR_INVALID_DATABASE]: Failed to open Realm file at path '/data/user/0/*******/files/********.realm': file is in streaming format but has an invalid footer cookie (0). The file is probably truncated.

It appears to happen when the app is started (potentially woken up via a worker) and we attempt to open a realm, however we do not understand what could be causing this issue because neither user had any other errors around the time this occurred (that we can see). It seems like it may have started overnight when the device was sleeping because both instances started in the early am hours and then the user reported it repeatedly crashing after attempting to use it.

Currently we have no way of resolving this issue, so our only course of action for now is to have the user re-install the application. For now we are planning on simply catching this error and then deleting the realm and re-opening a new one, but this is not an ideal solution either.

Stacktrace & log output

Fatal Exception: java.lang.IllegalStateException: [RLM_ERR_INVALID_DATABASE]: Failed to open Realm file at path '/data/user/0/*******/files/********.realm': file is in streaming format but has an invalid footer cookie (0). The file is probably truncated.
       at io.realm.kotlin.internal.interop.CoreErrorConverter.asThrowable(CoreErrorConverter.kt:44)
       at io.realm.kotlin.internal.interop.realmcJNI.open_realm_with_scheduler(realmcJNI.java)
       at io.realm.kotlin.internal.interop.realmc.open_realm_with_scheduler(realmc.java)
       at io.realm.kotlin.internal.interop.RealmInterop.realm_open(RealmInterop.kt:193)
       at io.realm.kotlin.internal.interop.RealmInterop.realm_open$default(RealmInterop.kt:181)
       at io.realm.kotlin.internal.ConfigurationImpl.openRealm$suspendImpl(ConfigurationImpl.kt:107)
       at io.realm.kotlin.internal.ConfigurationImpl.openRealm(ConfigurationImpl.kt)
       at io.realm.kotlin.internal.RealmImpl$1.invokeSuspend(RealmImpl.kt:107)
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
       at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
       at kotlinx.coroutines.BlockingCoroutine.joinBlocking(BlockingCoroutine.java:85)
       at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(BuildersKt__Builders.kt:59)
       at kotlinx.coroutines.BuildersKt.runBlocking(Builders.kt:1)
       at io.realm.kotlin.internal.platform.CoroutineUtilsSharedJvmKt.runBlocking(CoroutineUtilsSharedJvm.kt:22)
       at io.realm.kotlin.internal.platform.CoroutineUtilsSharedJvmKt.runBlocking$default(CoroutineUtilsSharedJvm.kt:21)
       at io.realm.kotlin.internal.RealmImpl.<init>(RealmImpl.kt:106)
       at io.realm.kotlin.internal.RealmImpl.<init>(RealmImpl.kt)
       at io.realm.kotlin.internal.RealmImpl$Companion.create$io_realm_kotlin_library(RealmImpl.kt:263)
       at io.realm.kotlin.Realm$Companion.open(Realm.kt:82)
       ******************************************************************************************
       at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
       at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
       at android.os.Handler.handleCallback(Handler.java:942)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loopOnce(Looper.java:201)
       at android.os.Looper.loop(Looper.java:288)
       at android.os.HandlerThread.run(HandlerThread.java:67)

Can you reproduce the bug?

No

Reproduction Steps

No response

Version

1.8.0

What Atlas App Services are you using?

Local Database only

Are you using encryption?

No

Platform OS and version(s)

Android 13

Build environment

Android Studio version: Android Studio Flamingo | 2022.2.1 Patch 2
Android Build Tools version: 30.0.3
Gradle version: 7.5

[cmelchior]

Yes, that error does point to the Realm file being corrupted somehow.

Can you provide some more information about how you are using Realm?

• Are you using encryption?
• Are you using compaction? Either RealmConfiguration.compactOnLaunch or manual?
• Are you using RealmConfiguration.initialRealmFile?
• Are you manually copying or moving the Realm file around?
• Where are you using Realm. Judging from the stack trace it looks like an Android app?
• Do you have Android Backup enabled? https://developer.android.com/guide/topics/data/backup

[sipersso]

I am also seeing this. The original realm is fine, but I am writing copies of the realm files and these gets corrupted. I have only been getting a few bug reports so it doesn't seem to be frequent. I do use encryption and compactOnLaunch. No initial realm file. It is an Android app. I do allow backups but is using backup_rules.txt to exclude .realm files. Are there more files I should exclude?

[cmelchior]

@sipersso How are you writing these copies? Using the Realm.writeCopyTo API or some other API?

A Realm actually contains more than the database file itself, there is also a <realmFileName>.note (which we use to track communication between processes) and a <realmFileName>.management directory that contains a few files.

If these files are restored and combined with a new Realm file, this could definitely lead to errors like this one.

[sipersso]

Yes, I use write copy to. The strange thing is that these are in "cache" directories and only used for uploading data once to a server. I am never actually reading from these files from within the app. So I doubt the backup rules will matter since the realm files are just temporary. The user realm file is untouched and doesn't have the issue.

Now that I think of it, the configuration for these files is not using using encryption or compaction, and the folders are deleted once the file is uploaded to my server.

[charleshberlin]

Are you using encryption? No
Are you using compaction? Either RealmConfiguration.compactOnLaunch or manual? Yes we are using compactOnLaunch with the default callback
Are you using RealmConfiguration.initialRealmFile? No
Are you manually copying or moving the Realm file around? No
Where are you using Realm. Judging from the stack trace it looks like an Android app? Android app
Do you have Android Backup enabled? https://developer.android.com/guide/topics/data/backup No, we explicitly set allowBackup to false

[clementetb]

Do you have access to any of these corrupted Realm files?

[charleshberlin]

Sadly no, they were on release builds on customer phones and they were instructed to delete the app and reinstall

[sipersso]

I have sample files... where should I send them?

[clementetb]

@sipersso send them to realm-help@mongodb.com

[sipersso]

@clementetb Files sent. Not sure if this helps, but an interesting observation is that the files all seem to have even sizes (512kb, 1024kb, 2048kb, etc).

[clementetb]

Thanks, we have received them.

[clementetb]

The core team is investigating the issue. As a workaround to prevent this issue you could disable compactOnLaunch.

[sipersso]

@clementetb Just to clarify a bit on the files I uploaded.

1: I have a default realm that the user typically writes to. This realm does have compactOnLaunch enabled.
2: The user can write a copy using writeCopyTo(tmpConfiguration). The tmpConfiguration does not have compactOnLaunch enabled

It is the copy that gets corrupted. The default realm, which is using compaction is fine as far as I can see in my case.

[clementetb]

Thanks for the clarification, I will forward these details to the core team.