io.realm.exceptions.RealmFileException when launching application #4524

jollyjoker992 · 2017-04-20T09:02:18Z

I have researched many and many times about this issue but I have not found any solution.
I am implementing an Android application with Realm Android 3.0 and using Fabric to find out crash report.
Below is Realm initialize code in Application class:

 Realm.init(this);
        RealmConfiguration config = new RealmConfiguration.Builder().name(REALM_SCHEMA_NAME)
                .schemaVersion(REALM_SCHEMA_VERSION)
                .migration(new Migration())
                .build();
        Realm.setDefaultConfiguration(config);
        Realm realm = Realm.getDefaultInstance(); // Automatically run migration if needed
        realm.close();

I get many crash report from Fabric related with io.realm.exceptions.RealmFileException such as:

Unable to open a realm at path '/data/data/com.xxx/files/xxx.realm': Incompatible histories.
Unable to open a realm at path '/data/data/com.xxx/files/xxx.realm': bad realm file header (Complete the library for nested tables [3-5h] #3). (bad realm file header (Complete the library for nested tables [3-5h] #3))
Unrecoverable error. mmap() failed: Out of memory size xxx
etc ...

So, can you tell me the reason of this issue? Thank you.

The text was updated successfully, but these errors were encountered:

Zhuinden · 2017-04-20T10:25:36Z

mmap() failed is generally caused by unclosed Realm instances on background threads.

Bad file header can be caused by compacting in multiple processes.

jollyjoker992 · 2017-04-21T01:16:20Z

@Zhuinden : I do not use compact method, so why do it happen?

kneth · 2017-04-24T09:39:15Z

@jollyjoker992 Does your app have multiple processes? Does it happen the first time your app is launched? Did the device run out of disk space?

jollyjoker992 · 2017-04-25T03:37:27Z

@kneth : No, my app is single process. It does not happen in the first time launching app, It happens in many time when launching app. I have attached code above and it is located in Application class. From Fabric report, all devices has enough free disk space.

kneth · 2017-04-25T07:55:04Z

@jollyjoker992 Is there a pattern in which devices you see the crashes? How easy is it to reproduce (i.e., how often does it happen)?

jollyjoker992 · 2017-04-26T03:00:47Z

@kneth : Because I get crash report from users by Fabric, so I cannot reproduce it. So, sometime It happens when I launch app, I trace log and see all of issues I mentioned above. From Fabric, it happens in:

Devices : SHAFT (41%), Sony (34%), Fujisu (24%) and all of them have enough free disk space and do not root before.
OS : 6 (76%), 5 (13%), 7 (6%)

Can you tell me all of reasons make all of above issue? I will try to verify each reason and hope find out the root cause. Thank you.

kneth · 2017-04-26T08:06:59Z

@jollyjoker992 So no obvious patterns 😢

If you can post the stack trace/log from one or two of the crashes, it might be useful.

jollyjoker992 · 2017-04-27T02:43:59Z

@kneth : Below is one of crash logs I just found right now. I will update more log later. Thank you.

Fatal Exception: java.lang.RuntimeException: Unable to create application com.xxx.xxx.xxxApplication: io.realm.exceptions.RealmFileException: Unable to open a realm at path '/data/data/com.xxx.xxx.xxx/files/xxx.realm': Bad Realm file header (#3). (Bad Realm file header (#3)) (/data/data/com.xxx.xxx.xxx/files/xxx.realm) in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_SharedRealm.cpp line 109 Kind: ACCESS_ERROR.
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4809)
       at android.app.ActivityThread.access$1600(ActivityThread.java:154)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1452)
       at android.os.Handler.dispatchMessage(Handler.java:102)
       at android.os.Looper.loop(Looper.java:224)
       at android.app.ActivityThread.main(ActivityThread.java:5526)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)
Caused by io.realm.exceptions.RealmFileException: Unable to open a realm at path '/data/data/com.xxx.xxx.xxx/files/xxx.realm': Bad Realm file header (#3). (Bad Realm file header (#3)) (/data/data/com.xxx.xxx.xxx/files/xxx.realm) in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_SharedRealm.cpp line 109
       at io.realm.internal.SharedRealm.nativeGetSharedRealm(SharedRealm.java)
       at io.realm.internal.SharedRealm.<init>(SharedRealm.java:187)
       at io.realm.internal.SharedRealm.getInstance(SharedRealm.java:229)
       at io.realm.internal.SharedRealm.getInstance(SharedRealm.java:204)
       at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:124)
       at io.realm.Realm.getDefaultInstance(Realm.java:210)
       at com.lab.xxx.xxxApplication.initAndMigrateRealmIfNeeded(xxxApplication.java:60)
       at com.lab.xxx.xxxApplication.onCreate(xxxApplication.java:45)
       at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1014)
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4806)
       at android.app.ActivityThread.access$1600(ActivityThread.java:154)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1452)
       at android.os.Handler.dispatchMessage(Handler.java:102)
       at android.os.Looper.loop(Looper.java:224)
       at android.app.ActivityThread.main(ActivityThread.java:5526)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)

Zhuinden · 2017-04-27T07:07:13Z

A possible fix for I think this error is being investigated in #4559

kneth · 2017-04-27T08:09:29Z

@jollyjoker992 With #4559 merged, you can try a snapshot release and see if it solves your issue.

jollyjoker992 · 2017-04-27T10:05:30Z

@Zhuinden
@kneth
Thanks for your support. I will try to use snapshot. But, can you tell me about plan to release version includes this commit?

kneth · 2017-04-28T08:11:59Z

@jollyjoker992 No exact date but we will probably do a release in a week or two (we don't have fixed date as support load or critical bugs are hard to predict).

jollyjoker992 · 2017-04-28T10:16:23Z

@kneth : I tried to use Snap shot for testing this issue. So, Can you close this issue? I will re-open it or note for you after a few days for testing. Thank you for your support.

jollyjoker992 · 2017-05-01T02:18:58Z

@kneth : Hi. I have tried to use Realm Snap-shot 3.2.0 to test the latest version (the last version is 3.0.0). The issue 'Bad realm file header' maybe resolved, I have not seen it in crash report. But, the below issues still occurs.

Unable to open a realm at path '/data/data/com.xxx/files/xxx.realm': Incompatible histories.
Unrecoverable error. mmap() failed: Out of memory size xxx

I close Realm instance follow App life cycle, but when system kills my app (on low memory case), I cannot close it. So, Is it a root cause? Pls help me to investigate problem by showing me some cases that these issues may occur. Thank you so much.

Zhuinden · 2017-05-01T09:01:14Z

Unrecoverable error. mmap() failed: Out of memory size xxx

This is generally caused if the Realm file is too large, which is generally when there is at least one unclosed Realm instance on a non-looper background thread.

cmelchior · 2017-11-14T08:17:05Z

If your Realm file is corrupted, it shouldn't be possible to open it on any thread?

jollyjoker992 · 2017-11-14T08:22:15Z

@cmelchior : I think there are some cases meets my above question. For example, when I open more than 2 async transaction at the same time, we have more than 2 Realm instance in back ground thread but one of them throws the corrupted exception, another instance is in non-finished transaction and when I catch this exception, I delete the realm file immediately but I cannot because there are a realm instance has not been closed :(

jollyjoker992 · 2017-12-02T15:21:38Z

@cmelchior @Zhuinden @kneth Thanks for your support. I'm currently catching the Exception when openning Realm instance then delete it if need and recover Realm file by supporting from server side.

Cannot delete Realm file although I use only 1 instance in Main thread but when I put the log to get global Realm instance count, I saw that there are 2 instances and 1 instance randomly. I guess that there are a transaction async is in progress, so I block this thread until the getGlobalInstanceCount return 1. It's resolved.
I will try to reduce opening many transactions at the same time and hope it will be resolved but I have a question for you.

If open many transactions at the same time causes Realm file size becomes too large, so It seems like a bug, right? Can you control it to reduce the Realm file size like compact method is defined ?

Zhuinden · 2017-12-02T15:26:17Z

@jollyjoker992 have you tried with Realm 4.2.0? It should have a bunch of multi-process related fixes (because of Realm Core 4.0.3).

jollyjoker992 · 2017-12-02T15:27:47Z

@Zhuinden : No, I have not yet but I will try it 👍 Thanks you

jollyjoker992 · 2017-12-02T15:30:30Z

@Zhuinden : Realm 4.2 release note is not mentioned about the issue below? It is fixed ?

Fatal Exception: io.realm.exceptions.RealmError: Unrecoverable error. mmap() failed: Out of memory size: 1207959552 offset: 0 in /home/cc/repo/realm/release/realm/realm-library/src/main/cpp/io_realm_internal_SharedRealm.cpp line 217
       at io.realm.internal.SharedRealm.nativeGetSharedRealm(SharedRealm.java)
       at io.realm.internal.SharedRealm.<init>(SharedRealm.java:190)
       at io.realm.internal.SharedRealm.getInstance(SharedRealm.java:237)
       at io.realm.internal.SharedRealm.getInstance(SharedRealm.java:206)
       at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:127)
       at io.realm.Realm.getDefaultInstance(Realm.java:266)
       at com.xxx.xxx.data.source.local.RealmProcessor.init(RealmProcessor.java:29)
       at com.xxx.xxx.xxxApplication.onCreate(xxxApplication.java:41)
       at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1014)
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4804)
       at android.app.ActivityThread.access$1600(ActivityThread.java:154)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1452)
       at android.os.Handler.dispatchMessage(Handler.java:102)
       at android.os.Looper.loop(Looper.java:234)
       at android.app.ActivityThread.main(ActivityThread.java:5524)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)

Zhuinden · 2017-12-02T16:41:34Z

@jollyjoker992 I mean that says your Realm file grew to 1207959552 bytes which means it was 1.125 GB large.

Is that normal for your amount of data?

huunam118 · 2017-12-08T03:09:18Z

@Zhuinden , @cmelchior
Thanks much for your contributing !
Currently, we try to reproduce this issue by trying to open too much transactions. But seem the realm files did not grow up even if it is crashed by 100 queued transactions. As we said before, we only used Realm on main thread. Any other idea about how the realm files size too large? Or have you got an example about that?

Zhuinden · 2017-12-08T08:12:42Z

@huunam118 you mean you do Realm.getDefaultInstance() on UI thread, and you only do realm.executeTransaction()?

huunam118 · 2017-12-09T01:05:20Z

@Zhuinden
Actually, we only call Realm.getDefaultInstance() on UI thread to get its instance. About transaction we used both executeTransactionAsync and executeTransaction, and there are some simple queries.

jollyjoker992 · 2017-12-14T08:31:48Z

@Zhuinden @cmelchior I have a question relate this issue, pls help me.

Can I copy all data from old Realm file (corrupted with a large file size) to a new Realm file (create new file) without query and insert ? Of course, I want to keep both of them and switch to the expected file when working with them and just copy data (I don't want to clone a file with a large size).

Thank you

beeender · 2017-12-14T09:05:53Z

Can I copy all data from old Realm file (corrupted with a large file size) to a new Realm file (create new file) without query and insert ? Of course, I want to keep both of them and switch to the expected file when working with them and just copy data (I don't want to clone a file with a large size).

If it still can be opened and you just want to reduce the Realm file size, use Realm.compact() or RealmConfiguration.Builder.compactOnLaunch() will solve it.

But it is still important why the file grows too big, normally it is just because of Realm instances are not closed on the background thread.

jollyjoker992 · 2017-12-14T09:08:38Z

@beeender : I think because I opened too many transaction at the same time so the Realm file grow up with the large size.

Realm.compact() or RealmConfiguration.Builder.compactOnLaunch()

cannot help me :(

Reference: #4524 (comment)

beeender · 2017-12-14T09:13:08Z

I think because I opened too many transaction at the same time so the Realm file grow up with the large size.

So it is because of you really write that much of data to the Realm? One thing you can do is query for the data you don't need and delete them by RealmResults.deleteAllFromRealm() . The disk space can be reused later or if you want to reclaim the space immediately, you can call Realm.compactRealm() afterwards.

jollyjoker992 · 2017-12-14T09:17:55Z

@beeender : No, I did not write too much data because the data is very simple. Reference to here, I think it's the root cause make my Realm file grow up with a large size. But I'm currently focusing on the issue that :

How can I copy all realm data from the old file (with the large size but has not corrupted) into a new file without query and insert.

beeender · 2017-12-14T09:20:13Z

No, I did not write too much data because the data is very simple. Reference to here, I think it's the root cause make my Realm file grow up with a large size.

If that is the case, Realm.compact() should just solve your issue. Have you tried that? it would be strange if it doesn't help.

Can the large Realm file still can be opened?

jollyjoker992 · 2017-12-14T09:21:36Z

@beeender : I tried it but It cannot help me until the Realm file being corrupted.

beeender · 2017-12-14T09:29:11Z

If "Realm file being corrupted." means that it fails to open the file by mmap() failed xxx, then the file itself is not corrupted. It cannot be opened just because of there is not enough virtual memory space to mmap the Realm file. In this case, since you cannot open the file on that device (you probably can still open it on a 64bit device, but it is not practical for you I think.), you cannot query the data and rewrite it to another file.

jollyjoker992 · 2017-12-14T09:34:24Z

@beeender : Thanks for your supporting. I tried to use the compact() method when launching the application but It seems cannot help me anything until the file being corrupted. So, when the compact() method should be called to compact Realm file? When launching app or when close app? And If the file grows up because of here, the compact() method can help me?

beeender · 2017-12-14T10:36:23Z

compact() should reclaim disk space which is not being used including the case you mentioned. See https://realm.io/docs/java/latest#faq-large-realm-file-size

compact operation may take some time to finish depends on how much valid data stared.
Consider to use those APIs

https://realm.io/docs/java/4.3.1/api/io/realm/RealmConfiguration.Builder.html#compactOnLaunch--

https://realm.io/docs/java/4.3.1/api/io/realm/Realm.html#getInstanceAsync-io.realm.RealmConfiguration-io.realm.Realm.Callback-

jollyjoker992 · 2017-12-14T10:40:59Z

@beeender : Should I compact when closing app? Ex: compact() then close Realm instance (I just use only 1 instance of Realm in my app)

beeender · 2017-12-14T10:47:06Z

Realm.getInstanceAsync() + Realm.Builder.compactOnLaunch() will be the my choice if I want to do that. But it depends on your use case if you can manage the app's life cycle well.

jollyjoker992 · 2017-12-14T10:49:14Z

@beeender Thanks for your support. I will try it and feedback about the result for you ASAP. Many thanks from my heart 🙇‍♂️

jollyjoker992 · 2018-01-17T05:44:31Z

@beeender Sorry because mention you but I need help from you. To detect and recover data after Realm file is corrupted, I need to make a Realm file that is corrupted for testing. So pls help me to know how to corrupt a Realm file manually? Thank you.

bmunkholm · 2018-01-17T08:58:09Z

@jollyjoker992 You can just provide any file that's not a realm file. An empty one for instance.

jollyjoker992 · 2018-01-17T09:04:41Z

@bmunkholm I need a Realm crash when call Realm.init(Context). So, if I follow your solution, It's possible for testing?

jollyjoker992 · 2018-01-17T09:34:44Z

@Zhuinden @kneth @cmelchior @beeender
Sorry I need a help.
If the Realm file is corrupted, the method Realm.init(appContext) will throw an Exception or only Realm.getInstance() will throw an Exception?

cmelchior · 2018-01-17T09:35:56Z

Only Realm.getInstance() . Realm.init() just load and setup the Realm library, so it isn't connected to opening any files.

jollyjoker992 · 2018-01-17T09:41:46Z

@cmelchior Thanks. Sorry because I mentioned you many times. I am currently try to find the root cause of corrupted Realm file in my app. Thank you again.

karapigeon added T-Help S:Waiting For User labels Apr 20, 2017

realm-ci removed the S:Waiting For User label Apr 21, 2017

kneth added the S:Waiting For User label Apr 24, 2017

realm-ci removed the S:Waiting For User label Apr 25, 2017

kneth added Reproduction-Required S:Waiting For User labels Apr 25, 2017

realm-ci removed the S:Waiting For User label Apr 26, 2017

kneth added the S:Waiting For User label Apr 26, 2017

realm-ci removed the S:Waiting For User label Apr 27, 2017

kneth added the S:Waiting For User label Apr 27, 2017

realm-ci removed the S:Waiting For User label Apr 27, 2017

kneth added the S:Waiting For User label Apr 28, 2017

realm-ci removed the S:Waiting For User label Apr 28, 2017

cmelchior closed this as completed Apr 28, 2017

github-actions bot locked as resolved and limited conversation to collaborators Mar 16, 2024

io.realm.exceptions.RealmFileException when launching application #4524

io.realm.exceptions.RealmFileException when launching application #4524

Comments

jollyjoker992 commented Apr 20, 2017

Zhuinden commented Apr 20, 2017

jollyjoker992 commented Apr 21, 2017

kneth commented Apr 24, 2017

jollyjoker992 commented Apr 25, 2017 • edited

kneth commented Apr 25, 2017

jollyjoker992 commented Apr 26, 2017

kneth commented Apr 26, 2017

jollyjoker992 commented Apr 27, 2017 • edited

Zhuinden commented Apr 27, 2017

kneth commented Apr 27, 2017

jollyjoker992 commented Apr 27, 2017

kneth commented Apr 28, 2017

jollyjoker992 commented Apr 28, 2017

jollyjoker992 commented May 1, 2017

Zhuinden commented May 1, 2017

cmelchior commented Nov 14, 2017

jollyjoker992 commented Nov 14, 2017

jollyjoker992 commented Dec 2, 2017 • edited

Zhuinden commented Dec 2, 2017 • edited

jollyjoker992 commented Dec 2, 2017

jollyjoker992 commented Dec 2, 2017

Zhuinden commented Dec 2, 2017 • edited

huunam118 commented Dec 8, 2017

Zhuinden commented Dec 8, 2017

huunam118 commented Dec 9, 2017 • edited

jollyjoker992 commented Dec 14, 2017 • edited

beeender commented Dec 14, 2017

jollyjoker992 commented Dec 14, 2017 • edited

beeender commented Dec 14, 2017

jollyjoker992 commented Dec 14, 2017

beeender commented Dec 14, 2017

jollyjoker992 commented Dec 14, 2017

beeender commented Dec 14, 2017

jollyjoker992 commented Dec 14, 2017 • edited

beeender commented Dec 14, 2017

jollyjoker992 commented Dec 14, 2017

beeender commented Dec 14, 2017

jollyjoker992 commented Dec 14, 2017

jollyjoker992 commented Jan 17, 2018

bmunkholm commented Jan 17, 2018 • edited

jollyjoker992 commented Jan 17, 2018

jollyjoker992 commented Jan 17, 2018

cmelchior commented Jan 17, 2018

jollyjoker992 commented Jan 17, 2018

jollyjoker992 commented Apr 25, 2017 •

edited

jollyjoker992 commented Apr 27, 2017 •

edited

jollyjoker992 commented Dec 2, 2017 •

edited

Zhuinden commented Dec 2, 2017 •

edited

Zhuinden commented Dec 2, 2017 •

edited

huunam118 commented Dec 9, 2017 •

edited

jollyjoker992 commented Dec 14, 2017 •

edited

jollyjoker992 commented Dec 14, 2017 •

edited

jollyjoker992 commented Dec 14, 2017 •

edited

bmunkholm commented Jan 17, 2018 •

edited