OverlappingFileLockException crashes with up to date libraries versions

[JeremyR34]

• Android Studio version: 3.6.3
• Firebase Components & versions:
  com.google.firebase:firebase-core 17.4.0
  com.google.firebase:firebase-analytics 17.4.0
  com.google.firebase:firebase-messaging 20.1.6
  com.crashlytics.sdk.android:crashlytics 2.10.1
  com.google.firebase:firebase-dynamic-links 19.1.0
  com.google.firebase:firebase-database 19.3.0
  com.google.firebase:firebase-config 19.1.4

Describe the problem

After releasing a new version ni production with up to date Firebase libraries we have many new crashes (Google Play developer console & Firebase Crashlytics).

Steps to reproduce:

Up-to-date libraries versions.
Notes

• it seems to happen only on Android 7 & 8 devices.
• it may be related to this: Fixing the OverlappingFileLockException. #1234

Relevant Code:

Fatal Exception: java.nio.channels.OverlappingFileLockException
       at sun.nio.ch.SharedFileLockTable.checkList(SharedFileLockTable.java:255)
       at sun.nio.ch.SharedFileLockTable.add(SharedFileLockTable.java:152)
       at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1080)
       at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
       at com.google.firebase.installations.CrossProcessLock.acquire(CrossProcessLock.java:52)
       at com.google.firebase.installations.FirebaseInstallations.getPrefsWithGeneratedIdMultiProcessSafe(FirebaseInstallations.java:376)
       at com.google.firebase.installations.FirebaseInstallations.doNetworkCall(FirebaseInstallations.java:327)
       at com.google.firebase.installations.FirebaseInstallations.lambda$doRegistrationInternal$0(FirebaseInstallations.java:323)
       at com.google.firebase.installations.FirebaseInstallations$$Lambda$5.run(FirebaseInstallations.java:4)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
       at java.lang.Thread.run(Thread.java:764)

Another kind of new crashes we have that seems also related (FirebaseInstallations):

Fatal Exception: java.lang.Error: java.io.IOException: Bad file descriptor
       at sun.nio.ch.FileKey.create(FileKey.java:46)
       at sun.nio.ch.SharedFileLockTable.<init>(SharedFileLockTable.java:120)
       at sun.nio.ch.FileLockTable.newSharedFileLockTable(FileLockTable.java:47)
       at sun.nio.ch.FileChannelImpl.fileLockTable(FileChannelImpl.java:993)
       at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1015)
       at java.nio.channels.FileChannel.lock(FileChannel.java:875)
       at com.google.firebase.installations.CrossProcessLock.acquire(CrossProcessLock.java:52)
       at com.google.firebase.installations.FirebaseInstallations.getPrefsWithGeneratedIdMultiProcessSafe(FirebaseInstallations.java:376)
       at com.google.firebase.installations.FirebaseInstallations.doNetworkCall(FirebaseInstallations.java:327)
       at com.google.firebase.installations.FirebaseInstallations.lambda$doRegistrationInternal$0(FirebaseInstallations.java:323)
       at com.google.firebase.installations.FirebaseInstallations$$Lambda$5.run(FirebaseInstallations.java)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at java.lang.Thread.run(Thread.java:760)

Caused by java.io.IOException: Bad file descriptor
       at sun.nio.ch.FileKey.init(FileKey.java)
       at sun.nio.ch.FileKey.create(FileKey.java:44)
       at sun.nio.ch.SharedFileLockTable.<init>(SharedFileLockTable.java:120)
       at sun.nio.ch.FileLockTable.newSharedFileLockTable(FileLockTable.java:47)
       at sun.nio.ch.FileChannelImpl.fileLockTable(FileChannelImpl.java:993)
       at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1015)
       at java.nio.channels.FileChannel.lock(FileChannel.java:875)
       at com.google.firebase.installations.CrossProcessLock.acquire(CrossProcessLock.java:52)
       at com.google.firebase.installations.FirebaseInstallations.getPrefsWithGeneratedIdMultiProcessSafe(FirebaseInstallations.java:376)
       at com.google.firebase.installations.FirebaseInstallations.doNetworkCall(FirebaseInstallations.java:327)
       at com.google.firebase.installations.FirebaseInstallations.lambda$doRegistrationInternal$0(FirebaseInstallations.java:323)
       at com.google.firebase.installations.FirebaseInstallations$$Lambda$5.run(FirebaseInstallations.java)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at java.lang.Thread.run(Thread.java:760)

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[andirayo]

@JeremyR34 : Thank you for the bug report! It's late now in the Bay Area, but we will look into this tomorrow!

[andirayo]

@JeremyR34 :
Thank you for reporting this. We are looking into this.
You correctly identified FirebaseInstallations (firebase-installations) as the cause of these issues. fyi: FirebaseInstallations is an infrastructure service used by many Firebase services (e.g. FirebaseMessaging).

Correct me if I'm wrong: You report 3 crashes.

• [A] The first in your list is an OverlappingFileLockException.
• [B] The other two are java.io.IOException: Bad file descriptor

Case [A] indeed looks identical to an issue we have fixed in the past (as you pointed out yourself: #1234). Are you sure that this exception was correctly attributed to the version numbers you list in the description? Is it possible that the exceptions happened to older versions of your app?
fyi: We are first looking into case [B], the IOExceptions...

[andirayo]

@JeremyR34 :
We identified the issue as a java.lang.Error wrapping a possible IOException for Java versions 7 (>= v40) and Java 8.
Compare: https://stackoverflow.com/questions/35727180/

We will fix this issue by catching java.lang.Error when trying to acquire a CrossProcessLock.
Thanks for bringing this to our attention.
The fix will most likely be released with v16.3.1 of FirebaseInstallations (and FirebaseMessaging v20.1.8) on May 21, 2020.

[andirayo]

This issue should be fixed with #1521.
This update is scheduled to be released on May 21, 2020.

[JeremyR34]

Case [A] indeed looks identical to an issue we have fixed in the past (as you pointed out yourself: #1234). Are you sure that this exception was correctly attributed to the version numbers you list in the description? Is it possible that the exceptions happened to older versions of your app?

I'm sure of the versions, the issue starts happening on a new release rollout. That's why I was surprised to find an old fix related to this.

I reverted my libraries versions on a new update and the issue is gone.
FYI I use:
com.google.firebase:firebase-core 17.2.1
com.google.firebase:firebase-analytics 17.2.1
com.google.firebase:firebase-messaging 20.1.0
com.crashlytics.sdk.android:crashlytics 2.10.1
com.google.firebase:firebase-dynamic-links 19.0.0
com.google.firebase:firebase-database 19.2.0
com.google.firebase:firebase-config 19.0.4

Side note: you give me a release update, but how to check if a particular fix is included in a release. I wanted to check when and if the #1234 has been released but did not find this info. On the https://firebase.google.com/support/release-notes/android page there is only a resume but not full details.

[andirayo]

@JeremyR34 :
Thanks for the update.
Do you think it's possible that the OverlappingFileLockException is just a side-effect of the java.lang.Error: java.io.IOException: Bad file descriptor exception, i.e. that the exceptions are happening on the same devices?

fyi: I reopened the issue and we will look into this on Monday.

[JeremyR34]

No I don't think so. I have less java.lang.Error: java.io.IOException: Bad file descriptor than OverlappingFileLockException exceptions and Crashlytics shows different device models. But both only happened on Android 7 & 8.

OverlappingFileLockException:

java.lang.Error: java.io.IOException: Bad file descriptor

[andirayo]

@JeremyR34 , thanks a lot. That's very helpful!

[andirayo]

@JeremyR34, can you confirm that the java.lang.Error: java.io.IOException are fixed if you compile your application with the latest version of FirebaseInstallations on Github.

Here are instructions how to publish with the latest Github version of a Firebase SDK:
https://github.com/firebase/firebase-android-sdk#publishing

[JeremyR34]

I'm sorry but unfortunately I can't for now. My only way to know if it is fixed is to release a new version of my app, and waiting for crashes to appear. Next release is not already planned and I don't want to try it before and put more users at risk of crashes until that.
And I'm not sure to have the technical skills & knowledge it requires to do it without help (I read the page of your link but don't understand).

[andirayo]

Understood.
In fact we have had an idea why the OverlappingFileLockException might still be happening to you. We will try to get this in for the release on May 21, but we can't promise it.

[JeremyR34]

I don't know if it was gracefully handled by the SDK but I suppose it was not; usually when crashes are logged in the Google Play Console they are real crashes on user devices.

[ankitaj224]

@JeremyR34 Can you please update your app to use the latest version of firebase-messaging:20.2.0 which contains the fix for this issue?

[JeremyR34]

I will. I should release a new version in the next few days so I will be able to try the fix.

[JeremyR34]

Unfortunately the new version does not completely fix the issue.
I had to rollback versions once again.
I do not have java.io.IOException: Bad file descriptor exceptions anymore.

I found 2 different stack traces but issue seems to be always there:
com.google.firebase.installations.FirebaseInstallations.getPrefsWithGeneratedIdMultiProcessSafe(FirebaseInstallations.java:407)

Fatal Exception: java.nio.channels.OverlappingFileLockException
       at sun.nio.ch.SharedFileLockTable.checkList(SharedFileLockTable.java:255)
       at sun.nio.ch.SharedFileLockTable.add(SharedFileLockTable.java:152)
       at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1080)
       at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
       at com.google.firebase.installations.CrossProcessLock.acquire(CrossProcessLock.java:52)
       at com.google.firebase.installations.FirebaseInstallations.getPrefsWithGeneratedIdMultiProcessSafe(FirebaseInstallations.java:407)
       at com.google.firebase.installations.FirebaseInstallations.doRegistrationInternal(FirebaseInstallations.java:320)
       at com.google.firebase.installations.FirebaseInstallations.doGetId(FirebaseInstallations.java:299)
       at com.google.firebase.installations.FirebaseInstallations.access$lambda$0(FirebaseInstallations.java)
       at com.google.firebase.installations.FirebaseInstallations$$Lambda$1.run(FirebaseInstallations.java:2)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
       at java.lang.Thread.run(Thread.java:764)

Fatal Exception: java.nio.channels.OverlappingFileLockException
       at sun.nio.ch.SharedFileLockTable.checkList(SharedFileLockTable.java:255)
       at sun.nio.ch.SharedFileLockTable.add(SharedFileLockTable.java:152)
       at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1080)
       at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
       at com.google.firebase.installations.CrossProcessLock.acquire(CrossProcessLock.java:52)
       at com.google.firebase.installations.FirebaseInstallations.getPrefsWithGeneratedIdMultiProcessSafe(FirebaseInstallations.java:407)
       at com.google.firebase.installations.FirebaseInstallations.doNetworkCall(FirebaseInstallations.java:335)
       at com.google.firebase.installations.FirebaseInstallations.lambda$doRegistrationInternal$0(FirebaseInstallations.java:331)
       at com.google.firebase.installations.FirebaseInstallations$$Lambda$5.run(FirebaseInstallations.java:4)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
       at java.lang.Thread.run(Thread.java:764)

Some stats:

[JeremyR34]

I do not call FirebaseInstanceID delete method either.

[ankitaj224]

Thanks for the confirmation. I tried but couldn't reproduce this issue with the latest version of firebase-installations library, I will need some more details pertaining to your app.

1. Does your have multiple processes?
2. Does your app initialize Firebase using multiple Projects?

[JeremyR34]

It looks like a tricky bug to reproduce.
To answer your questions:

1. No
2. No

[hramchandani]

Hi @ankitaj224. My answers are identical to @JeremyR34. My app does not have multiple processes. My app does not initialize Firebase using multiple projects.

[hramchandani]

Hi @ankitaj224. I would like to provide the following information regarding my recent experience with this OverlappingFileLockException error. Maybe it will help in finding its resolution.

I have one project (Project A) with the following dependencies and versions:
classpath 'com.google.firebase:perf-plugin:1.3.1'
implementation 'com.google.firebase:firebase-messaging:20.2.1'
implementation 'com.google.firebase:firebase-analytics:17.4.3'
implementation 'com.google.firebase:firebase-ads:19.2.0'
implementation 'com.google.firebase:firebase-perf:19.0.7'

I released and encountered the error in question.

I reverted Project A to now have the following dependencies and versions:
classpath 'com.google.firebase:perf-plugin:1.3.0'
implementation 'com.google.firebase:firebase-messaging:19.0.1'
implementation 'com.google.firebase:firebase-core:17.0.0'
implementation 'com.google.firebase:firebase-ads:18.0.0'
implementation 'com.google.firebase:firebase-perf:18.0.1'

I released. It has been some weeks and so far I have not encountered the error in question.

I have another project (Project B)
I released this project with the following dependencies and versions (similar to the versions for Project A when it experienced the error):
classpath 'com.google.firebase:perf-plugin:1.3.1'
implementation 'com.google.firebase:firebase-analytics:17.4.3'
implementation 'com.google.firebase:firebase-messaging:20.2.1'
implementation 'com.google.firebase:firebase-perf:19.0.7'

However, it has been a week since I released Project B and so far I have not encountered the error in question.

Both projects have users using Android 7 and 8. Important to note Project A is used far more than Project B.

For some reason the same dependencies and versions in Project A give this OverlappingFileLockException error but in Project B do not.

Hopefully this information is of some assistance

[ankitaj224]

@hramchandani This is an interesting finding. Thank you so much for reporting an update.

Also, I highly appreciate your patience with this issue, it has been difficult bug to reproduce.

What we know so far:

• OverlappingFileLockException crash is seen only in Android 7 & 8.
• It's random ( Project A runs into this crash but not Project B).
• Projects seeing this crash are not configured with Multiple Firebase Options/ Processes.
• From @JeremyR34 reported stats: this crash happens when the app is in foreground ( & not in the background).

Do you call FirebaseInstanceId.getInstance().getId() in either of your project A/ B explicitly?

FYI: This issue will take a while to figure out due to the complexity in reproducing this issue & my work on wrapping up other high priority issues. But I am definitely brainstorming on how to reproduce this issue.

Thanks.

[hramchandani]

Hi @ankitaj224 I understand it is a difficult error to reproduce and resolve.

Yes. I do call FirebaseInstanceId.getInstance().getId() in Project A, which is the project which gives the OverlappingFileLockException error. I call the addOnCompleteListener to get an updated device token. I do not call it in Project B because I have not updated that project as yet. It still uses FirebaseInstanceId.getInstance().getToken() to get an updated device token.

[JeremyR34]

For information I do not call FirebaseInstanceId.getInstance().getId() in my projects.

[ankitaj224]

Ah, I see. That rules out another possibility because Jeremy's projects don't call getId. I will try & spend some more time to reproduce this issue. I will keep this thread posted on my progress.

[zubinxiong]

I have the same problem. and FirebaseInstanceId is called in my projects like the flowing code.

• firebase-analytics:17.4.3
• firebase-messaging:20.2.1

---

 public void reportFcmDaily() {
        try {
            FirebaseInstanceId.getInstance().getInstanceId()
                    .addOnCompleteListener(task -> {
                        if (!task.isSuccessful()) {
                            return;
                        }

                        String token = task.getResult().getToken();
                        if (!TextUtils.isEmpty(token)) {
                            // upload task
                        }
                    });
        } catch (Exception e) {
        }
    }

[luhongwu]

Hi. I am experiencing the same issue with the following implementations:

implementation 'com.google.firebase:firebase-analytics:17.5.0'
implementation 'com.google.firebase:firebase-messaging:20.2.4'
implementation 'com.google.firebase:firebase-crashlytics:17.2.1'
implementation 'com.google.firebase:firebase-perf:19.0.8'
(com.google.firebase:firebase-installations:16.3.3)

I see it fixed in com.google.firebase:firebase-installations:16.3.1, but not work.

[roberto-lem]

Hi, I'm seeing the same crash on some of our users.
We use the following firebase SDKs and versions:

com.google.firebase:firebase-messaging:20.2.4
com.google.firebase:firebase-analytics:17.4.4

We've noticed it started happening after updating from these versions:

com.google.firebase:firebase-messaging:20.1.0
com.google.firebase:firebase-analytics:17.2.1

This is the stacktrace:

Fatal Exception: java.nio.channels.OverlappingFileLockException
       at sun.nio.ch.SharedFileLockTable.checkList(SharedFileLockTable.java:255)
       at sun.nio.ch.SharedFileLockTable.add(SharedFileLockTable.java:152)
       at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1016)
       at java.nio.channels.FileChannel.lock(FileChannel.java:875)
       at com.google.firebase.installations.CrossProcessLock.acquire(CrossProcessLock.java:52)
       at com.google.firebase.installations.FirebaseInstallations.getMultiProcessSafePrefs(FirebaseInstallations.java:577)
       at com.google.firebase.installations.FirebaseInstallations.doNetworkCallIfNecessary(FirebaseInstallations.java:354)
       at com.google.firebase.installations.FirebaseInstallations.lambda$doRegistrationOrRefresh$2(FirebaseInstallations.java:350)
       at com.google.firebase.installations.FirebaseInstallations$$Lambda$4.run(FirebaseInstallations.java)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at java.lang.Thread.run(Thread.java:761)

It happens to users running Android 7 and 8. See image below

[JeremyR34]

@ankitaj224 Any update?
I was considering updating my lib versions but regarding the recent comments the issue seems still there.

[ankitaj224]

@roberto-lem Thanks for reaching & confirming that this issue persists in the latest versions as well. If you had any success reproducing this issue, please share the steps. Also, What % of app crashes are due to this error?

@JeremyR34 Unfortunately, no. I couldn't make progress on this issue due to other high-priority tasks. Last I tried reproducing this for a multi-threaded, multi-process Firebase app, I couldn't. As its clear from multiple reports this issue still persists in the latest versions & is specific to Android 7 & 8.

I`ll internally reach out to the Android build team to see if they can help. I would request you to please hold on updating till I confirm on what's going on & if we have a fix specific to these Android Versions.

Extremely sorry for the inconvenience & late replies.

[luhongwu]

Any update about this issue? Already crashed about 3000+ now, any firebase compents we used all updated to the lastest version, but issue still there.

implementation 'com.google.firebase:firebase-analytics:17.5.0'  
implementation 'com.google.firebase:firebase-messaging:20.2.4'  
implementation 'com.google.firebase:firebase-crashlytics:17.2.1'
implementation 'com.google.firebase:firebase-perf:19.0.8'    

[luhongwu]

I saw a fix in firebase-installations:16.3.1 that try to catch the error but actually failed? the OverlappingFileLockException not match IOException or Error.
c3d4b09

[JeremyR34]

I am forced to postpone the migration to the new version of Crashlytics because of this bug.
Apparently we have to update our applications before November 15, 2020 otherwise we will no longer receive crash reports afterwards. So there is no much time left.

@ankitaj224 can you check @luhongwu proposal, it seems a quick fix that we can test early, don't you think?

[ankitaj224]

@JeremyR34 Really sorry for the inconvenience. I have raised the priority of this issue with my team and should have some update by early next week. I understand that catching the OverlappingFileLockException is a quick fix. However, we need to evaluate what the impact of catching this exception will be on the Firebase Installation service.

Thanks for sharing the timelines with us. I will keep that in mind when addressing this issue.

[ankitaj224]

Hi @JeremyR34 , @luhongwu , @roberto-lem and others,

Do any of you have an Android bug report for this crash from the user device that you can share with us? This will help us formalize & confirm the theory on what's causing this crash.

Thanks,
Ankita

[JeremyR34]

I don't sorry.

[luhongwu]

Sorry, i dit not

[hramchandani]

Hi @ankitaj224. I know you are working on this issue so I apologize for asking for another update so quickly. I want to go back to @JeremyR34 comment where he states "I am forced to postpone the migration to the new version of Crashlytics because of this bug. Apparently we have to update our applications before November 15, 2020 otherwise we will no longer receive crash reports afterwards". Is there any workaround we can perform to ensure we continue to receive crash reports but maybe update to an intermediate version so this bug does not occur? Thank you for any assistance you can provide.

[ankitaj224]

Hi @hramchandani

Thanks for reaching out. I have already submitted a quick fix into Firebase Installations library that should avoid the occurrence of this bug. However, this fix will be made available externally as part of November 12, 2020 release. Would that suffice as for the workaround you were looking for?

[JeremyR34]

Hi @ankitaj224, do you confirm your fix has been released yesterday? Can you please confirm the lib version?

[JeremyR34]

For information, I released an update with com.google.firebase:firebase-bom:26.1.0, and the issue seems to be fixed, I don't have this crash occuring anymore.

[ankitaj224]

Hi @JeremyR34 ,

Thanks for confirming about the fix. Sorry, I was on vacation & couldn't confirm earlier that the fix has been released as part of latest firebase-installations v 16.3.4

I will be closing this issue. Please feel free to reopen if anyone continues to see this issue.

Thanks,
Ankita

[CCCCoder]

Hi @ankitaj224 ,
I am afraid this crash continues to occur in the latest Firebase installation version 16.3.4.
We use the following firebase SDKs and versions:

api platform('com.google.firebase:firebase-bom:26.1.0') api 'com.google.firebase:firebase-analytics-ktx' api 'com.google.firebase:firebase-crashlytics-ktx' api 'com.google.firebase:firebase-perf-ktx' api 'com.google.firebase:firebase-messaging-ktx'

[ankitaj224]

Hi @CCCCoder

Thanks for reaching out. Could you please attach some stack traces of the crash you are noticing using the latest Firebase installation version 16.3.4. It will be helpful in nailing down what's happening. I find it strange that catching the OverlappingFileLockException isn't working.