[Firestore] Connection not recovering quickly after being offline

[ryanjohn1]

[REQUIRED] Step 2: Describe your environment

• Android Studio version: 3.2.1
• Firebase Component: Firestore
• Component version: 17.1.5

[REQUIRED] Step 3: Describe the problem

Firestore currently has an exponential backoff mechanism to prevent reduce wasted resources when an internet connection is lost. The issue is that it can take a long time to come back when the connection is re-established, usually over a minute.
This is especially jarring to the user when their connection comes back so that they get a push notification to inform them of something new (say a message in a chat app), they tap the notification and the screen loads but the message they were just informed about does not arrive through the snapshot listener. If they sit on the screen for a minute or two (very unlikely) then their message will eventually load. This leaves them feeling the product is broken.

I'd seen in the changelog a comment that seems to suggest work has been done on this issue but I'm still seeing similar behaviour. Our iOS client improved dramatically with the equivalent recent change with the message arriving in 0-10 seconds after the internet connection is reestablished.

Steps to reproduce:

1. Add a snapshot listener to some data on Firestore, ensure the data is coming through correctly when data is changed.
2. Turn the device to airplane mode to kill the internet connection
3. Change the data on Firestore
4. Wait 2 minutes
5. Turn airplane mode off
6. (Open app if closed) time how long the changed data takes to arrive

Expected: data arrives within a few seconds after the internet connection is reestablished
Actual: data arrives after 30s to 2mins after internet connection is reestablished

The same behaviour occurs whether or not you stay in the app with the listener attached or remove the listener and background the app. Killing the app completely causes the expected result to occur.

[rsgowman]

Hi @ryanjohn1,

We had a similar report a while back, but were unfortunately unable to reproduce it locally. (Mostly; see below). We implemented a "failsafe" here:

#151

Basically, it's supposed to trigger keepalive packets which should notice a dead connection within 30s. So if grpc/android did get into a bad state, that should cause a recovery within 30s. As a result, I'd expect 30s to be an upper bound of "badness".

The only way we were able to reproduce the problem locally, is by using the android emulator (as opposed to a physical device). Apparently, the emulator doesn't always properly report dead cell connections back to grpc, so we were able to occasionally get into a bad state by playing around with wifi/cell/airplane mode.

That said, the setup was a little different from what you've described. I'll spin up a minimal app here to attempt to reproduce the issue. But in the meantime, could you:

a) confirm if you're using a physical device (as opposed to the emulator). If you're using the emulator, please try to reproduce the issue on a physical device. If you have multiple devices, could you also confirm if you can repro on all of them, or only some of them?
b) approximately how often can you reproduce the issue? 100% of the time?
c) If you're able to share the (preferably minimal!) code to reproduce the issue, that would also be useful.

[ryanjohn1]

Hi @rsgowman,

Thanks for your response, I'll go through your response in order:
a) Yes this is on physical devices and it's not limited to a single device. Its the same on both devices I've tested on. One on Android 9.0 and one on 7.0.
b) I did most of my testing initially with v17.1.2 of Firestore before I found the changelog and upgraded to 17.1.5 and found roughly the same behaviour.
I've done some further testing just now on 17.1.5 I followed the above steps while staying in the app with the listener added while the device went offline for 2 minutes and then timed how long it took to get Firestore data through again after the Wifi symbol suggested it had successfully reconnected and was getting data.
I did 12 repeats and got:
110 seconds, 10 seconds, 22 seconds, 26 seconds, 18 seconds, 65 seconds, 78 seconds, 102 seconds, 70 seconds, 28 seconds, 20 seconds, 30 seconds

c) This was using the full code for the screen which actually added 4 snapshot listeners: one to a Document, two to subcollections of that document and then one to a subcollection of a different collection. I only changed the data to one document in one of the subcollections of the main document.

Is there anything more specific about the code it would be helpful to see?

[rsgowman]

I've got this reproducing just fine locally and can see the problem; nothing further needed from your end. I'll update here once I've got a solution.

[majid701]

I'm noticing the same issue on my end. Our app has an important offline functionality so this is required to work as expected for us. Any suggestions to mitigate the issue or timeline on when it will be fixed?

thanks

[wilhuff]

Currently there's no workaround for this issue. #217 is a proposed fix.

If all goes well this could get into our next release, but as usual we can't promise timelines.

[rsgowman]

#217 has now landed in master and will likely go out in our next release. Anecdotally, #217 improves recovery times to around 2s or less (which is about the time it takes for wifi/cell connections to re-establish themselves), though YMMV.