Firestore networking fails intermittently #8503
Assignees
Labels
Comments
|
It's also common to see a bunch of these errors in the logs as well: FYI, when Firestore reaches the backend immediately, the logs look like this. Pretty much the same, apart from the network errors. |
|
I enabled Firestore logging, and got more comprehensive logs (with sensitive info redacted). And here are the log messages when it finally connects, 6 minutes later. |
|
Hi @kwan-research, the error message you reported is usually caused by no connection from your device to your database, possibly due to network issues. With this, could you provide/try the following:
Thanks. |
|
I've encountered the problem both at home (fibre + wifi) and in a co-working space (wifi). In both locations the wifi is excellent, suitable for video-conferencing. It occurs on both a physical device (iPhone 7) and various simulators. I don't have any proxies in place. If it was a proxy issue, the app would never reach the server, ever. But most of the time it connects immediately, and when it does have a problem it connects eventually (but it can take minutes). Note the Firebase authentication and messaging always work fine. This problem only affects Firestore, and seems to be caused by grpc. I'm not sure if the Quickstart app will reproduce the problem, because it looks like it will run happily in offline mode, so the grpc problems won't affect it. |
|
FYI, this problem doesn't occur in the Android version of the app, which uses equivalent Firestore calls. |
|
@kwan-research Thanks for writing in. Could you try to reproduce the problem on the Quickstart app? While the app does support offline mode, it will continue to try reconnecting with the Firestore servers while it is offline. Can you also try to maintain a connection to the Firestore API endpoint URL separately outside of the SDK? This way, we can be sure that the connection failures are exclusively part of the SDK. Thanks! |
|
Same problem in the Quickstart app. These are the logs when I ran in a simulator (iPhone 11 pro max) and didn't interact with the app. I ran it a second time, waited for the "could not reach backend" message, then successfully set up an account in the app with email and password. But shortly after the DNS resolution failure messages still appeared. So it looks like Firebase Auth is connecting successfully, while Firestore is failing. I'm not sure what you mean by maintaining a connection to the Firestore API endpoint URL. I tried loading this URL in a Chrome browser, and it always immediately failed with the expected 403 permission denied. |
|
It took a few tries, but I got the quickstart to generate the same errors on my iPhone 7 as well. To summarize, connection failures occur on both
It's a long shot, but I'm in Sydney, Australia. Any chance the gRPC issue is triggered by round-trip latency? |
|
A couple more data points
|
|
@kwan-research Thanks for the data points! No obvious cause stands out at the moment, but I will dig into this some more. |
|
Let me know if there are any more tests I can run or logs that I can enable. This is a P1 for me - I don't want to risk App Store submission when launch times can take a few minutes. |
|
@kwan-research I've been trying to reproduce this without success unfortunately. For me, the quickstart app never showed the @rizafran Can you try reproducing the error on quickstart? |
|
Disregard previous (deleted) comment regarding credentials. It was a red herring. I just realized that I can edit Pod files, so I've added log messages to the Firestore code in quickstart and narrowed down the cause. In firebase-ios-sdk/Firestore/core/src/remote/grpc_stream.cc it's taking a really long time to get the first call to GrpcStream::OnRead. So it looks like the problem is further down, in the grpc stack. I'll keep looking. |
|
I enabled logging in the grpc stack, and the problem seems to be in DNS resolution. These are the messages when it finally connects. |
|
Calls to getaddrinfo("firestore.googleapis.com", "https", &hints, &result) fail with EAI_NONAME in 30 seconds. |
|
The underlying issues seems to be grpc's use of The Dart project ran into a similar issue here. It seems that In To reproduce, make sure It sounds like you'll need to punt this to the grpc team. |
|
@kwan-research Thanks for investigating further and posting your findings! Can you open an issue with the grpc github repo directly? That way, the bug can be triaged and handled appropriately by the code-owners. |
|
Punted to grpc/grpc#27092 In case anyone else runs into this problem, a (rather desperate) work-around is to use the |
|
@kwan-research Thanks for sharing the thorough investigation! |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
[REQUIRED] Step 1: Describe your environment
[REQUIRED] Step 2: Describe the problem
Steps to reproduce:
Occasionally (feels like a quarter of the time) when I launch my app it fails to retrieve data from the Firestore server. The snapshot listener is called with cached local data, but nothing from the server. Then, a few minutes later, data arrives successfully.
My app waits for non-cached server data before proceeding, so this is really noticeable. It has been happening all year, but I assumed it was a hardware issue (an old second-hand iPhone 7). But it happens just as often in Xcode simulators.
Note that most of the time Firestore works fine, which makes this bug difficult to reproduce. And yes, I have a very reliable internet connection.
To test the hypothesis that the main thread was blocked, I tried adding a function that calls itself asynchronously every few seconds and prints a message. But it ran fine.
Relevant Code:
Here's a typical log for the app (called Metropolis). It uses Firebase Messaging as well, although that's probably not relevant. And I've seen Firestore take a lot longer than this to reach the server.
The text was updated successfully, but these errors were encountered: