-
Notifications
You must be signed in to change notification settings - Fork 147
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bandwidth Exhausted while doing Firestore calls through NodeJS Server deployed on Google App Engine. #765
Comments
@zulqarnainhanif Thanks for filing this issue and sorry to hear about your troubles. We have seen other reports internally and believe that the underlying issue is a RESOURCE_EXHAUSTED error from the backend. To debug your case, we will likely need project specific information, which will help us look at the workload at the time you experienced this error. Do you mind sharing your Project ID with us via a customer service request (via https://cloud.google.com/support/)? Thank you! I will close this issue for now as this is likely project specific. |
I am re-opening this for now as @mikelehen tracked down an internal root cause for these error messages. The error seems to only reproduce with Node v10.16.3 and v12.11.0 and is likely caused by nodejs/node#29223 You can work around this by downgrading the network layer for Firestore to an implementation that does not rely on Node's HTTP/2 network stack. This can be done as such:
|
@zulqarnainhanif We were able to confirm that nodejs/node#29223 is indeed the root cause. We hope to get this fixed in the underlying Node implementation soon. |
The fix for the underlying issue in the HTTP2 layer will be part of Node 10.18.1 (nodejs/node#30796), Node 12.3.2 (nodejs/node#30865) and Node 12.4.1 (nodejs/node#31069). We still have to wait for these builds to be released and become available on GCP, but since there is nothing to be done in this library, I am closing this issue for now. Thank you for your patience. |
We are facing the same issue I tried your support link @schmidt-sebastian but it seems there's no place to create a ticket unless we pay for it and upgrade to paid support plans? Can you please let us know if this is something that will be fixed soon? or what solution we should have meanwhile? |
@arashkay Unfortunately, you will have to wait until Node 10.18.1 is deployed to our container frameworks. Per https://cloud.google.com/functions/docs/concepts/exec, the version is currently at 10.15.3. |
This is also happening to me in the Node 11.12 Docker image. |
@ryansydnor When you can switch to one of the Node version ranges mentioned above (Node 10.18.1+, Node 12.3.2+ and Node 12.4.1+), then you can switch back to the main GRPC stack. This should again reduce your binary size and startup time. Do note that Node's version ranges are distinct and that the bug is addressed individually in each version - i.e. the fix in 10.1.8.1 is not part of Node 11.x or Node 12.0. |
If I use current stable node version 12.16.2 will it be sufficient? |
It should be. |
Experiencing a similar issue when performing some reasonably light stress testing on my API which interacts with Firestore using Performing a stress test which replicates 600 users sending 4 requests each arriving at a rate of around 10 users p/s over 60s (2400 API requests in total each of which perform several reads and writes). I am doing such a stress test as I have recently moved to Distributed Counters off the back of an issue I was having in relation to contention issues - see firebase/firebase-admin-node#856 (comment). Approx 2k of these requests succeed as expected and around 400 fail due to the following error:
|
This seems to have resolved the |
@dooleyb1 Hi, how did you solve it ? |
We are seeing this issue randomly occurring in our App Engine Flex NodeJS instances running Node 14.16.1 and Firestore Admin SDK 9.2.0. Once it starts all further Firestore requests fail with |
I had the same problem for reading all data from firebase, after a while, I figured out if we stop this process and run a new process we can pass the error and continue the job. then I used child-process and it helped me:
This is a solution that works 100 percent, but it's a little advance, and beginners-intermediates may not able to implement it. I have written a gist and explained full implemention: |
@navidshad how do you pass the "where to start reading again" part? Do you use some kind of cursor? I tried the same approach with child-process some years ago, it almost worked, but I think I got stuck with cursors (it was |
Actually, I have written a cursor that manages everything relative to that issue. I can put the code here. This is a gist I have written today to explain this solution: |
The following are the dependencies used in my NodeJs project.
While doing query request on firestore, sometime we get the error mentioned below and all the calls to firestore after that starts giving the same error. After another deployment with no changes all the apis works fine. I am still unable to figure out the leading cause of this error.
Can someone please explain how to resolve this issue or handle this error so that it doesn't effect the other apis?
The text was updated successfully, but these errors were encountered: