Firestore/triggers grpc-js No connection established errors

[clytras]

Related issues

googleapis/nodejs-firestore#768

firebase/firebase-js-sdk#1783

grpc/grpc-node#1027

Version info

node: 10.15.1

firebase-functions: 3.3.0

firebase-tools: 7.6.0

firebase-admin: 8.6.0

Test case

exports.addDriverLocation = functions.region(region).firestore
  .document('drivers/{userId}/locations/{locationId}')
  .onCreate((snap, context) => {
    const { userId = -3 } = context.params;
    const record = snap.data();
    const { location = {}} = record;
    const {
      uuid,
      coords: {
        latitude,
        longitude
      } = {},
      extras
    } = location;

    return snap.ref.set({
      userId,
      // server timestamp
      sts: admin.firestore.Timestamp.fromDate(new Date()),
      // client timestamp
      cts: admin.firestore.Timestamp.fromDate(new Date(timestamp)),
      latitude,
      longitude,
      extras
    });

Steps to reproduce

1. Create a firestore trigger function

Expected behavior

Every call should be get executed without gRPC related errors.

Actual behavior

Every now and then we get a No connection established error:

Error: No connection established
    at Http2CallStream.call.on (/srv/functions/node_modules/@grpc/grpc-js/build/src/client.js:96:45)
    at Http2CallStream.emit (events.js:194:15)
    at Http2CallStream.EventEmitter.emit (domain.js:459:23)
    at process.nextTick (/srv/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

It encountered 8600 exact errors for the past week:

For every other firestore operation, we used old grpc as described here googleapis/nodejs-firestore#768 (comment):

const Firestore = require('@google-cloud/firestore');
const grpc = require('grpc');
const firestore = new Firestore({ grpc });

but we don't know how to set old grpc for functions.region(region).firestore to get rid of all these errors.

We are not sure if here is the right place to post this issue or the https://github.com/googleapis/nodejs-firestore. If this is the case, the please close this issue and we'll go and report it on the nodejs-firestore repo.

Were you able to successfully deploy your functions?

No errors during functions deploy.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[shankie-codes]

I'm also having this issue – just from today.

[LureOfAdventure]

I've been having the same issues as well

[mrmodelo]

Issue here as well when deploying a scheduled function. Works for about an hour then crashes. There are other threads going on but fail to address the firebase scheduled function side of things.

[tantzygames]

I've also been getting this error intermittently yesterday and today on a range of https.onCall functions

[diginikkari]

Getting also many occurencies of this error last few days with firestore triggers.

trigger: firestore create trigger.
runtime: Node.js 8
Last deployed: 16 October 2019 at 13:55:30 UTC+3
Region: europe-west1

firebase packages:
"firebase": "^7.1.0",
"firebase-admin": "8.6.0",
"firebase-functions": "^3.2.0",
"firebase-tools": "^7.5.0",

[clytras]

@mrmodelo this occurs to every kind of function that involves firestore and it's related to firestore changing grpc to grpc-js. There is a workaround for other kind of functions as you can see on my initial post, but I can't find a way to set old grpc for firestrore funtion triggers.

[adamkpurdy]

Seeing the same error on Firestore delete triggers. I just started noticing this issue this morning.

"firebase-admin": "^8.0.0",
"firebase-functions": "^3.1.0",

Region: North America / east-coast EST

[nourziady]

Same issue here, started to get No connection established error week ago

[adamkpurdy]

This looks resolved. My triggers are firing again. Can anyone else confirm?

[jhk115]

@adamkpurdy It seems to be sporadically failing on our project still.

[mrmodelo]

Since first seeing the problem, it will usually pass once after deployment, and then fail consistently after

[clytras]

@adamkpurdy since last functions deploy, I can't see any errors, but for me it's too early to tell. I'll leave this open for some days and if there won't be any more related errors I'll close this.

[adamkpurdy]

I'm seeing the error again.

[LureOfAdventure]

I am still experiencing this issue as well.

[jhk115]

Trying to narrow down our issue - are you guys doing anything with storage inside these functions?

[glawson]

I was experiencing this since yesterday morning. I found this thread about 30 minutes ago, figured since grpc was mentioned I'd reinstall. I blew away node_modules, package-lock.json, then reinstalled and redeployed. Everything is back to normal so far. I haven't checked what is any package changed yet, so fingers crossed.

[LureOfAdventure]

@jhk115 I'm not doing anything with storage. Only Firestore.

[adamkpurdy]

@jhk115 only Firestore

[mrmodelo]

@jhk115 This is because of a dependency that firestore is using called grpc-js as mentioned by @clytras

@mrmodelo this occurs to every kind of function that involves firestore and it's related to firestore changing grpc to grpc-js. There is a workaround for other kind of functions as you can see on my initial post, but I can't find a way to set old grpc for firestrore funtion triggers.

There have been workarounds referenced if you look further back in the thread and to the linked issues on the original comment, however there are still situations where you can not manage this dependency (to our knowledge so far)

[adamkpurdy]

Hey peoples:

I just did a dep upgrade after reading grpc/grpc-node#1027:

    "firebase-admin": "^8.6.0",
    "firebase-functions": "^3.3.0"

I've had three successful triggers fired. Anybody else wanna try and post their results?

[michaeltroya]

@adamkpurdy Upgraded also, working for me so far!

[mrmodelo]

I had a rogue reference in my package-lock.json

After nuking that and starting from a clean slate I was able to get the correct grpc-js version. For anyone else running into this issue right now, you need to delete BOTH your node_modules and your package-lock.json

For those interested it was under google-gax as a required dependency. It should read:

@grpc/grpc-js": "0.6.9"

I have since had three scheduled calls go off with no failures. Fingers crossed this is the one, thanks for the timely responses from all and the community support!

[jhk115]

It looks promising on our end! Thanks, everyone.

[floriangbh]

Updating from "firebase-functions": "^3.2.0" to "firebase-functions": "^3.3.0" and re deploy worked for me !
Thanks everyone.

[floriangbh]

Finally the problem came back after 1h... Waiting for a fix !

[mrmodelo]

@floriangbh please go into package-lock.json and see which grps version is being referenced. Also may be good to do clean slate and delete your node_modules and package-lock.json and run an npm install

[adamkpurdy]

@floriangbh I did the same as @Mrmodel is mentioning. I haven't seen the issue since, and I've been firing delete triggers since yesterday without a problem. Please post your results after a clean slate.

[floriangbh]

Yes I confirm this is ok since yesterday after the cleanup ! Thanks

[ewindso]

I still experience this, running firebase-functions 3.3.0 and firebase-admin 8.6.0. I tried deleting node_modules and package-lock.json and reinstalling, but still got grpc-js 0.6.4...

[OneHoopyFrood]

Dealing with this same issue just trying to use firestore. Was using @google-cloud/firestore like OP, but tried switching to firebase-admin to see if it helped. No dice. I get a few successful calls thorugh when the service is newly launched, but after a few minutes, I begin getting this error message.

[adamkpurdy]

@ewindso & @colepanike I just checked my collections for the proper triggers, and all is still firing as expected. I hate to ask the basics but are you sure you're deleting your /src/functions/node_modules & /src/functions/package-lock.json, updating your deps, and pushing your changes that include your new deps in your /src/functions/package.json to your firebase app? Again, I know these are basic questions, but I am not having the failure in my functions console, and my firestore data is updating as expected via the triggers.

[OneHoopyFrood]

Nope, you were right to do your due diligence. I hadn't gotten a fully clean state. Looks like the workaround does work for me.

[LureOfAdventure]

Workaround worked for me, boys!

[ewindso]

@adamkpurdy all good -- yes i did that, however i used yarn... I did it again and went back to npm and it seems to work now. Thanks!

[PVSPHANINDRA]

Why this error happened from no where in my project like 3 days back and the functions are completely working fine now even though i didn't make any changes to package.json.
I like to know the reason behind the error.

[VitaliyPtitsyn]

call functions stop work for no reason. just thro "No connection established" and out function execution to 36secons,
error { Error: No connection established
at Http2CallStream.call.on (/srv/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
at emitOne (events.js:121:20)
at Http2CallStream.emit (events.js:211:7)
at process.nextTick (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
at _combinedTickCallback (internal/process/next_tick.js:132:7)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
code: 14,
details: 'No connection established',
metadata: Metadata { internalRepr: Map {}, options: {} } }

Function execution took 35381 ms, finished with status code: 500	more_vert

[jeremiahjoven]

Also having the same exact errors and don't know how to fix.

Failure: { Error: No connection established
at Http2CallStream.call.on (/srv/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
at emitOne (events.js:121:20)
at Http2CallStream.emit (events.js:211:7)
at process.nextTick (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
at _combinedTickCallback (internal/process/next_tick.js:132:7)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
code: 14,
details: 'No connection established',
metadata: Metadata { internalRepr: Map {}, options: {} } }

[jeremiahjoven]

Updating Firebase FireFunctions worked

[andyshear]

I am still receiving this error at random after a full clean and update...

Trigger types: cloud storage, firestore create/update, http, and scheduled

Error: No connection established at Http2CallStream.call.on (......)

Removed node-modules and package-lock

Runtime node 10

Here is updated package.json dependencies

"dependencies": {
    "@google-cloud/storage": "^4.0.0",
    "firebase-admin": "8.6.1",
    "firebase-functions": "^3.3.0",
    "fs-extra": "^8.1.0",
    "request": "^2.88.0",
    "sharp": "^0.23.1"
  },

package-lock

"@grpc/grpc-js": {
      "version": "0.6.9",
      "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-0.6.9.tgz",
      "integrity": "...",
      "optional": true,
      "requires": {
        "semver": "^6.2.0"
      }
    },
    "@grpc/proto-loader": {
      "version": "0.5.2",
      "resolved": "https://registry.npmjs.org/@grpc/proto-loader/-/proto-loader-0.5.2.tgz",
      "integrity": "...",
      "optional": true,
      "requires": {
        "lodash.camelcase": "^4.3.0",
        "protobufjs": "^6.8.6"
      }
    },

"google-gax": {
      "version": "1.7.5",
      "resolved": "https://registry.npmjs.org/google-gax/-/google-gax-1.7.5.tgz",
      "integrity": "...",
      "optional": true,
      "requires": {
        "@grpc/grpc-js": "0.6.9",
        "@grpc/proto-loader": "^0.5.1",
        "abort-controller": "^3.0.0",
        "duplexify": "^3.6.0",
        "google-auth-library": "^5.0.0",
        "is-stream-ended": "^0.1.4",
        "lodash.at": "^4.6.0",
        "lodash.has": "^4.5.2",
        "node-fetch": "^2.6.0",
        "protobufjs": "^6.8.8",
        "retry-request": "^4.0.0",
        "semver": "^6.0.0",
        "walkdir": "^0.4.0"
      }
    },

Clean build and deploy last night, scheduled functions received this error on about 20% of its calls

Any thoughts? Going to try another clean/deploy today, will update if I make progress.

[clytras]

@andyshear make sure you npm install after deleting node_modules and package-lock.json and then redeploy everything by just using firebase deploy.

I will close this because I haven't encountered not a single one No connection established since I deleted package-lock.json reinstall and redeploy as @mrmodelo suggest:

I had a rogue reference in my package-lock.json

After nuking that and starting from a clean slate I was able to get the correct grpc-js version. For anyone else running into this issue right now, you need to delete BOTH your node_modules and your package-lock.json

For those interested it was under google-gax as a required dependency. It should read:

@grpc/grpc-js": "0.6.9"

I have since had three scheduled calls go off with no failures. Fingers crossed this is the one, thanks for the timely responses from all and the community support!

[andyshear]

@clytras I do a npm install after full resource delete otherwise it would never build ;)

I received the no connection error yesterday afternoon 1430 MST, this is after 24hr and one full delete/re-install cycle.. Second delete and re-install was around the time of my comment yesterday. So I continued to receive errors for 4hr after that, then nothing..

I have not seen the error since, leading me to believe something was updated on Googles end (cached version of my package-lock.json possibly??) Just odd to me that the problem wasnt resolved on deployment of a "clean" package.

I will open another issue if this problem persists.

[iustin-nita]

This issue is not solved. Deleting package-lock.json and npm modules doesn't solve the problem.

[chardos]

I'm also getting the same issue using firebase-admin v8.10.0. Fails consistently, then works consistently a few times, then continues to fail.

[jeremiahjoven]

Since this wouldn't work on my Firestore project, we got it to work with npm exif-js and blueimp-load-image. But you have to allow data CORS. https://www.agiratech.com/how-to-fix-cors-issue-in-firebase-storage/

Basically after you allow CORS on your storage bucket, you load the image url, orient with Exif-JS and then display on canvas with the correct orientation.

A lot of process is involved. Sadly, the latest MacOS 10.15.4 update bricked the EXIF again on Safari.

[samikm]

Hi , Has anyone found a solution to this ? I am facing a similar problem. I am reading a CSV file and deleting some records mentioned in the file . The program runs fine in the beginning and then throws Error: 14 UNAVAILABLE: No connection established.
My code is below

const csv = require('csv-parser')
const fs = require('fs')
const results = [];

var admin = require("firebase-admin");

var serviceAccount = require("D:\NodeJS\FirestoreConnection1\trackkarona-firebase-adminsdk-lr090-3d8f687f9b.json");

admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://trackkarona.firebaseio.com"
});

let db = admin.firestore();

let drugRef = db.collection('Med_Data_Test3');

let drugDocumentId = '';
var drugData ;
var FAQDoc ;
var waitTill ;
seconds = 1;

fs.createReadStream('onemg_main_faq.csv')
.pipe(csv())
.on('data', (data) => {
let query = drugRef.where('Brand_Name', '==', data.Brand_Name).get()
.then(snapshot => {

if (snapshot.empty) {
console.log('No matching documents.');
return;
}

snapshot.forEach(doc => {
	        drugDocumentId  = doc.id ;
			drugData = doc.data();
 			console.log('data : ', drugDocumentId , drugData.Brand_Name );  
			
			
			FAQDoc = db.collection('Med_Data_Test3').doc(drugDocumentId).collection('FAQs').get()
			 .then(snapshot => {
				   if (snapshot.empty) {
					console.log('No matching FAQ documents.');
					return;
					} 
					waitTill = new Date(new Date().getTime() + seconds * 1000);
					while(waitTill > new Date()){}
					snapshot.forEach(doc => {
					let deleteFAQ = db.collection('Med_Data_Test3').doc(drugDocumentId).collection('FAQs').doc(doc.id).delete();
                    console.log('Deleted FAQ' , doc.id);
					})
			 })
			  .catch(err => {
				console.log('Error getting documents', err);
			});
			
							
                  })
})
	.catch((error) => {
		  console.error(error);
		});

	 }) // on data

.on('end', () => {
console.log('Completed reading CSV file');
});

[samstr]

This started popping up again today for me inside a cloud function.

Error: 14 UNAVAILABLE: No connection established
    at Object.callErrorFromStatus (/var/task/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/var/task/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/var/task/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/var/task/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/var/task/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/var/task/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/var/task/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at Http2CallStream.cancelWithStatus (/var/task/node_modules/@grpc/grpc-js/build/src/call-stream.js:457:14)
    at ChannelImplementation.tryPick (/var/task/node_modules/@grpc/grpc-js/build/src/channel.js:237:32)
    at Object.updateState (/var/task/node_modules/@grpc/grpc-js/build/src/channel.js:106:26)
Caused by: Error
    at WriteBatch.commit (/var/task/node_modules/@google-cloud/firestore/build/src/write-batch.js:415:23)
    at DocumentReference.update (/var/task/node_modules/@google-cloud/firestore/build/src/reference.js:378:14)
    at _callee9$ (/var/task/node_modules/elusive/dist/utils-f06f54c4.js:277:69)
    at tryCatch (/var/task/node_modules/elusive/dist/asyncToGenerator-7a28bf2e.js:61:40)
    at Generator.invoke [as _invoke] (/var/task/node_modules/elusive/dist/asyncToGenerator-7a28bf2e.js:290:22)
    at Generator.prototype.<computed> [as next] (/var/task/node_modules/elusive/dist/asyncToGenerator-7a28bf2e.js:113:21)
    at asyncGeneratorStep (/var/task/node_modules/elusive/dist/asyncToGenerator-7a28bf2e.js:752:24)
    at _next (/var/task/node_mo

[rndlaine]

Still having this issue. Clearing the node_modules and reinstalling the deps didn't work. Any idea on what's causing this?

[j127]

I don't know if it helps anyone, because I'm not using grpc-js, but I got an error in my Firebase function logs:

Exception from a finished function: Error: 14 UNAVAILABLE: No connection established

It looks like the function had spontaneously stopped working around 11:30pm last night.

I found this issue thread and ran npm update to update the dependencies. I redeployed, and it started working again.

These were the packages that got updated:

• firebase-functions from "^3.6.1" to "^3.7.0"
• typescript from "^3.8.0" to "^3.9.5"
• also one of the @types packages, but I don't think that would have affected it.

[milan-tribe-ai]

Just for a record, I started to experience the same exact problem 3 days ago. Clean slate does not really help. Spontaneously some function will run but the majority will fail. The logs output snippet is this: "events.js:174

    throw er; // Unhandled 'error' event
    ^

Error: 14 UNAVAILABLE: No connection established
at Object.callErrorFromStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
at Object.onReceiveStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/client.js:327:49)
at Object.onReceiveStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181)
at Http2CallStream.outputStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:117:74)
at Http2CallStream.maybeOutputStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:156:22)
at Http2CallStream.endCall (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:142:18)
at Http2CallStream.cancelWithStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:467:14)
at ChannelImplementation.tryPick (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/channel.js:238:32)
at ChannelImplementation._startCallStream (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/channel.js:268:14)
at Http2CallStream.start (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:445:22)
Emitted 'error' event at:
at Object.onReceiveStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/client.js:327:28)
at Object.onReceiveStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181)

No workaround found yet that works for rme.

[salvadorbarb1]

Experiencing same issue here, recently created project, testing on emulators suite.

[othondev]

Same error here. No solution yet?