Too much contention on these documents

[sshahdev]

firebase sdk keeps throwing this error without actually showing which file is producing it.
It's not possible to trace which file using this log. Along with that, it's not allowing us to catch the exception. So that means as soon as we get this error it crashes the node server.

We are using firebase-admin 8.10.0

{"stack":"Error: 10 ABORTED: Too much contention on these documents. Please try again.\n at Object.callErrorFromStatus (/srv/node_modules/@grpc/grpc-js/build/src/call.js:30:26)\n at Object.onReceiveStatus (/srv/node_modules/@grpc/grpc-js/build/src/client.js:174:52)\n at Object.onReceiveStatus (/srv/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:340:141)\n at Object.onReceiveStatus (/srv/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:303:181)\n at Http2CallStream.outputStatus (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:114:27)\n at Http2CallStream.maybeOutputStatus (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:153:22)\n at Http2CallStream.endCall (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:140:18)\n at Http2CallStream.handleTrailers (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:262:14)\n at ClientHttp2Stream.emit (events.js:198:13)\n at ClientHttp2Stream.EventEmitter.emit (domain.js:448:20)\n at emit (internal/http2/core.js:265:8)\n at process._tickCallback (internal/process/next_tick.js:63:19)","message":"10 ABORTED: Too much contention on these documents. Please try again.","code":10,"details":"Too much contention on these documents. Please try again.","metadata":{"internalRepr":{},"options":{}}}

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[schmidt-sebastian]

Please see the comment here: googleapis/nodejs-firestore#1020

Thank you!

[schmidt-sebastian]

@sshahdev Can you send me your project ID to mrschmidt(at)google.com? Our backend team wants to take a closer look.

[schmidt-sebastian]

Internal bug: b/154364083

[dooleyb1]

I am having this issue in a production environment, we sent out a notification to >3000 users which would have triggered an API request to access a specific document. From looking through the logs, this error caused some of our users (~130+ from what I can see) requests to fail as a result of this error which prevented them from carrying out expected functionality within our application.

@schmidt-sebastian I can provide further information to you if needs be for investigation.

[schmidt-sebastian]

@dooleyb1 Did you write to this document at the same time or had a transaction open that held a lock on this document (by calling Transaction.get())?

[dooleyb1]

@schmidt-sebastian Yes, the document represents a survey that users are trying to start simultaneously. We update a counter on the document to the number of users currently doing the survey as we have an upper limit on the number of users who can complete a given survey so want to prevent access to the document once this number has been reached. From reading transaction documentation I believed this was how I should handle such a use case.

A simple version of the transaction is as follows:

firestore.runTransaction(transaction => {
    return transaction.get(surveyRef)
        .then(async surveyDoc => {
            let surveyData = surveyDoc.data();

            // Get the availableQuotaPaths for user
            const userDoc = await firestore.collection('users').doc(uid).get();
            const usersEligibleQuotaPaths = await getUsersQuotaPathsForBundle(userDoc.data(), surveyDoc);

            // If the user has no availableQuotaPaths, don't let them start
            if (usersEligibleQuotaPaths.length <= 0) {
                return Promise.reject(`Capacity not available for bundle ${surveyId} for user ${uid}`);
            }

            // If there is quota paths for the user, select the one with highest weight and return questions
            let currentBestWeight = 0;
            let optimalQuotaPath = null;

            for (let index in usersEligibleQuotaPaths) {
                let quotaPathToCheck = usersEligibleQuotaPaths[index];

                let weight = surveyData['quotas'][quotaPathToCheck]['weight'];

                // If better weight at this quotaPath, update selection
                if (weight > currentBestWeight) {
                    optimalQuotaPath = quotaPathToCheck;
                    currentBestWeight = weight;
                } 
            }

            // Update the quota count for quotaPath
            transaction.update(surveyRef, {
                [`userQuotaPathMap.${uid}`]: optimalQuotaPath,
                [`quotas.${optimalQuotaPath}.currentQuota`]: admin.firestore.FieldValue.increment(1),
                [`quotas.${optimalQuotaPath}.usersInBundle`]: admin.firestore.FieldValue.increment(1),
                usersInBundle: admin.firestore.FieldValue.increment(1)
            });

            // Return the questions
            const surveyQuestionCollection = await firestore.collection('activeSurveys').doc(surveyId).collection('questions').get();
            let surveyQuestions = {};

            surveyQuestionCollection.docs.forEach(questionDoc => {
                surveyQuestions[questionDoc.id] = questionDoc.data();
            });

            return Promise.resolve(surveyQuestions);
        })
}).then(surveyQuestions => {
    // Handle transaction result here
})

[schmidt-sebastian]

A transactional lock on a document can cause contention errors to show up on reads for the document, since the read needs to access the same locked resource. Can you use a different document for your counter?

[dooleyb1]

A transactional lock on a document can cause contention errors to show up on reads for the document, since the read needs to access the same locked resource. Can you use a different document for your counter?

Yes I could use a different document but I'm not sure if this will solve my issue. It is likely that I will need to update this counter more than once per second as users open push notifications and from reading around this seems to be the update limit on Firebase is this correct? If so, how should one handle the necessity to update a document more than once per second?

[varun-git28]

Im also getting this error.
{ Error: Too much contention on these documents. Please try again.
at Http2CallStream.call.on (/srv/node_modules/@grpc/grpc-js/build/src/client.js:101:45)
at emitOne (events.js:121:20)
at Http2CallStream.emit (events.js:211:7)
at process.nextTick (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)
at _combinedTickCallback (internal/process/next_tick.js:132:7)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
code: 10,
details: 'Too much contention on these documents. Please try again.',
metadata: Metadata { options: undefined, internalRepr: Map {} } }