🐛 [@firebase-function-kits/firestore-incremental-capture] Task queue dispatch fails with "Queue does not exist" due to double kit instance prefixing in tasks.ts

[shettyvarun268]

[REQUIRED] Step 2: Describe your configuration

• Extension name: @firebase-function-kits/firestore-incremental-capture
• Extension version: 0.0.2-rc.2 (and 0.0.2-rc.1)
• Configuration values (redact info where appropriate):
  • LOCATION: us-central1
  • SYNC_COLLECTION_PATH: posts
  • SYNC_DATASET: firestore_export
  • SYNC_TABLE: posts_incremental
  • BACKUP_INSTANCE_ID: backup-inst-1
  • DATASET_LOCATION: us
  • DATAFLOW_REGION: us-central1
  • BUCKET_NAME: <project-id>.firebasestorage.app
  • INSTANCE_ID: firestore-incremental-capture

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

1. Install the kit into a project:

   firebase functions:kits:install @firebase-function-kits/firestore-incremental-capture@next

2. Configure .env with a watched collection (e.g. SYNC_COLLECTION_PATH=posts, SYNC_DATASET=firestore_export, SYNC_TABLE=posts_incremental).
3. Deploy the kit functions:

   firebase deploy --only functions:firestore-incremental-capture

4. Mutate or create a document in the watched collection (e.g. posts/test-doc-1).
5. Observe the function logs in Cloud Logging for kit-firestore-incremental-capture-syncdata.

Expected result

syncData should successfully enqueue a task to syncChangelogTask via Cloud Tasks. The queue projects/<project>/locations/us-central1/queues/kit-firestore-incremental-capture-syncChangelogTask receives the payload, dispatches it to the function, and writes the changelog row to BigQuery.

Actual result

syncData fails with Error: Queue does not exist:

Error: Queue does not exist. If you just created the queue, wait at least a minute for the queue to initialize.
    at FunctionsApiClient.toFirebaseError (/workspace/node_modules/@firebase-function-kits/firestore-incremental-capture/node_modules/firebase-admin/lib/functions/functions-api-client-internal.js:421:16)
    at FunctionsApiClient.enqueue (/workspace/node_modules/@firebase-function-kits/firestore-incremental-capture/node_modules/firebase-admin/lib/functions/functions-api-client-internal.js:177:32)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async TaskQueue.enqueue (/workspace/node_modules/@firebase-function-kits/firestore-incremental-capture/node_modules/firebase-admin/lib/functions/functions.js:135:13)
    at async enqueue (/workspace/node_modules/@firebase-function-kits/firestore-incremental-capture/lib/tasks.js:55:5)
    at async handleDocumentWrite (/workspace/node_modules/@firebase-function-kits/firestore-incremental-capture/lib/handlers.js:120:5)

Root Cause Analysis

In src/tasks.ts (and compiled lib/tasks.js):

export function queueName(
  config: ResolvedCaptureConfig,
  functionName: string
): string {
  const region = config.location || process.env.FUNCTION_REGION;
  if (!region) {
    throw new Error("A region is required to resolve task queues.");
  }
  return `locations/${region}/functions/kit-${config.instanceId}-${functionName}`;
}

queueName() manually formats the resource name by prepending kit-${config.instanceId}-.

However, in firebase-admin/functions (v14.1.0+), FunctionsApiClient.resolveResourceId() already automatically inspects process.env.FIREBASE_KIT_INSTANCE_ID (which the Firebase CLI injects into the container environment) and prepends kit-${FIREBASE_KIT_INSTANCE_ID}-:

// firebase-admin/lib/functions/functions-api-client-internal.js
resolveResourceId(resourceId, scope = { scope: 'current' }) {
    switch (scope.scope) {
        case 'current': {
            const kitInstanceId = process.env.FIREBASE_KIT_INSTANCE_ID;
            if (validator.isNonEmptyString(kitInstanceId)) {
                return {
                    resourceId: `kit-${kitInstanceId}-${resourceId}`,
                    extensionOrKitId: kitInstanceId,
                };
            }
            return { resourceId };
        }
    }
}

Because queueName() already included kit-${config.instanceId}-, the Admin SDK double-prefixes the resource name:

kit-firestore-incremental-capture-kit-firestore-incremental-capture-syncChangelogTask

Cloud Tasks receives:

POST https://cloudtasks.googleapis.com/v2/projects/<project>/locations/us-central1/queues/kit-firestore-incremental-capture-kit-firestore-incremental-capture-syncChangelogTask/tasks

Because that double-prefixed queue does not exist, Cloud Tasks returns 404 Not Found, and the Admin SDK converts it into Error: Queue does not exist.

Proposed Fix

In src/tasks.ts, remove the manual kit-${config.instanceId}- prefix:

 export function queueName(
   config: ResolvedCaptureConfig,
   functionName: string
 ): string {
   const region = config.location || process.env.FUNCTION_REGION;
   if (!region) {
     throw new Error("A region is required to resolve task queues.");
   }
-  return `locations/${region}/functions/kit-${config.instanceId}-${functionName}`;
+  return `locations/${region}/functions/${functionName}`;
 }

Verification

We tested this exact patch locally in a project . With the manual prefix removed, task enqueuing succeeded immediately, and live Firestore CREATE, UPDATE, and DELETE mutations streamed cleanly into BigQuery table posts_incremental.

[IzaakGough]

@shettyvarun268 Thanks for the detailed write-up on this!

You're exactly right on this and dropping the kit-${config.instanceId}- prefix is the correct fix.
tests/tasks.test.ts asserts the double-prefixed names, so those can go too.

I did some digging and noticed firestore-vector-search has the same pattern: resolveQueueNames() in src/export-config.ts builds all four queue names with the prefix and passes them to getFunctions().taskQueue().

Fix for both kits coming shortly.