Added field in under Extension configuration did not trigger Typesense update

[junecv]

Description

Added field in under Extension configuration did not trigger document updates in Typesense

Steps to reproduce

Under Extension Configuration in Firebase extension, add a field to the existing list of fields under "Firestore Collection Fields (Optional)".

Expected Behavior

Expect the /typesense_sync/backfill/trigger:true collection takes care of the change.

Actual Behavior

There was no attempt by the extension function to update any document in Typesense. Tried to remove and recreate the backfill collection, still nothing. The change is only reflected when a document is actually updated in firebase.

Metadata

Typesense Version: <1.2.0

OS: MacOS 14.2.1 (23C71)

[jasonbosco]

When you changed the extension configuration, could you confirm that Firebase re-deployed both the backfill function and the indexingOnWrite function?

Could you try re-installing the extension from scratch with the new field added, and see if that helps?

Also, do you see any logs in the backfill functions in the Firebase web console, showing any errors?

[junecv]

Apologies I said there was no attempt by the extension function to update. I couldn't find anything under indexToTypesenseOnFirestoreWrite function log, but now I found below under backfillToTypesenseFromFirestore function log. Let me know if you find something below, and still want me to try re-installing. I probably will try with a new, smaller setup as I don't want to disrupt the live app.

--- backfilling function triggered ---

{
  "textPayload": "Backfilling [a, b, c (last one newly added)] fields in Firestore documents from products into Typesense Collection products on xyz.typesense.net",
  "insertId": "insertID",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "projectID",
      "function_name": "ext-firestore-typesense-search-backfillToTypesenseFromFirestore"
    }
  },
  "timestamp": "2024-01-10T04:09:17.988118Z",
  "severity": "INFO",
  "labels": {
    "execution_id": "executionID",
    "runtime_version": "nodejs16_20231231_16_20_2_RC00",
    "instance_id": "instanceID"
  },
  "logName": "projects/projectID/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/projectID/traces/traceID",
  "receiveTimestamp": "2024-01-10T04:09:18.307724632Z"
}

Following that is an error:

Import error ImportError: 0 documents imported successfully, 1000 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
    at ImportError.TypesenseError [as constructor] (/workspace/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ImportError (/workspace/node_modules/typesense/lib/Typesense/Errors/ImportError.js:25:28)
    at Documents.<anonymous> (/workspace/node_modules/typesense/lib/Typesense/Documents.js:154:39)
    at step (/workspace/node_modules/typesense/lib/Typesense/Documents.js:48:23)
    at Object.next (/workspace/node_modules/typesense/lib/Typesense/Documents.js:29:53)
    at fulfilled (/workspace/node_modules/typesense/lib/Typesense/Documents.js:20:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  importResults: [
    {
      code: 409,
      document: '{"document content"}',
      error: 'A document with id <DocID> already exists.',
      success: false
    },
.... 

Then these two messages:

Done backfilling to Typesense from Firestore
Function execution took 9275 ms, finished with status: 'ok'

And this error again:

Import error ImportError: 0 documents imported successfully, 7366 documents failed during import. Use `error.importResults` from the raised exception to get a detailed error reason for each document.
    at ImportError.TypesenseError [as constructor] (/workspace/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ImportError (/workspace/node_modules/typesense/lib/Typesense/Errors/ImportError.js:25:28)
    at Documents.<anonymous> (/workspace/node_modules/typesense/lib/Typesense/Documents.js:154:39)
    at step (/workspace/node_modules/typesense/lib/Typesense/Documents.js:48:23)
    at Object.next (/workspace/node_modules/typesense/lib/Typesense/Documents.js:29:53)
    at fulfilled (/workspace/node_modules/typesense/lib/Typesense/Documents.js:20:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  importResults: [
    {
      code: 409,
      document: '{"document content"}',
      error: 'A document with id <Doc ID> already exists.',
      success: false
    },
...

[jasonbosco]

Ah this is because during a backfill, we do action=create, instead of action=upsert, so Typesense throws an error saying the document being backfilled already exists.

We didn't account for this use-case of backfilling a 2nd time when a new field is added to an existing collection. Will push out an update for this shortly.

[junecv]

Yay! Thanks for arranging update for this. That would be very useful. A kind note, please also take into account the user-case that a field is removed (if that's not too much to ask).

[jasonbosco]

Could you try installing this pre-release version of the extension and let me know how it goes:

https://console.firebase.google.com/project/_/extensions/install?ref=typesense/firestore-typesense-search@1.3.0-rc.0

A kind note, please also take into account the user-case that a field is removed (if that's not too much to ask).

Yup, action=upsert does this automatically

[junecv]

I did, below steps and result.

1. setup a new collection in firestore (3 docs, 3 fields)
2. setup a new collection in typesense
3. installed 1.3.0-rc.0, in config, included only 2 of 3 fields in Firestore Collection Fields
4. upon completion, nothing show up in typesense, even though I had a typesense_sync backfill collection in place (before this whole setup)
5. deleted the typesense_sync backfill collection, created a new one
6. all docs appeared in typesense, with 2 fields
7. reconfigured the extension to include the third field
8. upon completion, typesense is not updated, all docs still have 2 fields only
9. repeated step 5
10. all docs in typesense updated, with 3 fields

Edit: no logs available

[jasonbosco]

4. upon completion, nothing show up in typesense, even though I had a typesense_sync backfill collection in place (before this whole setup)

This is expected, because only after the extension is installed, it starts listening for the creation of the typesense_sync backfill collection. It does not look at an existing typesense_sync collection.

8. upon completion, typesense is not updated, all docs still have 2 fields only

This is also expected, since the backfill only runs when explicitly triggered by creating the typesense backfill collection. It does not run automatically when the extension is re-configured / installed / re-installed.

So it sounds like things are working as designed, since all docs in Typesense now have 3 fields in Step 10?

[junecv]

Yes. I also tried to remove one field in config, and all work out as expected. Thanks for your lightning-fast communication and implementation!

[jasonbosco]

Happy to help! And thank you for all the feedback.

I'll publish this version live shortly.