Include Document Id(s) in Task Object Definition for Types "documentAdditionOrUpdate" and "documentDeletion"

[ClaytonMooreFHN]

Our team was attempting to leverage the task webhook to perform an action on documents that are created/updated/deleted inside an index. Currently the response from the Webhook does not provide any information on which specific document was created/updated/deleted, Task Object Definition:

{ "uid": 471, "indexUid": "foobar", "status": "succeeded", "type": "documentAdditionOrUpdate", "canceledBy": null, "details": { "receivedDocuments": 1, "indexedDocuments": 1 }, "error": null, "duration": "PT0.521215500S", "enqueuedAt": "2024-06-17T18:46:47.271172Z", "startedAt": "2024-06-17T18:46:47.2742343Z", "finishedAt": "2024-06-17T18:46:47.7954498Z" },

It would be a welcome addition, when the task is of a "document" type, to add an array of document Id's to the details of the task, for example:

{ "uid": 471, "indexUid": "foobar", "status": "succeeded", "type": "documentAdditionOrUpdate", "canceledBy": null, "details": { "receivedDocuments": 1, "recievedDocumentIds": ["51ef2b66-303e-49b1-bbf0-1a9a04478550"], "indexedDocuments": 1, "indexedDocumentIds": ["51ef2b66-303e-49b1-bbf0-1a9a04478550"] }, "error": null, "duration": "PT0.521215500S", "enqueuedAt": "2024-06-17T18:46:47.271172Z", "startedAt": "2024-06-17T18:46:47.2742343Z", "finishedAt": "2024-06-17T18:46:47.7954498Z" },

This would allow our webhook endpoint to perform updates on the document(s) that were modified.

For example: A document is updated with an array of id's corresponding to pdf assets in a CMS, which triggers the webhook, then the endpoint could lookup the document via the document id in the post body. From there it would find the asset id in the cms, parse the pdf content and append it on the document inside the meilisearch implementation.

If I am missing something or overlooked how to achieve this without this addition please educate me!

[macraig]

Hi @ClaytonMooreFHN , thanks for providing a detailed description of your feature request.

We discussed your proposal with the engineering team, and while we understand the value of having document IDs included in the webhook response for your use case, we have some concerns that prevent us from implementing this feature. The main issues are:

1. The document IDs are parsed during the indexing process by the indexer itself. Showing them in the task queue would mean adding a link between the indexer and the task queue during the indexing process, which is complex and might not be the most efficient approach.

2. Our main concern are the updates containing thousands or millions of documents. This would add a lot of data to store in the task queue and when we get tasks.

As an alternative, we suggest the following workaround:

You can store an _updated_at field in your documents when they are uploaded or updated. By filtering documents using this date, you can identify which documents have been modified.

You can also use the webhook to trigger a request, allowing you to react to the update being processed. You can use the the enqeuedAt date to filter the documents that were just processed. To balance the time between when the documents are sent and when Meilisearch registers them, you can adjust this by a small buffer (e.g., minus 10 seconds). This will require converting UTC to Unix timestamps since only numbers can be stored in Meilisearch.

Please let us know if this workaround would meet your needs!