Get and delete documents by filter (#236)

* specify the new fetch documents route * update open-api * try to be nice with bumpsh * specify the new analytics of the get/fetch documents route * group the analytics behind already used groups * fix a placeholder text * group the get and fetch route under a common section * rename the section to contain the name of both routes * clarify how to send the parameter to both routes * fix the type of the parameters * add the new fetch route to the api keys actions * capitalize json Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> * update open-api * Specify the new route to delete documents by filter * add the new analytics event * improve a comment * add the post - delete route to the api key actions * Fix the json payload Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com> * Update the details Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com> * delete the invalid_document_delete_filter error code and introduce the missing_document_filter * fix the open-api thingy --------- Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>
meilisearch · Jun 5, 2023 · 8b23eb8 · 8b23eb8
1 parent d6d5751
commit 8b23eb8
Show file tree

Hide file tree

Showing 6 changed files with 308 additions and 22 deletions.
diff --git a/open-api.yaml b/open-api.yaml
@@ -1664,6 +1664,7 @@ paths:
         - $ref: '#/components/parameters/limit'
         - $ref: '#/components/parameters/offset'
         - $ref: '#/components/parameters/fields'
+        - $ref: '#/components/parameters/filter'
     post:
       operationId: indexes.documents.create
       summary: Add or replace documents
@@ -1805,6 +1806,93 @@ paths:
           description: Not Found
     parameters:
       - $ref: '#/components/parameters/indexUid'
+  '/indexes/{indexUid}/documents/fetch':
+    post:
+      operationId: indexes.documents.fetch
+      summary: Get Documents
+      description: |
+        Get [documents](https://docs.meilisearch.com/learn/core_concepts/documents.html) by batch.
+      tags:
+        - Documents
+      security:
+        - apiKey: []
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                offset:
+                  type: number
+                  description: Number of documents to skip.
+                  default: 0
+                limit:
+                  type: number
+                  description: Maximum number of documents returned.
+                  default: 20
+                fields:
+                  type: array
+                  description: 'Array of attributes whose fields will be present in the returned documents. By default all attributes will be returned.'
+                  items:
+                    type: string
+                    example: '["title", "overview"]'
+                    default: '["*"]'
+                filter:
+                  $ref: '#/components/schemas/filter'
+            examples:
+              Example:
+                value:
+                  offset: 2
+                  limit: 5
+                  fields:
+                    - name
+                    - picture
+                  filter: 'doggo = "bernese mountain"'
+      responses:
+        '200':
+          description: Ok
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  results:
+                    type: array
+                    items:
+                      $ref: '#/components/schemas/document'
+                  limit:
+                    $ref: '#/components/schemas/limit'
+                  offset:
+                    $ref: '#/components/schemas/offset'
+                  total:
+                    $ref: '#/components/schemas/total'
+                required:
+                  - results
+                  - limit
+                  - offset
+                  - total
+              examples:
+                Example:
+                  value:
+                    results:
+                      - id: 25684
+                        title: American Ninja 5
+                        poster: 'https://image.tmdb.org/t/p/w1280/iuAQVI4mvjI83wnirpD8GVNRVuY.jpg'
+                        overview: 'When a scientists daughter is kidnapped, American Ninja, attempts to find her, but this time he teams up with a youngster he has trained in the ways of the ninja.'
+                        release_date: 725846400
+                      - id: 45881
+                        title: The Bridge of San Luis Rey
+                        poster: 'https://image.tmdb.org/t/p/w500/4X7quIcdkc24Cveg5XdpfRqxtYA.jpg'
+                        overview: "The Bridge of San Luis Rey is American author Thornton Wilder's second novel, first published in 1927 to worldwide acclaim. It tells the story of several interrelated people who die in the collapse of an Inca rope-fiber suspension bridge in Peru, and the events that lead up to their being on the bridge.[ A friar who has witnessed the tragic accident then goes about inquiring into the lives of the victims, seeking some sort of cosmic answer to the question of why each had to die. The novel won the Pulitzer Prize in 1928."
+                        release_date: 1072915200
+                    limit: 20
+                    offset: 0
+                    total: 2
+        '401':
+          $ref: '#/components/responses/401'
+        '404':
+          description: Not Found
   '/indexes/{indexUid}/documents/delete-batch':
     post:
       operationId: indexes.documents.removeBatch
@@ -1850,6 +1938,51 @@ paths:
         - $ref: '#/components/parameters/Content-Type'
     parameters:
       - $ref: '#/components/parameters/indexUid'
+  '/indexes/{indexUid}/documents/delete':
+    post:
+      operationId: indexes.documents.remove
+      summary: Delete documents
+      description: Delete a selection of documents based on a filter.
+      tags:
+        - Documents
+      security:
+        - apiKey: []
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                filter:
+                  $ref: '#/components/schemas/filter'
+            examples:
+              Example:
+                value:
+                  offset: 2
+                  limit: 5
+                  fields:
+                    - name
+                    - picture
+                  filter: 'doggo = "bernese mountain"'
+      responses:
+        '202':
+          description: Accepted
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/responses/202'
+              examples:
+                '202':
+                  $ref: '#/components/examples/202_documentDeletion'
+        '401':
+          $ref: '#/components/responses/401'
+        '404':
+          description: Not Found
+      parameters:
+        - $ref: '#/components/parameters/Content-Type'
+    parameters:
+      - $ref: '#/components/parameters/indexUid'
   '/indexes/{indexUid}/documents/{documentId}':
     get:
       operationId: indexes.documents.get

diff --git a/text/0034-telemetry-policies.md b/text/0034-telemetry-policies.md
@@ -43,7 +43,9 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat
 | Documents Searched by Multi-Search POST | Aggregated event on all received requests via the `POST`- `/multi-search` route during one hour or until a batch size reaches `500Kb`. |
 | Documents Added | Aggregated event on all received requests via the `POST` - `/indexes/:indexUid/documents` route during one hour or until a batch size reaches `500Kb`. |
 | Documents Updated | Aggregated event on all received requests via the `PUT` - `/indexes/:indexUid/documents` route during one hour or until a batch size reaches `500Kb`. |
-| Documents Deleted | Aggregated event on all received requests via the `DELETE` - `/indexes/:indexUid/documents`, `DELETE` - `/indexes/:indexUid/documents/:documentId`, `POST` - `indexes/:indexUid/documents/delete-batch` routes during one hour or until a batch size reaches `500Kb`.  |
+| Documents Fetched GET | Aggregated event on all received requests via the `GET` - `/indexes/:indexUid/documents` or `GET` - `/indexes/:indexUid/documents/:doc_id` routes during one hour or until a batch size reaches `500Kb`. |
+| Documents Fetched POST | Aggregated event on all received requests via the `POST` - `/indexes/:indexUid/documents/fetch` routes during one hour or until a batch size reaches `500Kb`.  |
+| Documents Deleted | Aggregated event on all received requests via the `DELETE` - `/indexes/:indexUid/documents`, `DELETE` - `/indexes/:indexUid/documents/:documentId`, `POST` - `indexes/:indexUid/documents/delete-batch`, and `POST` - `indexes/:indexUid/documents/delete` routes during one hour or until a batch size reaches `500Kb`.  |
 | Index Created | Occurs when an index is created via `POST` - `/indexes`. |
 | Index Updated | Occurs when an index is updated via `PUT` - `/indexes/:indexUid`. |
 | Indexes Swapped | Occurs when indexes are swapped via `POST` - `/swap-indexes`. |
@@ -112,15 +114,15 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat
 | `requests.99th_response_time`           | Highest latency from among the fastest 99% of successful search requests | 57ms    | `Documents Searched POST`, `Documents Searched GET`|
 | `requests.total_succeeded`              | Total number of successful requests in this batch | 3456 | `Documents Searched POST`, `Documents Searched GET`, `Documents Searched by Multi-Search POST` |
 | `requests.total_failed`                 | Total number of failed requests in this batch    | 24   | `Documents Searched POST`, `Documents Searched GET`, `Documents Searched by Multi-Search POST` |
-| `requests.total_received`               | Total number of received requests in this batch  | 3480 | `Documents Searched POST`, `Documents Searched GET`, `Documents Deleted`, `Health Seen`, `Tasks Seen`, `Documents Searched by Multi-Search POST` |
+| `requests.total_received`               | Total number of received requests in this batch  | 3480 | `Documents Searched POST`, `Documents Searched GET`, `Documents Deleted`, `Documents Fetched GET`, `Documents Fetched POST`, `Health Seen`, `Tasks Seen`, `Documents Searched by Multi-Search POST` |
 | `sort.with_geoPoint`                    | `true` if the sort rule `_geoPoint` was used in this batch, otherwise `false` | true | `Documents Searched POST`, `Documents Searched GET` |
 | `sort.avg_criteria_number`              | Average number of sort criteria among all requests containing the `sort` parameter in this batch | 2 | `Documents Searched POST`, `Documents Searched GET` |
 | `filter.with_geoRadius`                 | `true` if the filter rule `_geoRadius` was used in this batch, otherwise `false` | false | `Documents Searched POST`, `Documents Searched GET` |
 | `filter.with_geoBoundingBox`            | `true` if the filter rule `_geoBoundingBox` was used in this batch, otherwise `false`| false | `Documents Searched POST`, `Documents Searched GET` |
 | `filter.most_used_syntax`               | Most used filter syntax among all requests containing the `filter` parameter in this batch | string | `Documents Searched POST`, `Documents Searched GET` |
 | `q.max_terms_number`                    | Highest number of terms given for the `q` parameter in this batch | 5 | `Documents Searched POST`, `Documents Searched GET` |
-| `pagination.max_limit`                  | Highest value given for the `limit` parameter in this batch | 60 | `Documents Searched POST`, `Documents Searched GET` |
-| `pagination.max_offset`                 | Highest value given for the `offset` parameter in this batch | 1000 | `Documents Searched POST`, `Documents Searched GET` |
+| `pagination.max_limit`                  | Highest value given for the `limit` parameter in this batch | 60 | `Documents Searched POST`, `Documents Searched GET`, `Documents Fetched GET`, `Documents Fetched POST` |
+| `pagination.max_offset`                 | Highest value given for the `offset` parameter in this batch | 1000 | `Documents Searched POST`, `Documents Searched GET`, `Documents Fetched GET`, `Documents Fetched POST` |
 | `pagination.most_used_navigation`       | Most used search results navigation among all search requests in this batch. `estimated` / `exhaustive` | `estimated` | `Documents Searched POST`, `Documents Searched GET` |
 | `formatting.max_attributes_to_retrieve` | The maximum number of attributes to retrieve encountered among all requests in this batch. | 100 | `Documents Searched POST`, `Documents Searched GET` |
 | `formatting.max_attributes_to_highlight` | The maximum number of attributes to highlight encountered among all requests in this batch. | 100 | `Documents Searched POST`, `Documents Searched GET` |
@@ -180,8 +182,9 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat
 | `swap_operation_number`            | The number of swap operation given in `POST /swap-indexes` API call | 2 | `Indexes Swapped` |
 | `matching_strategy.most_used_strategy`      | Most used word matching strategy among all search requests in this batch | `last` | `Documents Searched POST`, `Documents Searched GET` |
 | `per_document_id` | `true` if `DELETE /indexes/:indexUid/documents/:documentUid` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |
-| `clear_all` | `true` if `DELETE /indexes/:indexUid/documents` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |
 | `per_batch` | `true` if `POST /indexes/:indexUid/documents/delete-batch` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |
+| `per_filter`| `true` if `POST /indexes/:indexUid/documents/delete` endpoint was used in this batch, otherwise `false` | false | `Documents Fetched GET`, `Documents Fetched POST`, `Documents Deleted` |
+| `clear_all` | `true` if `DELETE /indexes/:indexUid/documents` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |
 
 ----
 
@@ -394,6 +397,31 @@ This property allows us to gather essential information to better understand on
 | per_document_id | `true` if `DELETE /indexes/:indexUid/documents/:documentUid` endpoint is called in the aggregated event, otherwise `false` | `false` |
 | clear_all | `true` if `DELETE /indexes/:indexUid/documents` endpoint is called in the aggregated event, otherwise `false` | `false` |
 | per_batch | `true` if `POST /indexes/:indexUid/documents/delete-batch` endpoint is called in the aggregated event, otherwise `false` | `false` |
+| per_filter | `true` if `POST /indexes/:indexUid/documents/delete` endpoint is called in the aggregated event, otherwise `false` | `false` |
+
+## `Documents Fetched GET`
+
+> The Documents Fetched GET event is sent once an hour or when a batch reaches the maximum size of `500kb`.
+
+| Property name | Description | Example |
+|---------------|-------------|---------|
+| `requests.total_received` | Total number of request received in this batch | 325 |
+| `per_document_id` | `true` if `GET /indexes/:indexUid/documents/:doc_id` endpoint was used in this batch, otherwise `false` | false |
+| `per_filter` | `true` if `GET /indexes/:indexUid/documents` endpoint was used with a filter in this batch, otherwise `false` | false |
+| `pagination.max_limit` | Highest value given for the `limit` parameter in this batch | 60 |
+| `pagination.max_offset` | Highest value given for the `offset` parameter in this batch | 1000 |
+
+## `Documents Fetched POST`
+
+> The Documents Fetched POST event is sent once an hour or when a batch reaches the maximum size of `500kb`.
+
+| Property name | Description | Example |
+|---------------|-------------|---------|
+| `requests.total_received` | Total number of request received in this batch | 325 |
+| `per_document_id` | `false` | false |
+| `per_filter` | `true` if `POST /indexes/:indexUid/documents/fetch` endpoint was used with a filter in this batch, otherwise `false` | false |
+| `pagination.max_limit` | Highest value given for the `limit` parameter in this batch | 60 |
+| `pagination.max_offset` | Highest value given for the `offset` parameter in this batch | 1000 |
 
 ## `Settings Updated`
 

diff --git a/text/0060-tasks-api.md b/text/0060-tasks-api.md
@@ -103,6 +103,7 @@ List of global tasks by `type`:
 | name                | description                          |
 |---------------------|--------------------------------------|
 | providedIds         | Number of provided document ids.     |
+| originalFilter      | The filter used to delete documents. `null` if `filter` was not used for the deletion request. |
 | deletedDocuments    | Number of documents finally deleted. |
 
 ##### indexCreation

diff --git a/text/0061-error-format-and-definitions.md b/text/0061-error-format-and-definitions.md
@@ -878,6 +878,54 @@ HTTP Code: `400 Bad Request`
 
 ---
 
+## invalid_document_filter
+
+`Synchronous` / `Asynchronous`
+
+### Context
+
+This error occurs if a value with a different type than `String`, `Array of String` or `Array of array of String` for `filter` is specified.
+
+### Error Definition
+
+HTTP Code: `400 Bad Request`
+
+```json
+{
+    "message": "`:deserr_helper`",
+    "code": "invalid_document_filter",
+    "type": "invalid_request",
+    "link": "https://docs.meilisearch.com/errors#invalid_document_filter"
+}
+```
+
+---
+
+## missing_document_filter
+
+`Synchronous`
+
+### Context
+
+This error happens when `filter` is missing from a delete documents by filter operation.
+
+### Error Definition
+
+In the first case:
+
+HTTP Code: `400 Bad Request`
+
+```json
+{
+    "message": "`filter` field is mandatory.",
+    "code": "missing_document_filter",
+    "type": "invalid_request",
+    "link": "https://docs.meilisearch.com/errors#missing_document_filter"
+}
+```
+
+---
+
 ## invalid_document_limit
 
 `Synchronous`

diff --git a/text/0085-api-keys.md b/text/0085-api-keys.md
@@ -315,8 +315,8 @@ Create an API key.
 |------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | search           | Provides access to `GET` and `POST` methods on `/indexes/:authorizedIndexes/search` routes.                                                                                                                                     |
 | documents.add    | Provides access to `POST` and `PUT` on `/indexes/:authorizedIndexes/documents` routes.                                                                                                                                          |
-| documents.get    | Provides access to `GET` methods on `/indexes/:authorizedIndexes/documents` and `/indexes/:authorizedIndexes/documents/:documentId` routes.                                                                                     |
-| documents.delete | Provides access to `DELETE` method on `/indexes/:authorizedIndexes/documents/:documentId`, `indexes/:authorizedIndexes/documents/:documentId` and `POST` method on `/indexes/:authorizedIndexes/documents/delete-batch` routes. |
+| documents.get    | Provides access to `GET` methods on `/indexes/:authorizedIndexes/documents`, `/indexes/:authorizedIndexes/documents/:documentId` and `POST` methods on `/indexes/:authorizedIndexes/documents/fetch` routes.                                                                                     |
+| documents.delete | Provides access to `DELETE` method on `/indexes/:authorizedIndexes/documents/:documentId`, `indexes/:authorizedIndexes/documents/:documentId` and `POST` method on `/indexes/:authorizedIndexes/documents/delete-batch` and `/indexes/:authorizedIndexes/documents/delete` routes. |
 | indexes.create   | Provides access to `POST` `/indexes`. **⚠️ `indexes` field should indicate the newly created index or having `[*]` to permits access on it.**.                                                                                   |
 | indexes.get      | Provides access to `GET` `/indexes` and `/indexes/:authorizedIndexes`. **⚠️Non-authorized `indexes` are omitted from the response on `/indexes`**.                                                                               |
 | indexes.update   | Provides access to `PUT` `/indexes/:authorizedIndexes`.                                                                                                                                                                         |