feat: Adding custom token separators to search index

[adilansari]

Create a search index

$ -> http PUT localhost:8081/v1/search/indexes/shoes

{
  schema: {
  title: 'shoes_index',
  properties: {
    name: {
      type: 'string',
    },
  },
  options: {
    token_separators: [ '-' ],
  },
}           

Retrieve a search index

$ ➔  http GET localhost:8081/v1/projects/todo_app/search/indexes                                                                                                 ±[●][tokenize]
HTTP/1.1 200 OK
Content-Length: 169
Content-Type: application/json
Date: Wed, 21 Jun 2023 23:42:48 GMT
Server-Timing: total;dur=2
Vary: Origin

{
    "indexes": [
        {
            "name": "shoes",
            "schema": {
                "options": {
                    "token_separators": [
                        "-"
                    ]
                },
                "properties": {
                    "name": {
                        "type": "string"
                    }
                },
                "source": {
                    "type": "external"
                },
                "title": "shoes_index"
            }
        }
    ]
}

Typesense collection

$ ➔  http GET localhost:8108/collections X-TYPESENSE-API-KEY:<replace_me>                                                                                       ±[●][tokenize]
HTTP/1.1 200 OK
Connection: keep-alive
accept-ranges: none
access-control-allow-origin: *
content-encoding: gzip
content-type: application/json; charset=utf-8
transfer-encoding: chunked
vary: accept-encoding

[
    {
        "created_at": 1687390839,
        "default_sorting_field": "",
        "enable_nested_fields": true,
        "fields": [
            {
                "facet": false,
                "index": true,
                "infix": false,
                "locale": "",
                "name": "name",
                "optional": true,
                "sort": false,
                "type": "string"
            },
            {
                "facet": false,
                "index": false,
                "infix": false,
                "locale": "",
                "name": "_tigris_id",
                "optional": true,
                "sort": false,
                "type": "string"
            },
            {
                "facet": false,
                "index": false,
                "infix": false,
                "locale": "",
                "name": "_tigris_created_at",
                "optional": true,
                "sort": false,
                "type": "int64"
            },
            {
                "facet": false,
                "index": false,
                "infix": false,
                "locale": "",
                "name": "_tigris_updated_at",
                "optional": true,
                "sort": false,
                "type": "int64"
            }
        ],
        "name": "1:411:shoes",
        "num_documents": 0,
        "symbols_to_index": [],
        "token_separators": [
            "-"
        ]
    }
]

[reviewpad]

Reviewpad Report

‼️ Errors

• Unconventional commit detected: 'rejecting update on token separators' (c0138c7)

[himank]

You don't need a pointer to it, the slice will be empty if token_separators is not set.

[tigrisdata-argocd-bot]

🎉 This PR is included in version 1.0.0-beta.121 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀