Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update indexing settings #4868

Closed
wants to merge 4 commits into from
Closed

Update indexing settings #4868

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
abde4c6
Add updates to indexing settings and split endpoint
rdettai Apr 17, 2024
b39c33c
Update docs
rdettai Apr 17, 2024
fa8b3f5
Address wording and naming comments
rdettai May 7, 2024
c98ed8e
Use json enum instead of hand written proto enum
rdettai May 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
54 changes: 40 additions & 14 deletions docs/reference/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,48 +183,72 @@ quickwit index create --endpoint=http://127.0.0.1:7280 --index-config wikipedia_
### index update

`quickwit index update [args]`

*Synopsis*

```bash
quickwit index update
--index <index>
```

*Options*

| Option | Description |
|-----------------|-------------|
| `--index` | ID of the target index |
#### index update search-settings

Updates default search settings.
Updates search settings.
`quickwit index update search-settings [args]`

*Synopsis*

```bash
quickwit index update search-settings
--index <index>
--default-search-fields <default-search-fields>
--config-file <config-file>
```

*Options*

| Option | Description |
|-----------------|-------------|
| `--index` | ID of the target index |
| `--default-search-fields` | List of fields that Quickwit will search into if the user query does not explicitly target a field. Space-separated list, e.g. "field1 field2". If no value is provided, existing defaults are removed and queries without target field will fail. |
| `--config-file` | Location of a json or yaml file containing the new search settings. See https://quickwit.io/docs/configuration/index-config#search-settings. |
#### index update retention-policy

Configure or disable the retention policy.
Updates or disables the retention policy.
`quickwit index update retention-policy [args]`

*Synopsis*

```bash
quickwit index update retention-policy
--index <index>
[--period <period>]
[--schedule <schedule>]
[--config-file <config-file>]
[--disable]
```

*Options*

| Option | Description |
|-----------------|-------------|
| `--index` | ID of the target index |
| `--period` | Duration after which splits are dropped. Expressed in a human-readable way (`1 day`, `2 hours`, `1 week`, ...) |
| `--schedule` | Frequency at which the retention policy is evaluated and applied. Expressed as a cron expression (0 0 * * * *) or human-readable form (hourly, daily, weekly, ...). |
| `--disable` | Disable the retention policy. Old indexed data will not be cleaned up anymore. |
| `--config-file` | Location of a json or yaml file containing the new retention policy. See https://quickwit.io/docs/configuration/index-config#retention-policy. |
| `--disable` | Disables the retention policy. Old indexed data will not be cleaned up anymore. |
#### index update indexing-settings

Updates indexing settings.
`quickwit index update indexing-settings [args]`

*Synopsis*

```bash
quickwit index update indexing-settings
--config-file <config-file>
```

*Options*

| Option | Description |
|-----------------|-------------|
| `--config-file` | Location of a json or yaml file containing the new indexing settings. See https://quickwit.io/docs/configuration/index-config#indexing-settings. |
### index clear

Clears an index: deletes all splits and resets checkpoint.
Expand Down Expand Up @@ -368,6 +392,7 @@ quickwit index ingest
[--input-path <input-path>]
[--batch-size-limit <batch-size-limit>]
[--wait]
[--v2]
[--force]
[--commit-timeout <commit-timeout>]
```
Expand All @@ -380,8 +405,9 @@ quickwit index ingest
| `--input-path` | Location of the input file. |
| `--batch-size-limit` | Size limit of each submitted document batch. |
| `--wait` | Wait for all documents to be commited and available for search before exiting |
| `--v2` | Ingest v2 (experimental! Do not use me.) |
| `--force` | Force a commit after the last document is sent, and wait for all documents to be committed and available for search before exiting |
| `--commit-timeout` | Timeout for ingest operations that require waiting for the final commit (`--wait` or `--force`). This is different from the `commit_timeout_secs` indexing setting which sets the maximum time before commiting splits after their creation. |
| `--commit-timeout` | Timeout for ingest operations that require waiting for the final commit (`--wait` or `--force`). This is different from the `commit_timeout_secs` indexing setting, which sets the maximum time before commiting splits after their creation. |

*Examples*

Expand Down
61 changes: 37 additions & 24 deletions docs/reference/rest-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,45 +309,58 @@ The response is the index metadata of the created index, and the content type is
| `sources` | List of the index sources configurations. | `Array<SourceConfig>` |


### Update an index (search settings and retention policy only)
### Update an index

```
PUT api/v1/indexes/<index id>
PUT api/v1/indexes/<index id>/indexing-settings
PUT api/v1/indexes/<index id>/search-settings
PUT api/v1/indexes/<index id>/retention-policy
```

Updates the search settings and retention policy of an index. This endpoint follows PUT semantics (not PATCH), which means that all the updatable fields of the index configuration are replaced by the values specified in this request. In particular, omitting an optional field like retention_policy will delete the associated configuration. Unlike the create endpoint, this API only accepts JSON payloads.
These endpoints follows PUT semantics (not PATCH), which means that all the fields of the updated configuration are replaced by the values specified in the request. Values that are not specified will be reset to their defaults. The API accepts JSON with `content-type: application/json` and YAML `content-type: application/yaml`.
- The search settings update is automatically picked up by the janitor service on its next state refresh.
rdettai marked this conversation as resolved.
Show resolved Hide resolved
- The retention policy update is automatically picked up when the next query is executed.
rdettai marked this conversation as resolved.
Show resolved Hide resolved
- The indexing settings update is not automatically picked up by the indexer nodes, they need to be manually restarted.

#### PUT payload
#### PUT payloads

| Variable | Type | Description | Default value |
|---------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------|
| `search_settings` | `SearchSettings` | Search settings object as specified in the [index config docs](../configuration/index-config.md#search-settings). | |
| `retention` | `Retention` | Retention policy object as specified in the [index config docs](../configuration/index-config.md#retention-policy). | |
| Endpoint | Type | Description |
|---------------------|--------------------|------------------------------------------------------------------------------------------|
| `/search-settings` | `SearchSettings` | See [search settings config docs](../configuration/index-config.md#search-settings). |
| `/retention-policy` | `Retention` | See [retention policy config docs](../configuration/index-config.md#retention-policy). |
| `/indexing-settings`| `IndexingSettings` | See [indexing settings config docs](../configuration/index-config.md#indexing-settings). |


**Payload Example**
**Payload Examples**

curl -XPUT http://0.0.0.0:8080/api/v1/indexes --data @index_update.json -H "Content-Type: application/json"
curl -XPUT http://0.0.0.0:8080/api/v1/indexes/my-index/search-settings --data @search_settings_update.json -H "Content-Type: application/json"

```json title="index_update.json
```json title="search_settings_update.json
{
"search_settings": {
"default_search_fields": ["body"]
},
"retention": {
"period": "3 days",
"schedule": "@daily"
}
"default_search_fields": ["body"]
}
```

:::warning
Calling the update endpoint with the following payload will remove the current retention policy.
```json
curl -XPUT http://0.0.0.0:8080/api/v1/indexes/my-index/retention-policy --data @retention_policy_update.json -H "Content-Type: application/json"

```json title="retention_policy_update.json
{
"search_settings": {
"default_search_fields": ["body"]
}
"period": "3 days",
"schedule": "daily"
}
```

curl -XPUT http://0.0.0.0:8080/api/v1/indexes/my-index/indexing-settings --data @indexing_settings_update.json -H "Content-Type: application/json"

```json title="indexing_settings_update.json
{
"merge_policy": {
"type": "limit_merge",
"max_merge_ops": 3,
"merge_factor": 10,
"max_merge_factor": 12
},
"commit_timeout_secs": 5
}
```

Expand Down