This repository has been archived by the owner on Mar 21, 2024. It is now read-only.
Separators settings api #254
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ManyTheFish
added
OpenAPI
ManyTheFish
changed the title
macraig
reviewed
Jul 19, 2023
Co-authored-by: Maria Craig <marycraig90@gmail.com>
Co-authored-by: Maria Craig <marycraig90@gmail.com>
macraig
previously approved these changes
Jul 19, 2023
This was referenced Jul 25, 2023
bidoubiwa
reviewed
Aug 8, 2023
meili-bors bot
added a commit
to meilisearch/meilisearch
that referenced
this pull request
Aug 10, 2023
3946: Settings customizing tokenization r=irevoire a=ManyTheFish
# Pull Request
This pull Request allows the User to customize Meilisearch Tokenization by providing specialized settings.
## Small documentation
All the new settings can be set and reset like the other index settings by calling the route `/indexes/:name/settings`
### `nonSeparatorTokens`
The Meilisearch word segmentation uses a default list of separators to segment words, however, for specific use cases some of the default separators shouldn't be considered separators, the `nonSeparatorTokens` setting allows to remove of some tokens from the default list of separators.
***Request payload `PUT`- `/indexes/articles/settings/non-separator-tokens`***
```json
["`@",` "#", "&"]
```
### `separatorTokens`
Some use cases need to define additional separators, some are related to a specific way of parsing technical documents some others are related to encodings in documents, the `separatorTokens` setting allows adding some tokens to the list of separators.
***Request payload `PUT`- `/indexes/articles/settings/separator-tokens`***
```json
["§", "&sep"]
```
### `dictionary`
The Meilisearch word segmentation relies on separators and language-based word-dictionaries to segment words, however, this segmentation is inaccurate on technical or use-case specific vocabulary (like `G/Box` to say `Gear Box`), or on proper nouns (like `J. R. R.` when parsing `J. R. R. Tolkien`), the `dictionary` setting allows defining a list of words that would be segmented as described in the list.
***Request payload `PUT`- `/indexes/articles/settings/dictionary`***
```json
["J. R. R.", "J.R.R."]
```
these last feature synergies well with the `stopWords` setting or the `synonyms` setting allowing to segment words and correctly retrieve the synonyms:
***Request payload `PATCH`- `/indexes/articles/settings`***
```json
{
"dictionary": ["J. R. R.", "J.R.R."],
"synonyms": {
"J.R.R.": ["jrr", "J. R. R."],
"J. R. R.": ["jrr", "J.R.R."],
"jrr": ["J.R.R.", "J. R. R."],
}
}
```
### Related specifications:
- meilisearch/specifications#255
- meilisearch/specifications#254
### Try it with Docker
```bash
$ docker pull getmeili/meilisearch:prototype-tokenizer-customization-3
```
## Related issue
Fixes #3610
Fixes #3917
Fixes https://github.com/meilisearch/product/discussions/468
Fixes https://github.com/meilisearch/product/discussions/160
Fixes https://github.com/meilisearch/product/discussions/260
Fixes https://github.com/meilisearch/product/discussions/381
Fixes https://github.com/meilisearch/product/discussions/131
Related to #2879
Fixes #2760
## What does this PR do?
- Add a setting `nonSeparatorTokens` allowing to remove a token from the default separator tokens
- Add a setting `separatorTokens` allowing to add a token in the separator tokens
- Add a setting `dictionary` allowing to override the segmentation on specific words
- add new error code `invalid_settings_non_separator_tokens` (invalid_request)
- add new error code `invalid_settings_separator_tokens` (invalid_request)
- add new error code `invalid_settings_dictionary` (invalid_request)
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
4 tasks
|
|
||
| ***Request payload `PUT`- `/indexes/articles/settings/separator-tokens`*** | ||
| ```json | ||
| ["|", "/", "&sep"] |
There was a problem hiding this comment.
Isn't "/" a separator by default? Using this list as a reference: https://docs.rs/charabia/0.8.3/src/charabia/separators.rs.html#16-62
macraig
approved these changes
Sep 20, 2023
macraig
added a commit
that referenced
this pull request
Sep 25, 2023
* Bump API version * User dictionary settings API (#255) * Create 0123-user-dictionary-settings-api.md * Update 0123-user-dictionary-settings-api.md * rename wordDictionary into dictionary * Update 0034-telemetry-policies.md * Update 0123-settings-api.md * Update open-api.yaml * Update text/0123-user-dictionary-settings-api.md Co-authored-by: Maria Craig <marycraig90@gmail.com> * Update text/0123-user-dictionary-settings-api.md Co-authored-by: Maria Craig <marycraig90@gmail.com> --------- Co-authored-by: Maria Craig <marycraig90@gmail.com> * Separators settings api (#254) * Create 0123-separators-settings-api.md * Update 0123-settings-api.md * Update 0034-telemetry-policies.md * Update text/0034-telemetry-policies.md Co-authored-by: Maria Craig <marycraig90@gmail.com> * Update text/0034-telemetry-policies.md Co-authored-by: Maria Craig <marycraig90@gmail.com> * Update text/0123-separators-settings-api.md Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> * Update open-api.yaml * Update text/0123-separators-settings-api.md --------- Co-authored-by: Maria Craig <marycraig90@gmail.com> Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> --------- Co-authored-by: María <maria@Marias-MacBook-Pro.local> Co-authored-by: Many the fish <legendre.maxime.isn@gmail.com> Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Allow the user to customize the list of separators used by Meilisearch to segment the documents and the queries by providing 2 new settings:
separatorTokens: List of the tokens that should be considered separatorsnonSeparatorTokens: List of the tokens that should not be considered separatorsChanges
Misc