Skip to content
28 changes: 13 additions & 15 deletions .vuepress/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,19 @@ module.exports = {
title: 'Updating MeiliSearch',
path: '/learn/advanced/updating.md',
},
{
title: 'Inner workings',
collapsable: false,
children: [
'/learn/advanced/concat',
'/learn/advanced/datatypes',
'/learn/advanced/indexation',
'/learn/advanced/prefix',
'/learn/advanced/storage',
'/learn/advanced/tokenization',
'/learn/advanced/typotolerance',
],
},
],
},
{
Expand Down Expand Up @@ -162,21 +175,6 @@ module.exports = {
'/reference/api/dump',
],
},
{
title: '🛠️ Under the hood',
path: '/reference/under_the_hood/',
collapsable: false,
children: [
'/reference/under_the_hood/bucket_sort',
'/reference/under_the_hood/concat',
'/reference/under_the_hood/datatypes',
'/reference/under_the_hood/indexation',
'/reference/under_the_hood/prefix',
'/reference/under_the_hood/storage',
'/reference/under_the_hood/tokenization',
'/reference/under_the_hood/typotolerance',
],
},
],
'/create/': [
{
Expand Down
14 changes: 13 additions & 1 deletion .vuepress/public/_redirects
Original file line number Diff line number Diff line change
Expand Up @@ -80,5 +80,17 @@
/reference/features/filtering.html /reference/features/filtering_and_faceted_search.html
/reference/features/faceted_search.html /reference/features/filtering_and_faceted_search.html

#move updating MeiliSearch
# Move under the hood articles to /learn
/reference/under_the_hood/concat.html /learn/advanced/concat.html
/reference/under_the_hood/datatypes.html /learn/advanced/datatypes.html
/reference/under_the_hood/indexation.html /learn/advanced/indexation.html
/reference/under_the_hood/prefix.html /learn/advanced/prefix.html
/reference/under_the_hood/storage.html /learn/advanced/storage.html
/reference/under_the_hood/tokenization.html /learn/advanced/tokenization.html
/reference/under_the_hood/typotolerance.html /learn/advanced/typotolerance.html

# Bucket sort
/reference/under_the_hood/bucket_sort.html /learn/core_concepts/relevancy.html#behavior

# Move updating MeiliSearch
/create/how_to/updating.html /learn/advanced/updating.html
2 changes: 1 addition & 1 deletion create/how_to/aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Choose the storage **device** and **size** for your MeiliSearch instance.

![Storage](/aws/05.storage.png)

The amount of storage space required can [vary drastically](/reference/under_the_hood/storage.md#measured-disk-usage) depending on the data you plan to index. In this example, we will use 25 GiB, which is more than enough for most small datasets (< 1 million documents).
The amount of storage space required can [vary drastically](/learn/advanced/storage.md#measured-disk-usage) depending on the data you plan to index. In this example, we will use 25 GiB, which is more than enough for most small datasets (< 1 million documents).

When you're ready, click on **Next: Add Tags** to continue.

Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ Based on their type, however, the fields will be handled and used in different w

Types:

- [String](/reference/under_the_hood/datatypes.md#string)
- [Numeric types](/reference/under_the_hood/datatypes.md#numeric-type): `integer`, `float`
- [Boolean](/reference/under_the_hood/datatypes.md#boolean)
- [Array](/reference/under_the_hood/datatypes.md#array)
- [Object](/reference/under_the_hood/datatypes.md#object)
- [`null` type](/reference/under_the_hood/datatypes.md#null-type)
- [String](/learn/advanced/datatypes.md#string)
- [Numeric types](/learn/advanced/datatypes.md#numeric-type): `integer`, `float`
- [Boolean](/learn/advanced/datatypes.md#boolean)
- [Array](/learn/advanced/datatypes.md#array)
- [Object](/learn/advanced/datatypes.md#object)
- [`null` type](/learn/advanced/datatypes.md#null-type)

### String

Expand Down Expand Up @@ -103,7 +103,7 @@ Will be processed as if all elements were arranged at the same level:
"Bruce Willis. Vin Diesel. Kung Fu Panda."
```

The strings above will be separated by soft and hard spaces exactly as explained in the [string example](/reference/under_the_hood/datatypes.md#examples).
The strings above will be separated by soft and hard spaces exactly as explained in the [string example](/learn/advanced/datatypes.md#examples).

### Object

Expand All @@ -126,7 +126,7 @@ In the example above, `movie_id`, `1564saqw12ss`, `title`, `Kung fu Panda` are a
"movie_id. 1564saqw12ss. title. Kung fu Panda."
```

These sentences will be separated by soft and hard spaces exactly as explained in the [string example](/reference/under_the_hood/datatypes.md#examples).
These sentences will be separated by soft and hard spaces exactly as explained in the [string example](/learn/advanced/datatypes.md#examples).

### null type

Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

In MeiliSearch, **you can perform a search with only a single letter as your query**. This is because we follow the philosophy of **prefix search**.

Prefix search is when document sorting starts by comparing the search query against the beginning of each word in your dataset. All documents with words that match the query term are added to the [bucket sort](/reference/under_the_hood/bucket_sort.md), before the [ranking rules](/learn/core_concepts/relevancy.md#ranking-rules) are applied sequentially.
Prefix search is when document sorting starts by comparing the search query against the beginning of each word in your dataset. All documents with words that match the query term are added to the [bucket sort](https://en.wikipedia.org/wiki/Bucket_sort), before the [ranking rules](/learn/core_concepts/relevancy.md#ranking-rules) are applied sequentially.

In other words, prefix search means that it's not necessary to type a word in its entirety to find documents containing that word—you can just type the first one or two letters.

Expand Down Expand Up @@ -34,8 +34,8 @@ response:

- `show`

MeiliSearch also handles typos while performing the prefix search. You can [read more about the typo rules on the dedicated page](/reference/under_the_hood/typotolerance.md#typo-tolerance-rules).
MeiliSearch also handles typos while performing the prefix search. You can [read more about the typo rules on the dedicated page](/learn/advanced/typotolerance.md#typo-tolerance-rules).

::: tip
We also [apply splitting and concatenating on search queries](/reference/under_the_hood/concat.md).
We also [apply splitting and concatenating on search queries](/learn/advanced/concat.md).
:::
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Typo tolerance

MeiliSearch **is [typo tolerant](/reference/under_the_hood/typotolerance.md#typo-tolerance-rules)**; this means **it understands your search even if there are typos or spelling mistakes**.
MeiliSearch **is [typo tolerant](/learn/advanced/typotolerance.md#typo-tolerance-rules)**; this means **it understands your search even if there are typos or spelling mistakes**.

#### Example

Expand Down
2 changes: 1 addition & 1 deletion learn/core_concepts/documents.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ A field is a set of two data items linked together: an attribute and a value. Do

An attribute functions a bit like a variable in most programming languages, i.e. it is a name that allows you to store, access, and describe some data. That data is the attribute's **value**.

Every field has a [data type](/reference/under_the_hood/datatypes.md) dictated by its value. Every value must be a valid [JSON data type](https://www.w3schools.com/js/js_json_datatypes.asp).
Every field has a [data type](/learn/advanced/datatypes.md) dictated by its value. Every value must be a valid [JSON data type](https://www.w3schools.com/js/js_json_datatypes.asp).

Take note that, in the case of strings, a value **[can contain at most 65535 positions](/reference/features/known_limitations.md#maximum-words-per-attribute). Words exceeding the 65535 position limit will be ignored.**

Expand Down
2 changes: 1 addition & 1 deletion learn/core_concepts/relevancy.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ In order to ensure relevant results, search responses are sorted based on a set

Each index possesses a list of ranking rules stored as an array in the [settings object](/reference/api/settings.md). This array is **fully customizable**, meaning you can **delete existing rules, add new ones, and reorder them as needed**.

Whenever a search query is made, MeiliSearch uses a [bucket sort](/reference/under_the_hood/bucket_sort.md) algorithm to rank documents. The first ranking rule is applied to all documents, while each subsequent rule is only applied to documents that are considered equal under the previous rule (i.e. as a tiebreaker).
Whenever a search query is made, MeiliSearch uses a [bucket sort](https://en.wikipedia.org/wiki/Bucket_sort) algorithm to rank documents. The first ranking rule is applied to all documents, while each subsequent rule is only applied to documents that are considered equal under the previous rule (i.e. as a tiebreaker).

**The order in which ranking rules are applied matters.** The first rule in the array has the most impact, and the last rule has the least. Our default configuration meets most standard needs but [you can change it](/reference/api/ranking_rules.md#update-ranking-rules).

Expand Down
4 changes: 2 additions & 2 deletions learn/what_is_meilisearch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ Meilisearch helps the Rust community find crates on [crates.meilisearch.com](htt
## Features

- **Blazing fast** (answers < 50 milliseconds): Priority is given to fast answers for a smooth search experience.
- [Search as you type](/learn/what_is_meilisearch/features.md#search-as-you-type): Results are updated on each keystroke. To make this possible, we use [prefix-search](/reference/under_the_hood/prefix.md#prefix-search).
- [Search as you type](/learn/what_is_meilisearch/features.md#search-as-you-type): Results are updated on each keystroke. To make this possible, we use [prefix-search](/learn/advanced/prefix.md#prefix-search).
- [Typo tolerance](/learn/what_is_meilisearch/features.md#typo-tolerant): Understands typoes and misspellings.
- [Tokenization](/reference/under_the_hood/tokenization.md) in **English**, **Chinese**, and **all languages that uses space as a word divider**.
- [Tokenization](/learn/advanced/tokenization.md) in **English**, **Chinese**, and **all languages that uses space as a word divider**.
- **Return the whole document**: The entire document is returned upon search.
- **Highly customizable search and indexation**:
- [Custom ranking](/learn/core_concepts/relevancy.md): Customize the relevancy of the search engine and the ranking of the search results.
Expand Down
2 changes: 1 addition & 1 deletion learn/what_is_meilisearch/features.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ MeiliSearch's default [relevancy rules](/learn/core_concepts/relevancy.md) are d
## Typo tolerant

Instead of letting typos ruin your search experience, MeiliSearch will always find the results you expect.
Read more about typo tolerance in [this dedicated guide](/reference/under_the_hood/typotolerance.md).
Read more about typo tolerance in [this dedicated guide](/learn/advanced/typotolerance.md).

## Synonyms

Expand Down
2 changes: 1 addition & 1 deletion reference/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ This is **timeless knowledge**. We recommend keeping it open in a spare tab, whe

- [⭐ Features](/reference/features)
- [📒 API](/reference/api)
- [🛠️ Under the hood](/reference/under_the_hood)
- [🛠️ Under the hood](/learn/advanced)
2 changes: 1 addition & 1 deletion reference/features/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Feature references

This section is where we document all of MeiliSearch's user-facing features and tools. This does not include [API routes](/reference/api) or [details of the MeiliSearch engine](/reference/under_the_hood); these each have their own section.
This section is where we document all of MeiliSearch's user-facing features and tools. This does not include [API routes](/reference/api) or [details of the MeiliSearch engine](/learn/advanced); these each have their own section.

Understanding our reference documentation is easier with background knowledge of MeiliSearch's [core concepts](/learn/core_concepts). If you're a **new user**, you might prefer to start with an [explanation of MeiliSearch's features](/learn/what_is_meilisearch/features.md) or our [quick start guide](/learn/getting_started/quick_start.md).

Expand Down
4 changes: 2 additions & 2 deletions reference/features/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ Sets the maximum size of the index. Value must be given in bytes or explicitly s

The `index` stores processed data and is different from the `update` database, which handles [pending updates](/learn/advanced/asynchronous_updates.md).

[Learn more about MeiliSearch's database and storage engine.](/reference/under_the_hood/storage.md)
[Learn more about MeiliSearch's database and storage engine.](/learn/advanced/storage.md)

### Max UDB size

Expand All @@ -221,7 +221,7 @@ Sets the maximum size of the `update` database. Value must be given in bytes or

The `update` database handles [pending updates](/learn/advanced/asynchronous_updates.md). This is different from the `index` database, which only stores processed data.

[Learn more about MeiliSearch's database and storage engine.](/reference/under_the_hood/storage.md)
[Learn more about MeiliSearch's database and storage engine.](/learn/advanced/storage.md)

### Payload limit size

Expand Down
6 changes: 3 additions & 3 deletions reference/features/known_limitations.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Known limitations

Currently, MeiliSearch has a number of known limitations. Some of these limitations are the result of intentional design trade-offs, while others can be attributed to [LMDB](/reference/under_the_hood/storage.md), the key-value store that MeiliSearch uses [under the hood](/reference/under_the_hood).
Currently, MeiliSearch has a number of known limitations. Some of these limitations are the result of intentional design trade-offs, while others can be attributed to [LMDB](/learn/advanced/storage.md), the key-value store that MeiliSearch uses under the hood.

## Design limitations

Expand Down Expand Up @@ -38,7 +38,7 @@ If your query is `Hello, World`:
- `World` takes the position `9` of the attribute

::: note
`,` takes 8 positions as it is a hard separator. You can read more about word separators in our [article about datatypes](/reference/under_the_hood/datatypes.md#string).
`,` takes 8 positions as it is a hard separator. You can read more about word separators in our [article about datatypes](/learn/advanced/datatypes.md#string).
:::

If your query is `Hello - World`:
Expand All @@ -48,7 +48,7 @@ If your query is `Hello - World`:
- `World` takes the position `2` of the attribute

::: note
`-` takes 1 position as it is a soft separator. You can read more about word separators in our [article about datatypes](/reference/under_the_hood/datatypes.md#string).
`-` takes 1 position as it is a soft separator. You can read more about word separators in our [article about datatypes](/learn/advanced/datatypes.md#string).
:::

### Maximum attributes per document
Expand Down
2 changes: 1 addition & 1 deletion reference/features/language.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

We aim to provide global language support, and your feedback helps us [move closer to that goal](#improving-our-language-support). If you notice inconsistencies in your search results or the way your documents are processed, please open an issue on our [GitHub repository](https://github.com/meilisearch/MeiliSearch/issues/new/choose).

If you'd like to learn more about how different languages are processed in MeiliSearch, see our [tokenizer documentation](/reference/under_the_hood/tokenization.md).
If you'd like to learn more about how different languages are processed in MeiliSearch, see our [tokenizer documentation](/learn/advanced/tokenization.md).

## Improving our language support

Expand Down
2 changes: 1 addition & 1 deletion reference/features/search_parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ This will give you a list of documents that contain your query terms in at least

If you enclose search terms in double quotes (`"`), MeiliSearch will only return documents containing those terms in the order they were given. This is called a **phrase search**.

Phrase searches are case-insensitive and ignore [soft separators such as `-`, `,`, and `:`](/reference/under_the_hood/datatypes.md). Using a hard separator within a phrase search effectively splits it into multiple separate phrase searches: `"Octavia.Butler"` will return the same results as `"Octavia" "Butler"`.
Phrase searches are case-insensitive and ignore [soft separators such as `-`, `,`, and `:`](/learn/advanced/datatypes.md). Using a hard separator within a phrase search effectively splits it into multiple separate phrase searches: `"Octavia.Butler"` will return the same results as `"Octavia" "Butler"`.

You can combine phrase search and normal queries in a single search request. In this case, MeiliSearch will first fetch all documents with exact matches to the given phrase(s), and [then proceed with its default behavior](/learn/core_concepts/relevancy.md).

Expand Down
18 changes: 0 additions & 18 deletions reference/under_the_hood/README.md

This file was deleted.

7 changes: 0 additions & 7 deletions reference/under_the_hood/bucket_sort.md

This file was deleted.

4 changes: 2 additions & 2 deletions resources/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,12 +229,12 @@ Beware heavily multi-lingual datasets and datasets with many unique words, such

### Search speed

Because MeiliSearch uses a [memory map](/reference/under_the_hood/storage.md#lmdb), **search speed is based on the ratio between RAM and database size**. In other words:
Because MeiliSearch uses a [memory map](/learn/advanced/storage.md#lmdb), **search speed is based on the ratio between RAM and database size**. In other words:

- A big database + a small amount of RAM => slow search
- A small database + tons of RAM => lightning fast search

MeiliSearch also uses disk space as [virtual memory](/reference/under_the_hood/storage.md#memory-usage). This disk space does not correspond to database size; rather, it provides speed and flexibility to the engine by allowing it to go over the limits of physical RAM.
MeiliSearch also uses disk space as [virtual memory](/learn/advanced/storage.md#memory-usage). This disk space does not correspond to database size; rather, it provides speed and flexibility to the engine by allowing it to go over the limits of physical RAM.

At this time, the number of CPU cores has no direct impact on index or search speed. However, **the more cores you provide to the engine, the more search queries it will be able to process at the same time**.

Expand Down