add compactor details and other boltdb-shipper doc improvments #2622
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sandeepsukhani
requested review from
achatterjee-grafana and
oddlittlebird
as code owners
| @@ -48,22 +48,26 @@ Loki can be configured to run as just a single vertically scaled instance or as | |||
| When it comes to reads and writes, Ingesters are the ones which writes the index and chunks to stores and Queriers are the ones which reads index and chunks from the store for serving requests. | |||
|
|
|||
| Before we get into more details, it is important to understand how Loki manages index in stores. Loki shards index as per configured period which defaults to 7 days i.e when it comes to table based stores like Bigtable/Cassandra/DynamoDB there would be separate table per week containing index for that week. | |||
| In case of BoltDB files there is no concept of tables, so it creates a BoltDB file per period(i.e day in case of boltdb-shipper store). Files/Tables created per day are identified by a configured `prefix_` + `<period-number-since-epoch>`. | |||
| In case of BolDB Shipper a table is defined by a collection of many smaller BoltDB files, each file storing just 15 mins worth of index. Tables created per day are identified by a configured `prefix_` + `<period-number-since-epoch>`. | |||
There was a problem hiding this comment.
Needs an article and a comma:
In the case of BolDB Shipper, a table is defined by a collection of many smaller BoltDB files, each file storing just 15 mins worth of index. Tables created per day are identified by a configured prefix_ + <period-number-since-epoch>.
Codecov Report
@@ Coverage Diff @@
## master #2622 +/- ##
==========================================
- Coverage 62.87% 62.85% -0.02%
==========================================
Files 170 170
Lines 15049 15049
==========================================
- Hits 9462 9459 -3
- Misses 4826 4831 +5
+ Partials 761 759 -2
|
| Here `<period-number-since-epoch>` in case of boltdb-shipper would be day number since epoch. | ||
| For example, if you have prefix set to `loki_index_` and a write request comes in on 20th April 2020, it would be stored in table/file named `loki_index_18372` because it has been `18371` days since epoch, and we are in `18372`th day. | ||
| For example, if you have prefix set to `loki_index_` and a write request comes in on 20th April 2020, it would be stored in table named `loki_index_18372` because it has been `18371` days since epoch, and we are in `18372`th day. |
There was a problem hiding this comment.
Copy-edit suggestion:
For example, if you have a prefix set to loki_index_ and a written request comes in on 20th April 2020, it would be stored in a table named loki_index_18372 because it has been 18371 days since the epoch, and we are in 18372th day.
| Since sharding of index creates multiple files when using BoltDB, BoltDB Shipper would create a folder per day and add files for that day in that folder and names those files after ingesters which created them. | ||
|
|
||
| To reduce the size of files which helps with faster transfer speeds and reduced storage costs, they are stored after compressing them with gzip. |
There was a problem hiding this comment.
"help" not helps
| @@ -77,18 +81,21 @@ When running Loki in clustered mode there could be multiple ingesters serving wr | |||
| *NOTE: To avoid any loss of index when Ingester crashes it is recommended to run Ingesters as statefulset(when using k8s) with a persistent storage for storing index files.* | |||
|
|
|||
| Another important detail to note is when chunks are flushed they are available for reads in object store instantly while index is not since we only upload them every 15 Minutes with BoltDB shipper. | |||
| To avoid missing logs from queries which happen to be indexed in BoltDB files which are not shipped yet, while serving queries for in-memory logs, Ingesters would also do a store query for `now()` - (`max_chunk_age` + `30 Min`) to `<end-time-from-query-request>`. | |||
| Ingesters expose a new RPC for letting Queriers query the Ingester's local index for chunks which were recently flushed but its index might not be available yet with Queriers. | |||
| For all the queries which requires chunks to be read from the store, Queriers also query Ingesters over RPC for IDs of chunks which were recently flushed which is to avoid missing any logs from queries. | |||
There was a problem hiding this comment.
"require" not requires
|
|
||
| ### Queriers | ||
|
|
||
| Queriers lazily loads BoltDB files from shared object store to configured `cache_location`. | ||
| When a querier receives a read request, query range from request is resolved to period numbers and all the files for those period numbers are downloaded to `cache_location` if not already. | ||
| When a querier receives a read request, query range from request is resolved to period numbers and all the files for those period numbers are downloaded to `cache_location`, if not already. |
There was a problem hiding this comment.
Copy-edit:
When a querier receives a read request, the query range from the request is resolved to period numbers and all the files for those period numbers are downloaded to cache_location, if not already.
| Once we have downloaded files for a period we keep looking for updates in shared object store and download them every 15 Minutes by default. | ||
| Frequency for checking updates can be configured with `resync_interval` config. | ||
|
|
||
| To avoid keeping downloaded index files forever there is a ttl for them which defaults to 24 hours, which means if index files for a period are not used for 24 hours they would be removed from cache location. | ||
| ttl can be configured using `cache_ttl` config. | ||
|
|
||
| *NOTE: For better read performance and to avoid using node disk it is recommended to run Queriers as statefulset(when using k8s) with a persistent storage for downloading and querying index files.* |
There was a problem hiding this comment.
Remove "a" from "a persistent storage..."
| @@ -99,4 +106,25 @@ This problem would be faced even during rollouts which is quite common. | |||
| To avoid this, Loki disables deduplication of index when the replication factor is greater than 1 and `boltdb-shipper` is an active or upcoming index type. | |||
| While using `boltdb-shipper` please avoid configuring WriteDedupe cache since it is used purely for the index deduplication, so it would not be used anyways. | |||
|
|
|||
| ### Compactor | |||
|
|
|||
| Compactor is a BoltDB Shipper specific service which reduces the index size by deduping the index and merging all the files to a single file per table. | |||
There was a problem hiding this comment.
Change to "that reduces ..."
| Compactor is a BoltDB Shipper specific service which reduces the index size by deduping the index and merging all the files to a single file per table. | ||
| It is highly recommended running a Compactor since a single Ingester creates 96 files per day which includes a lot of duplicate index entries and querying multiple files per table adds up the overall query latency. | ||
|
|
||
| *NOTE: There should be only 1 compactor instance running at a time which otherwise could create problems and may lead to data loss. * |
There was a problem hiding this comment.
Change to "that otherwise..."
| ### Compactor | ||
|
|
||
| Compactor is a BoltDB Shipper specific service which reduces the index size by deduping the index and merging all the files to a single file per table. | ||
| It is highly recommended running a Compactor since a single Ingester creates 96 files per day which includes a lot of duplicate index entries and querying multiple files per table adds up the overall query latency. |
There was a problem hiding this comment.
Copy-edit:
We recommend running a Compactor since a single Ingester creates 96 files per day which include a lot of duplicate index entries and querying multiple files per table adds up the overall query latency.
|
@achatterjee-grafana Thanks for the review! I have taken care of all the suggested changes. |
achatterjee-grafana
approved these changes
Sep 15, 2020
oddlittlebird
approved these changes
Sep 15, 2020
| └── ingester-1 | ||
| ├── ingester-0-1587254400.gz | ||
| └── ingester-1-1587254400.gz | ||
| ... | ||
| ``` | ||
| *NOTE: We also add a timestamp to names of the files to randomize the names to avoid overwriting files when running Ingesters with same name and not have a persistent storage. Timestamps not shown here for simplification* |
There was a problem hiding this comment.
Suggested change
| *NOTE: We also add a timestamp to names of the files to randomize the names to avoid overwriting files when running Ingesters with same name and not have a persistent storage. Timestamps not shown here for simplification* | |
| **Note:** We also add a timestamp to names of the files to randomize the names to avoid overwriting files when running Ingesters with same name and not have a persistent storage. Timestamps not shown here for simplification. |
| @@ -77,18 +81,21 @@ When running Loki in clustered mode there could be multiple ingesters serving wr | |||
| *NOTE: To avoid any loss of index when Ingester crashes it is recommended to run Ingesters as statefulset(when using k8s) with a persistent storage for storing index files.* | |||
There was a problem hiding this comment.
Suggested change
| *NOTE: To avoid any loss of index when Ingester crashes it is recommended to run Ingesters as statefulset(when using k8s) with a persistent storage for storing index files.* | |
| **Note:** To avoid any loss of index when Ingester crashes it is recommended to run Ingesters as statefulset(when using k8s) with a persistent storage for storing index files. |
| Once we have downloaded files for a period we keep looking for updates in shared object store and download them every 15 Minutes by default. | ||
| Frequency for checking updates can be configured with `resync_interval` config. | ||
|
|
||
| To avoid keeping downloaded index files forever there is a ttl for them which defaults to 24 hours, which means if index files for a period are not used for 24 hours they would be removed from cache location. | ||
| ttl can be configured using `cache_ttl` config. | ||
|
|
||
| *NOTE: For better read performance and to avoid using node disk it is recommended to run Queriers as statefulset(when using k8s) with persistent storage for downloading and querying index files.* |
There was a problem hiding this comment.
Suggested change
| *NOTE: For better read performance and to avoid using node disk it is recommended to run Queriers as statefulset(when using k8s) with persistent storage for downloading and querying index files.* | |
| **Note:** For better read performance and to avoid using node disk it is recommended to run Queriers as statefulset(when using k8s) with persistent storage for downloading and querying index files. |
| Compactor is a BoltDB Shipper specific service that reduces the index size by deduping the index and merging all the files to a single file per table. | ||
| We recommend running a Compactor since a single Ingester creates 96 files per day which include a lot of duplicate index entries and querying multiple files per table adds up the overall query latency. | ||
|
|
||
| *NOTE: There should be only 1 compactor instance running at a time that otherwise could create problems and may lead to data loss. * |
There was a problem hiding this comment.
Suggested change
| *NOTE: There should be only 1 compactor instance running at a time that otherwise could create problems and may lead to data loss. * | |
| **Note:** There should be only 1 compactor instance running at a time that otherwise could create problems and may lead to data loss. |
| @@ -48,22 +48,26 @@ Loki can be configured to run as just a single vertically scaled instance or as | |||
| When it comes to reads and writes, Ingesters are the ones which writes the index and chunks to stores and Queriers are the ones which reads index and chunks from the store for serving requests. | |||
|
|
|||
| Before we get into more details, it is important to understand how Loki manages index in stores. Loki shards index as per configured period which defaults to 7 days i.e when it comes to table based stores like Bigtable/Cassandra/DynamoDB there would be separate table per week containing index for that week. | |||
There was a problem hiding this comment.
Suggested change
| Before we get into more details, it is important to understand how Loki manages index in stores. Loki shards index as per configured period which defaults to 7 days i.e when it comes to table based stores like Bigtable/Cassandra/DynamoDB there would be separate table per week containing index for that week. | |
| Before we get into more details, it is important to understand how Loki manages index in stores. Loki shards index as per configured period which defaults to seven days i.e when it comes to table based stores like Bigtable/Cassandra/DynamoDB there would be separate table per week containing index for that week. |
There was a problem hiding this comment.
Write out numbers zero through ten unless it would cause confusion.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds details about compactor to the docs and updates some other details related to recent changes in BoltDB Shipper.
Checklist