Storage backend examples #2357
Merged
Storage backend examples #2357
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cyriltovena
reviewed
Jul 16, 2020
cyriltovena
reviewed
Jul 16, 2020
docs/storage.md
Outdated
|
|
||
| ### File System | ||
|
|
||
| The file system is the simplest backend for chunks, although it's also susceptible to data loss as it's unreplicated. This is common for single binary deployments though, as well as for those trying out loki or doing local development on the project. Without significant expertise and knowledge of Loki's operational patterns, the file system backend is not suggested for production use unless you can tolerate log loss. |
There was a problem hiding this comment.
a bit harsh I think the single binary on PVC is production ready. I understand the point, but millions of Prometheus users are out there using a single binary in production for monitoring thousands of servers. If you could rephrase this so that it doesn't diminish the use case for filesystem that would be great.
I'd like Loki single binary on filesystem to be also perceived as a starting point.
cyriltovena
reviewed
Jul 16, 2020
|
|
||
| ## Period Configs | ||
|
|
||
| Loki aims to be backwards compatible and over the course of it's development has had many internal changes that facilitate better and more efficient storage/querying. Loki allows incrementally upgrading to these new storage _schemas_ and can query across them transparently. This makes upgrading a breeze. For instance, this is what it looks like when migrating from the v10 -> v11 schemas starting 2020-07-01: |
cyriltovena
reviewed
Jul 16, 2020
| # Storage | ||
|
|
||
| Loki uses a two pronged strategy regarding storage, which is responsible for both it's limitations and it's advantages. The main idea is that logs are large and traditional indexing strategies are prohibitively expensive and complex to run at scale. This often brings along ancillary procedure costs in the form of schema design, index management/rotation, backup/restore protocols, etc. Instead, Loki stores all the its log content unindexed in object storage. It then uses the Prometheus label paradigm along with a small but specialized index store to allow lookup, matching, and filtering based on the these labels. When a set of unique key/value label pairs are combined with their logs, we call this a _log stream_, which is generally analagous to a log file on disk. It may have labels like `{app="api", env="production", filename="/var/logs/app.log"}`, which together uniqely identify it. The object storage is responsible for storing the compressed logs cheaply while the index takes care of storing these labels in a way that enables fast, effective querying. | ||
|
|
There was a problem hiding this comment.
I would add a small reference to our goal toward not requiring an index store ? WDYT ?
There was a problem hiding this comment.
I'd like to leave this point out for now.
cyriltovena
approved these changes
Jul 16, 2020
Codecov Report
@@ Coverage Diff @@
## master #2357 +/- ##
=======================================
Coverage 61.63% 61.63%
=======================================
Files 160 160
Lines 13534 13534
=======================================
Hits 8342 8342
+ Misses 4567 4566 -1
- Partials 625 626 +1
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Writes a new default storage doc which includes some digestible example configurations.