Distributor overview docs #3371
Conversation
Co-authored-by: Ed Welch <ed@oqqer.com>
|
|
||
| ## Where does it live? | ||
|
|
||
| The distributor is the first component on Loki's write path downstream from any gateways providing auth or load balancing. It's responsible for validating, preprocessing, and applying a subset of rate limiting to incoming data before sending it to the ingester component. It is important that a load balancer sits in front of the distributor in order to properly balance traffic to them. |
There was a problem hiding this comment.
It is important that a load balancer sits in front of the distributor
Question (doesn't have to in the doc, but just curious): in our setup say in dev or ops-cluster, what( and where) is the component?
So when I send my logs via promtail to the client URL. That URL is basically the gateway and does Load balancing right?
There was a problem hiding this comment.
We have a bunch of extra stuff in front of this for Grafana Cloud, but it doesn't directly affect this open source project. Happy to talk about that in a less public place :).
|
|
||
| ### Validation | ||
|
|
||
| The first step the distributor takes is to ensure that all incoming data is according to specification. This includes things like checking that the labels are valid Prometheus labels as well as ensuring the timestamps aren't too old or too new or the log lines aren't too long. |
There was a problem hiding this comment.
well as ensuring the timestamps aren't too old or too new or the log lines aren't too long
Again curious about the timestamp and log lines contraints. Are these two constraints are configurable? or its kinda static limit we hardcode into Loki now?
There was a problem hiding this comment.
Yes, they come from the following in our limits config and are configurable per tenant as well:
# Whether or not old samples will be rejected.
# CLI flag: -validation.reject-old-samples
[reject_old_samples: <bool> | default = false]
# Maximum accepted sample age before rejecting.
# CLI flag: -validation.reject-old-samples.max-age
[reject_old_samples_max_age: <duration> | default = 336h]
# Duration for a table to be created/deleted before/after it's
# needed. Samples won't be accepted before this time.
# CLI flag: -validation.create-grace-period
[creation_grace_period: <duration> | default = 10m]
This is the first part in a new overview series which aims to provide information on Loki's component architecture.
/cc @kavirajk