grafana · sandeepsukhani · Jun 28, 2021 · Jun 24, 2021
@@ -14,17 +14,17 @@ Things like, host, application, and environment are great labels. They will be f
 
 Too many label value combinations leads to too many streams. The penalties for that in Loki are a large index and small chunks in the store, which in turn can actually reduce performance.
 
-To avoid those issues, don't add a label for something until you know you need it! Use filter expressions ( |= “text”, |~ “regex”, …) and brute force those logs. It works -- and it's fast.
+To avoid those issues, don't add a label for something until you know you need it! Use filter expressions (`|= "text"`, `|~ "regex"`, …) and brute force those logs. It works -- and it's fast.
 
-From early on, we have set a label dynamically using Promtail pipelines for `level`. This seemed intuitive for us as we often wanted to only show logs for `level=”error”`; however, we are re-evaluating this now as writing a query. `{app=”loki”} |= “level=error”` is proving to be just as fast for many of our applications as `{app=”loki”,level=”error”}`.
+From early on, we have set a label dynamically using Promtail pipelines for `level`. This seemed intuitive for us as we often wanted to only show logs for `level="error"`; however, we are re-evaluating this now as writing a query. `{app="loki"} |= "level=error"` is proving to be just as fast for many of our applications as `{app="loki",level="error"}`.
 
-This may seem surprising, but if applications have medium to low volume, that label causes one application's logs to be split into up to five streams, which means 5x chunks being stored.  And loading chunks has an overhead associated with it. Imagine now if that query were `{app=”loki”,level!=”debug”}`. That would have to load **way** more chunks than `{app=”loki”} != “level=debug”`.
+This may seem surprising, but if applications have medium to low volume, that label causes one application's logs to be split into up to five streams, which means 5x chunks being stored.  And loading chunks has an overhead associated with it. Imagine now if that query were `{app="loki",level!="debug"}`. That would have to load **way** more chunks than `{app="loki"} != "level=debug"`.
 
 Above, we mentioned not to add labels until you _need_ them, so when would you _need_ labels?? A little farther down is a section on `chunk_target_size`. If you set this to 1MB (which is reasonable), this will try to cut chunks at 1MB compressed size, which is about 5MB-ish of uncompressed logs (might be as much as 10MB depending on compression). If your logs have sufficient volume to write 5MB in less time than `max_chunk_age`, or **many** chunks in that timeframe, you might want to consider splitting it into separate streams with a dynamic label.
 
 What you want to avoid is splitting a log file into streams, which result in chunks getting flushed because the stream is idle or hits the max age before being full. As of [Loki 1.4.0](https://grafana.com/blog/2020/04/01/loki-v1.4.0-released-with-query-statistics-and-up-to-300x-regex-optimization/), there is a metric which can help you understand why chunks are flushed `sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster="dev"}[1m]))`.
 
-It’s not critical that every chunk be full when flushed, but it will improve many aspects of operation. As such, our current guidance here is to avoid dynamic labels as much as possible and instead favor filter expressions. For example, don’t add a `level` dynamic label, just `|= “level=debug”` instead.
+It’s not critical that every chunk be full when flushed, but it will improve many aspects of operation. As such, our current guidance here is to avoid dynamic labels as much as possible and instead favor filter expressions. For example, don’t add a `level` dynamic label, just `|= "level=debug"` instead.
 
 ## Label values must always be bounded
 
@@ -76,26 +76,26 @@ One issue many people have with Loki is their client receiving errors for out of
 There are a few things to dissect from that statement. The first is this restriction is per stream.  Let’s look at an example:
 
 ```
-{job=”syslog”} 00:00:00 i’m a syslog!
-{job=”syslog”} 00:00:01 i’m a syslog!
+{job="syslog"} 00:00:00 i'm a syslog!
+{job="syslog"} 00:00:01 i'm a syslog!
 ```
 
 If Loki received these two lines which are for the same stream, everything would be fine. But what about this case:
 
 ```
-{job=”syslog”} 00:00:00 i’m a syslog!
-{job=”syslog”} 00:00:02 i’m a syslog!
-{job=”syslog”} 00:00:01 i’m a syslog!  <- Rejected out of order!
+{job="syslog"} 00:00:00 i'm a syslog!
+{job="syslog"} 00:00:02 i'm a syslog!
+{job="syslog"} 00:00:01 i'm a syslog!  <- Rejected out of order!
 ```
 
 What can we do about this? What if this was because the sources of these logs were different systems? We can solve this with an additional label which is unique per system:
 
 ```
-{job=”syslog”, instance=”host1”} 00:00:00 i’m a syslog!
-{job=”syslog”, instance=”host1”} 00:00:02 i’m a syslog!
-{job=”syslog”, instance=”host2”} 00:00:01 i’m a syslog!  <- Accepted, this is a new stream!
-{job=”syslog”, instance=”host1”} 00:00:03 i’m a syslog!  <- Accepted, still in order for stream 1
-{job=”syslog”, instance=”host2”} 00:00:02 i’m a syslog!  <- Accepted, still in order for stream 2
+{job="syslog", instance="host1"} 00:00:00 i'm a syslog!
+{job="syslog", instance="host1"} 00:00:02 i'm a syslog!
+{job="syslog", instance="host2"} 00:00:01 i'm a syslog!  <- Accepted, this is a new stream!
+{job="syslog", instance="host1"} 00:00:03 i'm a syslog!  <- Accepted, still in order for stream 1
+{job="syslog", instance="host2"} 00:00:02 i'm a syslog!  <- Accepted, still in order for stream 2
 ```
 
 But what if the application itself generated logs that were out of order? Well, I'm afraid this is a problem. If you are extracting the timestamp from the log line with something like [the Promtail pipeline stage](https://grafana.com/docs/loki/latest/clients/promtail/stages/timestamp/), you could instead _not_ do this and let Promtail assign a timestamp to the log lines. Or you can hopefully fix it in the application itself.

@@ -34,7 +34,7 @@ scrape_configs:
 This config will tail one file and assign one label: `job=syslog`. You could query it like this:
 
 ```
-{job=”syslog”}
+{job="syslog"}
 ```
 
 This will create one stream in Loki.
@@ -67,9 +67,9 @@ We can query these streams in a few ways:
 
 
 ```
-{job=”apache”} <- show me logs where the job label is apache
-{job=”syslog”} <- show me logs where the job label is syslog
-{job=~”apache|syslog”} <- show me logs where the job is apache **OR** syslog
+{job="apache"} <- show me logs where the job label is apache
+{job="syslog"} <- show me logs where the job label is syslog
+{job=~"apache|syslog"} <- show me logs where the job is apache **OR** syslog
 ```
 
 In that last example, we used a regex label matcher to log streams that use the job label with two values. Now consider how an additional label could also be used:
@@ -99,7 +99,7 @@ scrape_configs:
 Now instead of a regex, we could do this:
 
 ```
-{env=”dev”} <- will return all logs with env=dev, in this case this includes both log streams
+{env="dev"} <- will return all logs with env=dev, in this case this includes both log streams
 ```
 
 Hopefully now you are starting to see the power of labels. By using a single label, you can query many streams. By combining several different labels, you can create very flexible log queries.
@@ -137,8 +137,9 @@ This regex matches every component of the log line and extracts the value of eac
 
 From that regex, we will be using two of the capture groups to dynamically set two labels based on content from the log line itself:
 
-action (e.g. action=”GET”, action=”POST”)
-status_code (e.g. status_code=”200”, status_code=”400”)
+action (for example, `action="GET"`, `action="POST"`)
+
+status_code (for example, `status_code="200"`, `status_code="400"`)
 
 And now let's walk through a few example lines:
 
@@ -152,15 +153,15 @@ And now let's walk through a few example lines:
 In Loki the following streams would be created:
 
 ```
-{job=”apache”,env=”dev”,action=”GET”,status_code=”200”} 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
-{job=”apache”,env=”dev”,action=”POST”,status_code=”200”} 11.11.11.12 - frank [25/Jan/2000:14:00:02 -0500] "POST /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
-{job=”apache”,env=”dev”,action=”GET”,status_code=”400”} 11.11.11.13 - frank [25/Jan/2000:14:00:03 -0500] "GET /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
-{job=”apache”,env=”dev”,action=”POST”,status_code=”400”} 11.11.11.14 - frank [25/Jan/2000:14:00:04 -0500] "POST /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
+{job="apache",env="dev",action="GET",status_code="200"} 11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
+{job="apache",env="dev",action="POST",status_code="200"} 11.11.11.12 - frank [25/Jan/2000:14:00:02 -0500] "POST /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
+{job="apache",env="dev",action="GET",status_code="400"} 11.11.11.13 - frank [25/Jan/2000:14:00:03 -0500] "GET /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
+{job="apache",env="dev",action="POST",status_code="400"} 11.11.11.14 - frank [25/Jan/2000:14:00:04 -0500] "POST /1986.js HTTP/1.1" 400 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"
 ```
 
 Those four log lines would become four separate streams and start filling four separate chunks.
 
-Any additional log lines that match those combinations of label/values would be added to the existing stream. If another unique combination of labels comes in (e.g. status_code=”500”) another new stream is created.
+Any additional log lines that match those combinations of label/values would be added to the existing stream. If another unique combination of labels comes in (for example, `status_code="500"`) another new stream is created.
 
 Imagine now if you set a label for `ip`. Not only does every request from a user become a unique stream. Every request with a different action or status_code from the same user will get its own stream.
 
@@ -191,7 +192,7 @@ Loki will effectively keep your static costs as low as possible (index size and
 To see how this works, let's look back at our example of querying your access log data for a specific IP address. We don't want to use a label to store the IP. Instead we use a [filter expression](../../logql#filter-expression) to query for it:
 
 ```
-{job=”apache”} |= “11.11.11.11”
+{job="apache"} |= "11.11.11.11"
 ```
 
 Behind the scenes, Loki will break up that query into smaller pieces (shards), and open up each chunk for the streams matched by the labels and start looking for this IP address.