"Displaying geohash tags from a Loki datasource in a Grafana Worl… (#28)

* Initial draft
* Optimised images with calibre/image-actions

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Author: jorgelbg

content/posts/2019/rename-kibana-index-patterns/rename-kibana-index-patterns.md

@@ -146,4 +146,4 @@ If you want to know more about the structure of the documents that Kibana
 persists in ES (for its internal use) you should check [this blog
 post](https://www.elastic.co/blog/kibana-under-the-hood-object-persistence)
 
-{{< /info >}}
+{{< /info >}}

content/posts/2020/loki-worldmap-panel.md

@@ -0,0 +1,99 @@
+---
+title: "Displaying geohash tags from a Loki datasource in a Grafana Worldmap Panel"
+date: 2020-03-18T00:00:00+01:00
+draft: true
+description: >
+  How to use Loki and the Worldmap panel plugin to show geohashes or latitude/longitude pairs in a map.
+tags: ["grafana", "loki", "worldmap", "geohash"]
+---
+
+[Loki](https://grafana.com/oss/loki/) is a new~ish project from [Grafana](https://grafana.com), yes
+the same company behind the popular open-source observability platform. Loki itself is a
+horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus.
+
+On fewer words, Loki is like Prometheus but for your logs. This means that although it doesn't
+provide all the full-text search capabilities of Elasticsearch, it allows filtering by a set of
+labels for each log stream. This translates into a more cost-effective solution.
+
+Even though Loki was announced on [KubeCon
+18'](https://kccna18.sched.com/event/GrXC/on-the-oss-path-to-full-observability-with-grafana-david-kaltschmidt-grafana-labs).
+It is still under active development. As expected the team is building in parallel the internal Loki
+components: ingestion, storage, and query layer and the visualization UI (Grafana). It was not until recently that I've started to play with Loki. My main experience is about
+running large [Solr](https://lucene.apache.org/solr/) and/or
+[Elasticsearch](https://www.elastic.co/de/elasticsearch) clusters, but I was looking for something
+a bit easier and cheaper to host by myself. After all, I was not interested in all
+the features from the ELK stack.
+
+At the moment my data consist of a few labels and no actual logline, which is ok. I'm
+more interested only in the labels. Some of those labels (in my case) contained the latitude (`lat`),
+longitude (`long`) and geohash (`geohash`) of a given location. I also included some descriptive
+information like `place` and `country`.
+
+I wanted to plot this into a map using the [Worldmap
+Panel](https://grafana.com/grafana/plugins/grafana-worldmap-panel/installation). Yet, it was a bit
+tricky. I had already run a container with the latest version of Grafana since I was using
+the Explore UI from Grafana with the Loki datasource to check the incoming data. I ran the
+following command to add the world map panel:
+
+```bash
+$ docker run -d -p 3000:3000 -e "GF_INSTALL_PLUGINS=grafana-worldmap-panel" grafana/grafana
+```
+
+My initial thought was to point the Worldmap Panel to the Loki datasource and run one of the
+[LogQL](https://github.com/grafana/loki/blob/master/docs/logql.md) queries against it. The query
+ended up looking like:
+
+```js
+sum(count_over_time({geohash=~".+"}[1h])) by (geohash,lat,long,country,place)
+```
+
+My initial thought was to point the Worldmap Panel to the Loki datasource and run one of the
+[LogQL](https://github.com/grafana/loki/blob/master/docs/logql.md) queries against it. The query
+ended up looking like:
+
+This didn't work. Using the query inspector from Grafana I noticed that the response payload
+from Loki was almost identical to the output of a Prometheus query. The next step was
+to try and replicate the setup shown in [this
+post](https://www.robustperception.io/using-geohashes-with-the-worldmap-panel-and-prometheus).
+
+Trying to show the data setting the Location Data as a geohash yielded this error:
+```
+Error: Missing geohash value
+```
+Which made a bit of sense, since we need to set the `{{ geohash }}` as the legend of our query. This
+makes the `geohash` label (when using a Prometheus datasource) available to the Worldmap panel. Since
+the legend input is missing from the Loki datasource it is not possible to do set it up this way.
+Since I already have the `lat` and `long` available from the query, I also tried to use the
+Location Data as a table, the setup ended up looking like:
+
+{{< picture "loki-table-fail" "Setup of the Loki datasource as a table" "50%" >}}
+
+Although this setup didn't produce any error it didn't visualize anything on the map either 🤷‍♂️.
+The missing **piece of the puzzle** is that we can configure the Loki datasource as a Prometheus
+datasource in Grafana 🤯.
+
+To do this we need to create a Prometheus datasource in Grafana but point it out to the Loki
+endpoint:
+
+{{< picture "loki-and-prometheus-datasources" "Both datasources configured in the Grafana instance" >}}
+
+As you can see we use the same URL, but we need to add the `/loki` path to the Prometheus datasource.
+
+Using Loki as a Prometheus datasources allows us to use the same query as before, but have
+access to the configuration options only available for a Prometheus datasource. We can now follow the
+instructions in the [mentioned
+post](https://www.robustperception.io/using-geohashes-with-the-worldmap-panel-and-prometheus) to
+visualize the labels stored in a Loki datasource in the Worldmap panel.
+
+## Summary
+
+TL;DR you can configure a Loki datasource as a Prometheus datasource
+in Grafana, which will provide the same bells and whistles of a normal Prometheus server. Keep in mind that
+LogQL, the query language implemented by Loki, is a subset of PromQL, which means that not all
+functions, aggregations or operators will be available.
+
+This step of configuring Loki as a Prometheus datasource is mandatory (until the publishing date of
+this post) to get the Worldmap Panel plugin playing nicely with your Loki datasource. I imagine that
+in the no so far future Grafana's Loki support will be easier to use and it will change its behavior
+depending on how you want to use the data, which will make things (like using a 3rd party plugin)
+easier.

layouts/shortcodes/picture.html

@@ -1,3 +1,5 @@
+{{ $width := default "100%" (.Get 2) }}
+
 <picture>
   {{ if not .Site.IsServer }}
   <source
@@ -14,5 +16,6 @@
     class="align-center"
     alt="{{ .Get 1 }}"
     loading="lazy"
+    style="max-width:{{ $width }};"
   />
 </picture>

static/images/loki-worldmap-panel/loki-and-prometheus-datasources.png

@@ -266,11 +266,6 @@ li {
   list-style-position: inside;
 }
 
-#post-body,
-p {
-  line-height: 1.7;
-}
-
 b {
   font-weight: 500;
   color: var(--text-color);
@@ -763,8 +758,8 @@ h1 {
   font-weight: 400;
   font-style: normal;
   font-size: 1.22rem;
-  line-height: 1.58;
-  letter-spacing: -0.003em;
+  line-height: 1.65;
+  letter-spacing: 0.025em;
 }
 
 .post h1 {
@@ -870,7 +865,7 @@ footer p.small {
 
 .post header h1 svg {
   display: none;
-  margin-left: -36px;
+  margin-left: -24px;
 }
 
 .post header h1 > a {
@@ -981,3 +976,9 @@ header .last-edited {
   text-transform: uppercase;
   margin-bottom: 30px;
 }
+
+picture {
+  display: flex;
+  justify-content: center;
+  margin-bottom: 20px;
+}