Skip to content

Commit

Permalink
"Displaying geohash tags from a Loki datasource in a Grafana Worl… (#28)
Browse files Browse the repository at this point in the history
* Initial draft
* Optimised images with calibre/image-actions

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
  • Loading branch information
jorgelbg and github-actions[bot] authored Mar 18, 2020
1 parent ef42bc1 commit 832de4d
Show file tree
Hide file tree
Showing 6 changed files with 112 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -146,4 +146,4 @@ If you want to know more about the structure of the documents that Kibana
persists in ES (for its internal use) you should check [this blog
post](https://www.elastic.co/blog/kibana-under-the-hood-object-persistence)

{{< /info >}}
{{< /info >}}
99 changes: 99 additions & 0 deletions content/posts/2020/loki-worldmap-panel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
---
title: "Displaying geohash tags from a Loki datasource in a Grafana Worldmap Panel"
date: 2020-03-18T00:00:00+01:00
draft: true
description: >
How to use Loki and the Worldmap panel plugin to show geohashes or latitude/longitude pairs in a map.
tags: ["grafana", "loki", "worldmap", "geohash"]
---

[Loki](https://grafana.com/oss/loki/) is a new~ish project from [Grafana](https://grafana.com), yes
the same company behind the popular open-source observability platform. Loki itself is a
horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus.

On fewer words, Loki is like Prometheus but for your logs. This means that although it doesn't
provide all the full-text search capabilities of Elasticsearch, it allows filtering by a set of
labels for each log stream. This translates into a more cost-effective solution.

Even though Loki was announced on [KubeCon
18'](https://kccna18.sched.com/event/GrXC/on-the-oss-path-to-full-observability-with-grafana-david-kaltschmidt-grafana-labs).
It is still under active development. As expected the team is building in parallel the internal Loki
components: ingestion, storage, and query layer and the visualization UI (Grafana). It was not until recently that I've started to play with Loki. My main experience is about
running large [Solr](https://lucene.apache.org/solr/) and/or
[Elasticsearch](https://www.elastic.co/de/elasticsearch) clusters, but I was looking for something
a bit easier and cheaper to host by myself. After all, I was not interested in all
the features from the ELK stack.

At the moment my data consist of a few labels and no actual logline, which is ok. I'm
more interested only in the labels. Some of those labels (in my case) contained the latitude (`lat`),
longitude (`long`) and geohash (`geohash`) of a given location. I also included some descriptive
information like `place` and `country`.

I wanted to plot this into a map using the [Worldmap
Panel](https://grafana.com/grafana/plugins/grafana-worldmap-panel/installation). Yet, it was a bit
tricky. I had already run a container with the latest version of Grafana since I was using
the Explore UI from Grafana with the Loki datasource to check the incoming data. I ran the
following command to add the world map panel:

```bash
$ docker run -d -p 3000:3000 -e "GF_INSTALL_PLUGINS=grafana-worldmap-panel" grafana/grafana
```

My initial thought was to point the Worldmap Panel to the Loki datasource and run one of the
[LogQL](https://github.com/grafana/loki/blob/master/docs/logql.md) queries against it. The query
ended up looking like:

```js
sum(count_over_time({geohash=~".+"}[1h])) by (geohash,lat,long,country,place)
```

My initial thought was to point the Worldmap Panel to the Loki datasource and run one of the
[LogQL](https://github.com/grafana/loki/blob/master/docs/logql.md) queries against it. The query
ended up looking like:

This didn't work. Using the query inspector from Grafana I noticed that the response payload
from Loki was almost identical to the output of a Prometheus query. The next step was
to try and replicate the setup shown in [this
post](https://www.robustperception.io/using-geohashes-with-the-worldmap-panel-and-prometheus).

Trying to show the data setting the Location Data as a geohash yielded this error:
```
Error: Missing geohash value
```
Which made a bit of sense, since we need to set the `{{ geohash }}` as the legend of our query. This
makes the `geohash` label (when using a Prometheus datasource) available to the Worldmap panel. Since
the legend input is missing from the Loki datasource it is not possible to do set it up this way.
Since I already have the `lat` and `long` available from the query, I also tried to use the
Location Data as a table, the setup ended up looking like:

{{< picture "loki-table-fail" "Setup of the Loki datasource as a table" "50%" >}}

Although this setup didn't produce any error it didn't visualize anything on the map either 🤷‍♂️.
The missing **piece of the puzzle** is that we can configure the Loki datasource as a Prometheus
datasource in Grafana 🤯.

To do this we need to create a Prometheus datasource in Grafana but point it out to the Loki
endpoint:

{{< picture "loki-and-prometheus-datasources" "Both datasources configured in the Grafana instance" >}}

As you can see we use the same URL, but we need to add the `/loki` path to the Prometheus datasource.

Using Loki as a Prometheus datasources allows us to use the same query as before, but have
access to the configuration options only available for a Prometheus datasource. We can now follow the
instructions in the [mentioned
post](https://www.robustperception.io/using-geohashes-with-the-worldmap-panel-and-prometheus) to
visualize the labels stored in a Loki datasource in the Worldmap panel.

## Summary

TL;DR you can configure a Loki datasource as a Prometheus datasource
in Grafana, which will provide the same bells and whistles of a normal Prometheus server. Keep in mind that
LogQL, the query language implemented by Loki, is a subset of PromQL, which means that not all
functions, aggregations or operators will be available.

This step of configuring Loki as a Prometheus datasource is mandatory (until the publishing date of
this post) to get the Worldmap Panel plugin playing nicely with your Loki datasource. I imagine that
in the no so far future Grafana's Loki support will be easier to use and it will change its behavior
depending on how you want to use the data, which will make things (like using a 3rd party plugin)
easier.
3 changes: 3 additions & 0 deletions layouts/shortcodes/picture.html
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{{ $width := default "100%" (.Get 2) }}

<picture>
{{ if not .Site.IsServer }}
<source
Expand All @@ -14,5 +16,6 @@
class="align-center"
alt="{{ .Get 1 }}"
loading="lazy"
style="max-width:{{ $width }};"
/>
</picture>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 9 additions & 8 deletions themes/jorgelbg/static/css/style.css
Original file line number Diff line number Diff line change
Expand Up @@ -266,11 +266,6 @@ li {
list-style-position: inside;
}

#post-body,
p {
line-height: 1.7;
}

b {
font-weight: 500;
color: var(--text-color);
Expand Down Expand Up @@ -763,8 +758,8 @@ h1 {
font-weight: 400;
font-style: normal;
font-size: 1.22rem;
line-height: 1.58;
letter-spacing: -0.003em;
line-height: 1.65;
letter-spacing: 0.025em;
}

.post h1 {
Expand Down Expand Up @@ -870,7 +865,7 @@ footer p.small {

.post header h1 svg {
display: none;
margin-left: -36px;
margin-left: -24px;
}

.post header h1 > a {
Expand Down Expand Up @@ -981,3 +976,9 @@ header .last-edited {
text-transform: uppercase;
margin-bottom: 30px;
}

picture {
display: flex;
justify-content: center;
margin-bottom: 20px;
}

0 comments on commit 832de4d

Please sign in to comment.