This repo accompanies the blog post Going off-label with Grafana Loki
This repo contains all the code needed to stream tweets into either Loki, GEL, or Grafana Cloud Logs. This repo does not include any code for deploying Loki or GEL. Loki is very easy to run on your laptop but you can also sign up for a free Grafana Cloud account.
GEL stands for Grafana Enterprise Logs. GEL is the paid version of Loki with many features that large companies use, such as out-of-the-box authn/authz and a GUI for speed of initial onboarding.
Use stream.py to stream Tweets into either Loki, GEL, or Grafana Cloud Logs. stream.py relies on environment variables which are defined at the beginning of the script and explained below.
Use test-loki.sh or test-cloud-or-gel.sh to insert the data from test-data.json into either Loki or GEL. Both scripts use the same environment variables as stream.py.
Use time_in_ns.sh to get current timestamps for insertion into test-data.json so you can find them in Loki or GEL without setting a huge time range. The timestamps that are in test-data.json as you read this are dated far back in the past.
Docker, or you can use your own Python environment if that's in your skillset
- Run
open-docker-env.shto create an interactive bash shell in the latestpython:3container image - Run
pip install requests. The "requests" library is the only dependency that's not in the Python 3 standard library - Export the following environment variables:
TWITTER_BEARER_TOKENLOKI_URLLOKI_USERNAME(if required)LOKI_PASSWORD(if required)
- Run
python3 stream.py
The paths for Grafana Cloud Logs, Loki, and GEL are the same: /loki/api/v1/push.
- The code that streams Tweets from the Twitter API filters out tweets that are not in English. The filtering occurs on line 45
- There is a 1.3s wait between each Tweet. The wait occurs on line 47. The author's Twitter account has a limit of 2 million tweets a month and 1.3s gives me exactly 2m tweets if the script ran for a whole month.