Table of Contents
Tempo is is a distributed tracing backend by Grafana, supporting Jaeger, Zipkin, and OpenTelemetry protocols.
Tempo is a single binary that can be configured to run in several modes to take up different roles in the ingestion/read pipeline, such as distributor, compactor and so on, read the official docs for a more detailed explanation..
Charmed Tempo consists of two charms, a coordinator and a worker.
The coordinator is responsible for creating the configuration files for each worker and deciding whether the cluster is in a coherent state, based on the roles that each worker has adopted.
Several worker applications can be deployed and related to the coordinator, taking on different roles and enabling independent scalability of the various components of the stack.
The coordinator charm acts as a single access point for bundle-level integrations, such as TLS, ingress, self-monitoring, etc..., and as a single source of truth for bundle-level configurations that otherwise would have to be repeated (and kept in sync) between the individual worker nodes.
This Juju bundle deploys Tempo and a small object storage server, consisting of the following interrelated charmed operators:
Note
s3-integrator itself doesn't act as an S3 object storage system. For the bundle to work, s3-integrator needs to point to something that is S3-compatible. You can achieve this through:
- Adding S3 credentials provided by your S3 storage to
s3-integratorby runningsync-s3-credentialsaction and configurings3-integratorwith relevant configurations to your storage. - A MinIO charm deployment or a similar charm that does the S3 work. See Deploying MinIO
This bundle is under development. Join us on:
Before deploying the bundle you may want to create a dedicated model for Tempo components:
juju add-model tempoThe bundle, by default, deploys Tempo in its scaling monolithic mode, which runs all required roles within a single process. Scaling monolithic mode is the way to deploy Grafana Tempo with horizontal scale out by instantiating more than one worker process and is useful if you want to get started quickly or want to work with Grafana Tempo in a development environment.
Note
What makes this mode 'monolithic' is that the worker application is configured to run with role=all
# render bundle with "edge" charms
tox -e render-bundle -- --channel=edge
juju deploy ./bundle.yaml --trustThe bundle will deploy:
- a
tempoapplication with 1 unit (tempo-coordinator-k8scharm) - a
tempo-workerapplication with 3 units (tempo-worker-k8scharm), configured withrole=all - an
s3-integratorapplication with 1 unit (s3-integratorcharm)
Note
The default number of worker units is an odd number to prevent split-brain situations, and greater than 1 for the deployment to be HA.
The bundle can also be rendered to deploy Tempo in its microservices mode, which runs each one of the required roles in distinct processes. Each required role is assigned to one worker unit to achieve the minimal deployment for the cluster to be considered consistent. Minimal microservices mode is the way to deploy Grafana Tempo in a distributed architecture without high availability.
Pass the --mode=minimal-microservices argument to render the bundle for this mode:
# render bundle with "edge" charms
tox -e render-bundle -- --channel=edge --mode=minimal-microservices
juju deploy ./bundle.yaml --trustThe bundle will deploy:
- a
tempoapplication with 1 unit (tempo-coordinator-k8scharm) - a
tempo-querierapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-query-frontendapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-ingesterapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-distributorapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-compactorapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-metrics-generatorapplication with 1 unit (tempo-worker-k8scharm) - an
s3-integratorapplication with 1 unit (s3-integratorcharm)
Finally, the bundle can be rendered to deploy the 'recommended' amount of nodes for its microservices mode. In this scenario, each worker node is scaled up to the recommended number of units. This is the recommended way to deploy a highly available Grafana Tempo for production environments according to Grafana Tempo's helm chart configuration.
Pass the --mode=recommended-microservices argument to render the bundle for this mode:
# render bundle with "edge" charms
tox -e render-bundle -- --channel=edge --mode=recommended-microservices
juju add-model tempo-prod
juju deploy ./bundle.yaml --trustThe bundle will deploy:
- a
tempoapplication with 1 unit (tempo-coordinator-k8scharm) - a
tempo-querierapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-query-frontendapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-ingesterapplication with 3 units (tempo-worker-k8scharm) - a
tempo-distributorapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-compactorapplication with 1 unit (tempo-worker-k8scharm) - a
tempo-metrics-generatorapplication with 1 unit (tempo-worker-k8scharm) - an
s3-integratorapplication with 1 unit (s3-integratorcharm)
We also make available some overlays for convenience:
- the
tlsoverlay adds an internal CA to encrypt all inter-workload communications.
In order to use the overlays above, you need to:
- Download the overlays (or clone the repository)
- Pass the
--overlay <path-to-overlay-file-1> --overlay <path-to-overlay-file-2> ...arguments to thejuju deploycommand
For example, to deploy the Tempo bundle with the tls overlay, you would do the following:
curl -L https://raw.githubusercontent.com/canonical/tempo-bundle/main/overlays/tls-overlay.yaml -O
juju deploy tempo-bundle --channel=edge --trust --overlay ./tls-overlay.yamltox -e render-bundle -- --channel=edge # creates bundle.yaml
charmcraft pack
charmcraft upload tempo-bundle.zip
charmcraft release tempo-bundle --channel=edge --revision=1