Skip to content

Commit

Permalink
rebrand Lightstep to Cloud Observability from ServiceNow and update p…
Browse files Browse the repository at this point in the history 
…rocedures to use OpenTelemetry
  • Loading branch information
Robin Whitmore committed Aug 28, 2023
1 parent dd9e35f commit 25a4d05
Showing 1 changed file with 189 additions and 147 deletions.
336 changes: 189 additions & 147 deletions content/en/docs/tasks/observability/distributed-tracing/lightstep/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
title: Lightstep
description: How to configure the proxies to send tracing requests to Lightstep.
title: Cloud Observability from ServiceNow (formerly Lightstep)
description: How to configure the proxies to send tracing requests to Cloud Observability (formerly Lightstep).
weight: 11
keywords: [telemetry,tracing,lightstep]
keywords: [telemetry,tracing,lightstep, servicenow, cloud observability]
aliases:
- /docs/tasks/telemetry/distributed-tracing/lightstep/
owner: istio/wg-policies-and-telemetry-maintainers
Expand All @@ -11,187 +11,229 @@ test: no

{{< boilerplate telemetry-tracing-tips >}}

This task shows you how to configure Istio to collect trace spans and send them to [Lightstep](https://lightstep.com).
Lightstep lets you analyze 100% of unsampled transaction data from large scale production software to produce meaningful
distributed traces and metrics that help explain performance behaviors and accelerate root cause analysis.
At the end of this task, Istio sends trace spans from the proxies to a Lightstep Satellite pool making them
available to the web UI. By default, all HTTP requests are captured (to see end-to-end traces, your code needs to forward OT headers even if it does not join the traces).
You use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to send Istio metrics to Cloud Observability using Istio proxies within each pod. Metrics are collected and forwarded to the OpenTelemetry Collector, which acts as a central collection and processing point and then sends those metrics to Cloud Observability.

If you only want to collect tracing spans directly from Istio (and not add specific instrumentation directly to your code), then you don't need to configure any tracers, as long as your services forward the [HTTP headers generated by traces](https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/headers#config-http-conn-man-headers-x-ot-span-context).
To set up Istio Metrics Ingestion using OpenTelemetry Collector, you:

This task uses the [Bookinfo](/docs/examples/bookinfo/) sample application as an example.
1. Create a configuration file for the OpenTelemetry Collector, specifying the sources, processors, and exporters to be used.
1. Create a configuration file for the Istio Operator for your Kubernetes environment.
1. Create access permissions for the Collector.
1. Configure Istio to send metrics to the OpenTelemetry Collector by modifying Istio's configuration files.
1. Create your [Cloud Observability access token](https://docs.lightstep.com/docs/create-and-manage-access-tokens) as a Kubernetes secret.
1. Verify that metrics from Istio are successfully ingested by the OpenTelemetry Collector and exported to Cloud Observability.

## Before you begin
## Prerequisites

1. Ensure you have a Lightstep account. [Sign up](https://go.lightstep.com/trial) for a free trial of Lightstep.
* Istio configured as a network mesh on a Kubernetes cluster
* A running OpenTelemetry Collector v0.77 or later, configured to [export metric data](https://docs.lightstep.com/docs/ingest-metrics-otel-collector) to Cloud Observability
* A good understanding of Kubernetes

1. If you're using [on-premise Satellites](https://docs.lightstep.com/docs/learn-about-satellites#on-premise-satellites), ensure you have a satellite pool configured with TLS certs and a secure GRPC port exposed. See
[Install and Configure Satellites](https://docs.lightstep.com/docs/install-and-configure-satellites) for details about setting up satellites.
## Configure the Collector

For [Lightstep Public Satellites](https://docs.lightstep.com/docs/learn-about-satellites#public-satellites) or [Developer Satellites](https://docs.lightstep.com/docs/learn-about-satellites#developer-satellites), your satellites are already configured. However you need to download [this certificate](https://docs.lightstep.com/docs/instrument-with-istio-as-your-service-mesh#cacertpem-file) to a local directory.
You use a Kubernetes ConfigMap file to configure the Collector to scrape Prometheus metrics and a deployment file to deploy it to Kubernetes.

1. Ensure sure you have a Lightstep [access token](https://docs.lightstep.com/docs/create-and-manage-access-tokens). Access tokens allow your app to communicate with your Lightstep project.
1. Create an `otel-collector-configmap.yaml` file by copying the following code.

## Deploy Istio
{{< text yaml >}}
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-conf
data:
otel-collector-config.yaml: |
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 5s
static_configs:
- targets: ['0.0.0.0:8888']
- job_name: "istio"
scrape_interval: 5s
metrics_path: "/stats/prometheus"
kubernetes_sd_configs:
- role: "pod"
relabel_configs:
// add labels

processors:
batch:

exporters:
logging:
loglevel: debug
otlp:
endpoint: ingest.lightstep.com:443
headers:
"lightstep-access-token": "{LS_ACCESS_TOKEN}"

service:
telemetry:
logs:
level: "debug"
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [logging,otlp]
{{/< text >}}

2. Create an `otel-collector-deployment.yaml` file by copying the following code.

How you deploy Istio depends on which type of Satellite you use.
{{< text yaml >}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
labels:
app: otel-collector
spec:
replicas: 1
selector:
matchLabels:
app: otel-collector
template:
metadata:
labels:
app: otel-collector
spec:
serviceAccountName: otel-collector
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:latest
args:
- "--config=/conf/otel-collector-config.yaml"
ports:
- containerPort: 55681
env:
- name: LS_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: lightstep-access-token
key: {LS_ACCESS_TOKEN}

volumeMounts:
- name: otel-collector-config-vol
mountPath: /conf
volumes:
- configMap:
name: otel-collector-conf
name: otel-collector-config-vol
{{< /text >}}

### Deploy Istio with On-Premise Satellites
## Configure Istio Operator

Create an `istio-operator.yaml` file by copying the following code.

{{< text yaml >}}
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
profile: default
components:
prometheus:
enabled: true
values:
global:
proxy:
autoInject: "enabled"
{{< /text >}}

These instructions do not assume TLS. If you are using TLS for your Satellite pool, follow the config for the [Public Satellite pool](#deploy-istio-with-public-or-developer-mode-satellites), but
use your own cert and your own pool's endpoint (`host:port`).
## Configure RBAC (Role-based access control)

Create an `otel-collector-rbac.yaml` file by copying the following code.

{{< text yaml >}}

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: otel-collector
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/metrics
- nodes/proxy
- nodes/stats
- pods
- services
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]

1. You need to deploy Istio with your Satellite address at an address in the format `<Host>:<Port>`, for example `lightstep-satellite.lightstep:9292`. You find this in your [configuration](https://docs.lightstep.com/docs/satellite-configuration-parameters#ports) file.
---

1. Deploy Istio with the following configuration parameters specified:
- `global.proxy.tracer="lightstep"`
- `meshConfig.defaultConfig.tracing.sampling=100`
- `meshConfig.defaultConfig.tracing.lightstep.address="<satellite-address>"`
- `meshConfig.defaultConfig.tracing.lightstep.accessToken="<access-token>"`
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: otel-collector
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: otel-collector
subjects:
- kind: ServiceAccount
name: otel-collector
namespace: default

You can set these parameters using the `--set key=value` syntax
when you run the install command. For example:
{{< /text >}}

{{< text bash >}}
$ istioctl install \
--set global.proxy.tracer="lightstep" \
--set meshConfig.defaultConfig.tracing.sampling=100 \
--set meshConfig.defaultConfig.tracing.lightstep.address="<satellite-address>" \
--set meshConfig.defaultConfig.tracing.lightstep.accessToken="<access-token>" \
{{< /text >}}
## Configure LS secret

### Deploy Istio with Public or Developer Mode Satellites
Create a `lightstep-secret.yaml` file to hold your access token by copying the following code and replacing `${LS_ACCESS_TOKEN}` with your access token.

Follow these steps if you're using the Public or Developer Mode Satellites, or if you're using on-premise Satellites with a TLS certificate.
{{< text yaml >}}
apiVersion: v1
kind: Secret
metadata:
name: lightstep-access-token
type: Opaque
data:
access_token: ${LS_ACCESS_TOKEN}
{{< /text yaml >}}

1. Store your satellite pool's certificate authority certificate as a secret in the default and `istio-system` namespace, the latter for use by the Istio gateways.
Download and use [this certificate](https://docs.lightstep.com/docs/instrument-with-istio-as-your-service-mesh#cacertpem-file).
If you deploy the Bookinfo application in a different namespace, create the secret in that namespace instead.
## Apply the configurations to your Kubernetes cluster

{{< text bash >}}
$ CACERT=$(cat Cert_Auth.crt | base64) # Cert_Auth.crt contains the necessary CACert
$ NAMESPACE=default
{{< /text >}}
1. Apply the deployment file.

{{< text bash >}}
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: lightstep.cacert
namespace: $NAMESPACE
labels:
app: lightstep
type: Opaque
data:
cacert.pem: $CACERT
EOF
kubectl apply -f otel-collector-deployment.yaml
{{< /text >}}

1. Deploy Istio with the following configuration parameters specified:
Verify that the OpenTelemetry Collector is running:

{{< text yaml >}}
global:
proxy:
tracer: "lightstep"
meshConfig:
defaultConfig:
tracing:
lightstep:
address: "ingest.lightstep.com:443"
accessToken: "<access-token>"
sampling: 100
tlsSettings
mode: "SIMPLE"
# Specifying ca certificate here will moute `lightstep.cacert` secret volume
# at all sidecars by default.
caCertificates="/etc/lightstep/cacert.pem"
components:
ingressGateways:
# `lightstep.cacert` secret volume needs to be mount at gateways via k8s overlay.
- name: istio-ingressgateway
enabled: true
k8s:
overlays:
- kind: Deployment
name: istio-ingressgateway
patches:
- path: spec.template.spec.containers[0].volumeMounts[-1]
value: |
name: lightstep-certs
mountPath: /etc/lightstep
readOnly: true
- path: spec.template.spec.volumes[-1]
value: |
name: lightstep-certs
secret:
secretName: lightstep.cacert
optional: true
{{< text bash >}}
kubectl get pods -l app=otel-collector
{{< /text >}}

## Install and run the Bookinfo app

1. Follow the [instructions to deploy the Bookinfo sample application](/docs/examples/bookinfo/#deploying-the-application).

1. Follow the [instructions to create an ingress gateway for the Bookinfo application](/docs/examples/bookinfo/#determine-the-ingress-ip-and-port).

1. To verify the previous step's success, confirm that you set `GATEWAY_URL` environment variable in your shell.

1. Send traffic to the sample application.
2. Apply the ConfigMap to your Kubernetes cluster

{{< text bash >}}
$ curl http://$GATEWAY_URL/productpage
kubectl apply -f otel-collector-configmap.yaml
{{< /text >}}

## Visualize trace data

1. Load the Lightstep [web UI](https://app.lightstep.com/). You'll see the three Bookinfo services listed in the Service Directory.

{{< image link="./istio-services.png" caption="Bookfinder services in the Service Directory" >}}

1. Navigate to the Explorer view.

{{< image link="./istio-explorer.png" caption="Explorer view" >}}
Verify that the ConfigMap has been created

1. Find the query bar at the top. The query bar allows you to interactively filter results by a **Service**, **Operation**, and **Tag** values.

1. Select `productpage.default` from the **Service** drop-down list.

1. Click **Run**. You see something similar to the following:

{{< image link="./istio-tracing-list-lightstep.png" caption="Explorer" >}}
{{< text bash >}}
kubectl get configmap otel-collector-conf
{{< /text >}}

1. Click on the first row in the table of example traces below the latency histogram to see the details
corresponding to your refresh of the `/productpage`. The page then looks similar to:
3. Apply the Secret to your Kubernetes cluster

{{< image link="./istio-tracing-details-lightstep.png" caption="Detailed Trace View" >}}
{{< text bash >}}
kubectl apply -f lightstep-secret.yaml
{{< /text >}}

The screenshot shows that the trace is comprised of a set of spans. Each span corresponds to a Bookinfo service invoked
during the execution of a `/productpage` request.
## View metrics in Cloud Observability

Two spans in the trace represent every RPC. For example, the call from `productpage` to `reviews` starts
with the span labeled with the `reviews.default.svc.cluster.local:9080/*` operation and the
`productpage.default: proxy client` service. This service represents the client-side span of the call. The screenshot shows
that the call took 15.30 ms. The second span is labeled with the `reviews.default.svc.cluster.local:9080/*` operation
and the `reviews.default: proxy server` service. The second span is a child of the first span and represents the
server-side span of the call. The screenshot shows that the call took 14.60 ms.
Once you have Cloud Observability ingesting the Istio metrics, you can begin using them to [build dashboards](https://docs.lightstep.com/docs/create-and-manage-dashboards) in Cloud Observability.

## Trace sampling

Istio captures traces at a configurable trace sampling percentage. To learn how to modify the trace sampling percentage,
visit the [Distributed Tracing trace sampling section](/docs/tasks/observability/distributed-tracing/mesh-and-proxy-config/#customizing-trace-sampling).

When using Lightstep, we do not recommend reducing the trace sampling percentage below 100%. To handle a high traffic mesh, consider scaling up the size of your satellite pool.

## Cleanup

If you are not planning any follow-up tasks, remove the Bookinfo sample application and any Lightstep secrets
from your cluster.

1. To remove the Bookinfo application, refer to the [Bookinfo cleanup](/docs/examples/bookinfo/#cleanup) instructions.

1. Remove the secret generated for Lightstep:

{{< text bash >}}
$ kubectl delete secret lightstep.cacert
{{< /text >}}

0 comments on commit 25a4d05

Please sign in to comment.