Skip to content

Files

Latest commit

 

History

History
 
 

istio

README.md

Istio check

Overview

Datadog monitors every aspect of your Istio environment, so you can:

  • Assess the health of Envoy and the Istio control plane with logs (see below).
  • Break down the performance of your service mesh with request, bandwidth, and resource consumption metrics (see below).
  • Map network communication between containers, pods, and services over the mesh with Network Performance Monitoring.
  • Drill into distributed traces for applications transacting over the mesh with APM.

To learn more about monitoring your Istio environment with Datadog, see the Istio blog.

Setup

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

Installation

Istio is included in the Datadog Agent. Install the Datadog Agent on your Istio servers or in your cluster and point it at Istio.

Envoy

If you want to monitor the Envoy proxies in Istio, configure the Envoy integration.

Configuration

Edit the istio.d/conf.yaml file (in the conf.d/ folder at the root of your Agent's configuration directory) to connect to Istio. See the sample istio.d/conf.yaml for all available configuration options.

Metric collection

To monitor the istiod deployment and istio-proxy in Istio v1.5+ there are two key components involved to collect the Prometheus-formatted metrics. Aligning with the Istio architecture, there is the data plane (the istio-proxy sidecar containers) and the control plane (the istiod service managing the proxies). These are both run as istio Agent checks, but have different responsibilities, and they are configured separately, as described below.

Data plane configuration

To monitor the Istio data plane, the Agent includes an istio.d/auto_conf.yaml file to automatically set up the monitoring for each of the istio-proxy sidecar containers. The Agent initializes this check for each sidecar container that it detects automatically. This configuration enables the reporting of istio.mesh.* metrics for the data exposed by each of these sidecar containers.

To customize the data plane portion of the integration, create an equivalent configuration file for Istio. Set ad_identifiers and istio_mesh_endpoint appropriately to set up the integration when an istio-proxy sidecar container is discovered. Refer to the provided istio.d/auto_conf.yaml and the example configuration file for all available configuration options. When you customize, set use_openmetrics: true and the exclude_labels to the following configuration:

    exclude_labels:
      - source_version
      - destination_version
      - source_canonical_revision
      - destination_canonical_revision
      - source_principal
      - destination_principal
      - source_cluster
      - destination_cluster
      - source_canonical_service
      - destination_canonical_service
      - source_workload_namespace
      - destination_workload_namespace
      - request_protocol
      - connection_security_policy
Control plane configuration

To monitor the Istio control plane and report the mixer, galley, pilot, and citadel metrics, you must configure the Agent to monitor the istiod deployment. In Istio v1.5+, apply the following Autodiscovery Annotations as pod annotations for the deployment istiod in the istio-system namespace:

ad.datadoghq.com/discovery.check_names: '["istio"]'
ad.datadoghq.com/discovery.init_configs: '[{}]'
ad.datadoghq.com/discovery.instances: |
     [
       {
         "istiod_endpoint": "http://%%host%%:15014/metrics",
         "use_openmetrics": "true"
       }
     ]

The method for applying these annotations varies depending on the Istio deployment strategy (Istioctl, Helm, Operator) used. Consult the Istio docs for the proper method to apply these pod annotations.

These annotations reference discovery as the <CONTAINER_IDENTIFIER> to match the default container name of the pods for the istiod deployment. If your container name is different, adjust accordingly.

OpenMetrics V2 vs OpenMetrics V1
Important Note: If you have multiple instances of Datadog collecting Istio metrics, make sure to use the same implementation of OpenMetrics for all of them. Otherwise, the metrics data fluctuates on the Datadog site.

When you enable the use_openmetrics configuration option, the Istio integration uses the OpenMetrics V2 implementation of the check.

In OpenMetrics V2, metrics are submitted more accurately by default and behave closer to Prometheus metric types. For example, Prometheus metrics ending in _count and _sum are submitted as monotonic_count by default.

OpenMetrics V2 addresses performance and quality issues in OpenMetrics V1. Updates include native metric types support, improved configuration, and custom metric types.

Set the use_openmetrics configuration option to false to use the OpenMetrics V1 implementation. To view the configuration parameters for OpenMetrics V1, see the conf.yaml.example file.

Disable sidecar injection for Datadog Agent pods

If you are installing the Datadog Agent in a container, Datadog recommends that you first disable Istio's sidecar injection.

Istio versions >= 1.10:

Add the sidecar.istio.io/inject: "false" label to the datadog-agent DaemonSet:

# (...)
spec:
  template:
    metadata:
      labels:
        sidecar.istio.io/inject: "false"
    # (...)

This can also be done with the kubectl patch command.

kubectl patch daemonset datadog-agent -p '{"spec":{"template":{"metadata":{"labels":{"sidecar.istio.io/inject":"false"}}}}}'

Istio versions <= 1.9:

Add the sidecar.istio.io/inject: "false" annotation to the datadog-agent DaemonSet:

# (...)
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    # (...)

Using the kubectl patch command:

kubectl patch daemonset datadog-agent -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}}}}}'

Log collection

Istio contains two types of logs. Envoy access logs that are collected with the Envoy integration and Istio logs.

Available for Agent versions >6.0

See the Autodiscovery Integration Templates for guidance on applying the parameters below. Collecting logs is disabled by default in the Datadog Agent. To enable it, see Kubernetes Log Collection.

Parameter Value
<LOG_CONFIG> {"source": "istio", "service": "<SERVICE_NAME>"}

Validation

Run the Agent's info subcommand and look for istio under the Checks section.

Data Collected

Metrics

See metadata.csv for a list of metrics provided by this check.

Events

The Istio check does not include any events.

Service Checks

See service_checks.json for a list of service checks provided by this integration.

Troubleshooting

Invalid chunk length error

If you see the following error on OpenMetricsBaseCheck (V1) implementation of Istio (Istio integration version 3.13.0 or older):

  Error: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)",
  InvalidChunkLength(got length b'', 0 bytes read))

You can use the Openmetrics V2 implementation of the Istio integration to resolve this error.

Note: you must upgrade to at minimum Agent 7.31.0 and Python 3. See the Configuration section on enabling Openmetrics V2.

Using the generic Openmetrics integration in an Istio deployment

If Istio proxy sidecar injection is enabled, monitoring other Prometheus metrics using the Openmetrics integration with the same metrics endpoint as istio_mesh_endpoint can result in high custom metrics usage and duplicated metric collection.

To ensure that your Openmetrics configuration does not redundantly collect metrics, either:

  1. Use specific metric matching in the metrics configuration option, or
  2. If using the wildcard * value for metrics, consider using the following Openmetrics integration options to exclude metrics already supported by the Istio and Envoy integrations.

Openmetrics V2 configuration with generic metric collection

Be sure to exclude Istio and Envoy metrics from your configuration to avoid high custom metrics billing. Use exclude_metrics if using the Openmetrics V2 configuration (openmetrics_endpoint enabled).

## Every instance is scheduled independent of the others.
#
instances:
  - openmetrics_endpoint: <OPENMETRICS_ENDPOINT>
    metrics: [*]
    exclude_metrics:
      - istio_*
      - envoy_*

Openmetrics V1 configuration (Legacy) with generic metric collection

Be sure to exclude Istio and Envoy metrics from your configuration to avoid high custom metrics billing. Use ignore_metrics if using the Openmetrics V1 configuration (prometheus_url enabled).

instances:
  - prometheus_url: <PROMETHEUS_URL>
    metrics:
      - *
    ignore_metrics:
      - istio_*
      - envoy_*

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles: