Add otel-cloud-stack chart #1058

jaronoff97 · 2024-03-01T17:36:14Z

Why make this?

This PR serves to finally close #562. This PR creates a helm chart that takes some inspiration from the kube-prometheus-stack chart from the prometheus community repo to create a fully-fledged collector starter for Kubernetes. Prometheus services and CRDs are optionally included to allow users a near-drop-in replacement for the prometheus experience. For non-prometheus users, this provides an easy way to get started with opentelemetry's Kubernetes offerings. This is accomplished by utilizing some clever templating for the configuration for the collector, which I found to be easier to reason about than embedding config in a helper tpl.

Features

Centralized tracing collector
Autoinstrumentation
Infrastructure metrics and logs collector
standalone kubernetes cluster stats collector
Allows for drop-in replacement for kube-prometheus-stack with a sharded statefulset collector
Allows for custom collectors to be set and validated against an exhaustive schema

Next steps

end to end testing to verify telemetry appears as expected between versions
documentation for what to expect OOTB
documentation for example configs (i include two samples below that are specific for my company)

For a config that gets you tracing, infra logs and metrics, kube cluster stats

clusterName: jacob-kind
opentelemetry-operator:
  enabled: true
collectors:
  tracing:
    env:
      - name: LS_TOKEN
        valueFrom:
          secretKeyRef:
            key: LS_TOKEN
            name: otel-collector-secret
    config:
      exporters:
        otlp:
          endpoint: ingest.lightstep.com:443
          headers:
            "lightstep-access-token": "${LS_TOKEN}"
      service:
        pipelines:
          traces:
            exporters: [otlp]
  daemon:
    env:
      - name: LS_TOKEN
        valueFrom:
          secretKeyRef:
            key: LS_TOKEN
            name: otel-collector-secret
    config:
      exporters:
        otlp:
          endpoint: ingest.lightstep.com:443
          headers:
            "lightstep-access-token": "${LS_TOKEN}"
      service:
        pipelines:
          metrics:
            exporters: [otlp]
          logs:
            exporters: [otlp]
  cluster:
    env:
      - name: LS_TOKEN
        valueFrom:
          secretKeyRef:
            key: LS_TOKEN
            name: otel-collector-secret
    config:
      exporters:
        otlp:
          endpoint: ingest.lightstep.com:443
          headers:
            "lightstep-access-token": "${LS_TOKEN}"
      service:
        pipelines:
          metrics:
            exporters: [otlp]
opAMPBridge:
  enabled: false

For a config that's a near drop in for kube-prometheus-stack

clusterName: jacob-kind
opentelemetry-operator:
  enabled: true
  manager:
    serviceMonitor:
      enabled: true
    extraArgs:
      - "--feature-gates=operator.observability.prometheus"
    image:
      tag: "v0.95.0"
collectors:
  tracing:
    enabled: false
  daemon:
    enabled: false
  cluster:
    enabled: false
  metrics:
    enabled: true
    targetAllocator:
      observability:
        metrics:
          enableMetrics: true
    observability:
      metrics:
        enableMetrics: true
    env:
      - name: LS_TOKEN
        valueFrom:
          secretKeyRef:
            key: LS_TOKEN
            name: otel-collector-secret
    config:
      exporters:
        otlp:
          endpoint: ingest.lightstep.com:443
          headers:
            "lightstep-access-token": "${LS_TOKEN}"
      service:
        pipelines:
          metrics:
            exporters: [otlp]
opAMPBridge:
  enabled: false
prometheus:
  customResources:
    enabled: true
kubernetesServiceMonitors:
  enabled: true
kubeApiServer:
  enabled: true
kubelet:
  enabled: true
kubeControllerManager:
  enabled: true
kubeDns:
  enabled: true
kubeEtcd:
  enabled: true
kubeScheduler:
  enabled: true
kubeProxy:
  enabled: true
kubeStateMetrics:
  enabled: true
nodeExporter:
  enabled: true

jaronoff97 · 2024-03-01T22:36:44Z

okay. so now this thing actually does something!! Next week I have to test out the kube-otel-stack functionality. But you can try for yourself by layering this values file over the existing values. (obviously this works for LS and youll needd to change to your vendor of choice ;))

Here's that config

clusterName: jacob-kind
opentelemetry-operator:
  enabled: true
collectors:
  tracing:
    env:
      - name: LS_TOKEN
        valueFrom:
          secretKeyRef:
            key: LS_TOKEN
            name: otel-collector-secret
    config:
      exporters:
        otlp:
          endpoint: ingest.lightstep.com:443
          headers:
            "lightstep-access-token": "${LS_TOKEN}"
      service:
        pipelines:
          traces:
            exporters: [otlp]
  daemon:
    env:
      - name: LS_TOKEN
        valueFrom:
          secretKeyRef:
            key: LS_TOKEN
            name: otel-collector-secret
    config:
      exporters:
        otlp:
          endpoint: ingest.lightstep.com:443
          headers:
            "lightstep-access-token": "${LS_TOKEN}"
      service:
        pipelines:
          metrics:
            exporters: [otlp]
          logs:
            exporters: [otlp]
  cluster:
    env:
      - name: LS_TOKEN
        valueFrom:
          secretKeyRef:
            key: LS_TOKEN
            name: otel-collector-secret
    config:
      exporters:
        otlp:
          endpoint: ingest.lightstep.com:443
          headers:
            "lightstep-access-token": "${LS_TOKEN}"
      service:
        pipelines:
          metrics:
            exporters: [otlp]
opAMPBridge:
  enabled: false

gen_schema/main.go

charts/otel-cloud-stack/kubelet_scrape_configs.yaml

jaronoff97 · 2024-03-04T22:50:19Z

charts/otel-cloud-stack/charts/crds/Chart.yaml

@@ -0,0 +1,3 @@
+apiVersion: v2


this matches what's done in kube-prometheus-stack such that we can install the necessary prometheus CRDs in the right order. the installation of this is entirely optional and disabled by default.

bismarck · 2024-03-05T19:36:21Z

@jaronoff97 it would be nice to have some self-monitoring of the operator and collectors? Not sure what form that would take (e.g. ServiceMonitor or static scrape job).

jaronoff97 · 2024-03-05T19:54:00Z

this is actually already possible and you can take a look at that in my sample configs in the For a config that's a near drop in for kube-prometheus-stack section :)

bismarck · 2024-03-06T19:51:36Z

@jaronoff97 I see you can bring your own ServiceAccount for your collector and target allocator. However, the ClusterRoleBinding is using the ServiceAccount that the operator is generating, not your override.

jaronoff97 · 2024-03-06T19:52:15Z

ah great catch! thank you.

TylerHelmuth

@jaronoff97 thanks for following through with this, I am very excited about it.

This is a large PR adding a lot of features. Is there anything we can do to make it smaller? My initial thought is we could start with it only installing the operator/collector and then in followup PRs introduce the target allocator and all the prom stuff.

Please add:

a README for the chart
a ci folder with an empty values.yaml so we can test the default install
please update the github workflow CI to test the chart.

TylerHelmuth · 2024-03-12T21:42:01Z

gen_schema/main.go

I'm guessing you used this to generate the schema file, do we need to check it in?

nope, i was thinking it may be helpful for future schema generation! but am happy to leave it out :)

I agree that it could be helpful in the future for all the charts. Let's add that in a separate pr.

sounds good!

TylerHelmuth · 2024-03-12T21:42:22Z

charts/otel-cloud-stack/Chart.yaml

@@ -0,0 +1,29 @@
+apiVersion: v2
+name: otel-cloud-stack
+version: 0.1.0


I'd rather start with 0.0.1

sounds good!

TylerHelmuth · 2024-03-12T21:44:32Z

charts/otel-cloud-stack/Chart.yaml

+  - name: jaronoff97
+  - name: anammedina21


Please list @TylerHelmuth @dmitryax and @Allex1. I'd love if you wanted to help maintain this chart and become an approver after it is merged, we did something similar with @puckpuck and the demo chart.

TylerHelmuth · 2024-03-12T21:45:35Z

charts/otel-cloud-stack/Chart.yaml

+    version: "0.0.0"
+    condition: prometheus.customResources.enabled
+  - name: opentelemetry-operator
+    repository: "file://../opentelemetry-operator"


Let's use https://open-telemetry.github.io/opentelemetry-helm-charts, thats what we do for the demo.

TylerHelmuth · 2024-03-12T21:45:59Z

charts/otel-cloud-stack/Chart.yaml

+  - name: jaronoff97
+  - name: anammedina21
+icon: https://raw.githubusercontent.com/cncf/artwork/a718fa97fffec1b9fd14147682e9e3ac0c8817cb/projects/opentelemetry/icon/color/opentelemetry-icon-color.png
+appVersion: 0.94.0


This appversion corresponds with the collector, but the sources only list the operator.

yeah i was thinking this would determine the version for everything this chart creates (read: collectors) but i also should pipe this in to the image used for the operator too

TylerHelmuth · 2024-03-12T21:50:11Z

charts/otel-cloud-stack/daemon_scrape_configs.yaml

What does this files do? It isn't part of template so it isn't included in any install of the chart. Is it an expected scrape config to use with the chart?

this is actually included in the chart via the logic in the collector.yaml. The chart dynamically pulls these values in and appends them (or creates a new) prometheus scrape config in the prometheus receiver

Should we move them in a configs folder?

TylerHelmuth · 2024-03-12T21:50:15Z

charts/otel-cloud-stack/kubelet_scrape_configs.yaml

What does this files do? It isn't part of template so it isn't included in any install of the chart. Is it an expected scrape config to use with the chart?

TylerHelmuth · 2024-03-12T21:51:01Z

charts/otel-cloud-stack/values.yaml

+# Top level field related to the OpenTelemetry Operator
+opentelemetry-operator:
+  # Field indicating whether the operator is enabled or not
+  enabled: false


Should this be enabled by default? The goal of the chart is to install an operator/collector all in one go right?

yep probably! for my testing it was easier to not have it by default initially, but i can change that.

TylerHelmuth · 2024-03-12T21:52:57Z

charts/otel-cloud-stack/values.yaml

+
+# This is the default configuration for all collectors generated by the chart.
+# Any collectors in the `collectors` are overlayed on top of this configuration.
+defaultCollectorConfig:


I think this name is misleading - it implies to me it configures the collector's configuration when it is configuring the OpenTelemetryCollector resource. Could we name it opentelemetry-collector?

see comment below for why this is done, i agree this could probably have a better name. baseCollectorCRConfig?

TylerHelmuth · 2024-03-12T21:53:41Z

charts/otel-cloud-stack/values.yaml

+  #   mountPath: /etc/config
+
+# Top level field specifying collectors configuration
+collectors:


I'd prefer to nest all things related to the collector under 1 root section.

the benefit of doing it this way is that anyone can add any collectors they want created to this key and they will be created as part of the chart. It also allows them to easily overlay values for specific collectors in different environments. i.e. if someone wanted to make a custom collector xcol and have a different sample rate in each env/region combo ([dev,prod] x [us-east-1, us-east-2]) they would be able to have a single xcol definition and only overlay the new sample rate rather than redefine the entire configuration.

jaronoff97 · 2024-03-12T22:38:13Z

@TylerHelmuth i think i could split this up in to the following sections:

hierarchical things (chart.yaml, unfilled values-yaml, some templates with nothing)
embed operator deployment
default collector configuration
autoinstrumentation
opamp bridge
prometheus features
some sample collector configurations for a true quickstart

do i need to also make a rendered folder to test that?

TylerHelmuth · 2024-03-13T16:55:40Z

@jaronoff97 I like that approach.

some sample collector configurations for a true quickstart

We can handle this via an example folder like the other charts.

jaronoff97 · 2024-03-13T20:37:14Z

@TylerHelmuth I see, so we would have some files like example/kube-prom-stack which has the values files overrides for doing that? This, rather than baking in these defaults?

TylerHelmuth · 2024-03-13T21:08:36Z

I see, so we would have some files like example/kube-prom-stack which has the values files overrides for doing that? This, rather than baking in these defaults?

I am not sure yet, I need to understand the how the default config files are used. The examples folder is strictly for viewing, they shouldn't be used or referenced in any templates

jaronoff97 · 2024-03-13T21:50:29Z

i see, in this case the chart is meant to serve as a quickstart with opentelemetry. Hopefully giving users full visibility in to their kube clusters the "otel" way. Or allowing people on prometheus with the kube-prometheus-stack a near exact replica but with the collector

TylerHelmuth · 2024-03-13T21:55:29Z

@jaronoff97 yup, I was referring to those yaml files. I may not understand what you meant by rendered folder.

Regardless, let's move forward with incremental PRs.

jaronoff97 · 2024-03-13T21:56:40Z

Closing this for now in favor of the approach specified here

jaronoff97 added 15 commits February 28, 2024 18:15

beginning

6255372

okay i think this is mostly accurate

b18552c

get ready to use a list

4f9f491

Collectors

9845575

instrumentation

a9b06c5

better naming

b94b0ac

dump schemas

d26c163

okay all the objects

607e746

whole kaboodle

f8c3e1e

Add prometheus components

b54a6a0

ignore ds stores

ec5eec1

begin doing stufff

b43c6b5

automatic bridge stuff

0cf5d29

updates to make it work

e3f4bed

fix env var naming

cfcf185

fix endpoints

8c138e2

jaronoff97 commented Mar 1, 2024

View reviewed changes

gen_schema/main.go Outdated Show resolved Hide resolved

Allex1 reviewed Mar 4, 2024

View reviewed changes

charts/otel-cloud-stack/kubelet_scrape_configs.yaml Show resolved Hide resolved

jaronoff97 added 2 commits March 4, 2024 17:01

update for prom crds

b549e6b

remove my values

b4f5703

jaronoff97 marked this pull request as ready for review March 4, 2024 22:49

jaronoff97 requested a review from a team as a code owner March 4, 2024 22:49

jaronoff97 commented Mar 4, 2024

View reviewed changes

remove todo

57b07f4

jaronoff97 mentioned this pull request Mar 5, 2024

Fix monitoring of collector when in daemonset mode lightstep/otel-collector-charts#72

Merged

bridge off by default

7cfa3b2

fix clusterrole template

3aae5ca

jaronoff97 requested a review from Allex1 March 11, 2024 15:19

some extra cleanup

f656d98

TylerHelmuth reviewed Mar 12, 2024

View reviewed changes

jaronoff97 closed this Mar 13, 2024

Add otel-cloud-stack chart #1058

Add otel-cloud-stack chart #1058

Conversation

jaronoff97 commented Mar 1, 2024 • edited Loading

Why make this?

Features

Next steps

jaronoff97 commented Mar 1, 2024 • edited Loading

Choose a reason for hiding this comment

bismarck commented Mar 5, 2024

jaronoff97 commented Mar 5, 2024

bismarck commented Mar 6, 2024

jaronoff97 commented Mar 6, 2024

TylerHelmuth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaronoff97 commented Mar 12, 2024

TylerHelmuth commented Mar 13, 2024

jaronoff97 commented Mar 13, 2024

TylerHelmuth commented Mar 13, 2024

jaronoff97 commented Mar 13, 2024

TylerHelmuth commented Mar 13, 2024

jaronoff97 commented Mar 13, 2024

jaronoff97 commented Mar 1, 2024 •

edited

Loading

jaronoff97 commented Mar 1, 2024 •

edited

Loading