Skip to content

chore: clustering mode is required for pyroscope #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Aug 18, 2025
Merged

chore: clustering mode is required for pyroscope #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
35328b9
chore: clustering mode is required for pyroscope
Emyrk Aug 15, 2025
95370e7
chore: enable agent clustering by default
Emyrk Aug 15, 2025
84276e2
compile.sh
Emyrk Aug 18, 2025
7271b1d
fix readme
Emyrk Aug 18, 2025
dddbfee
fix force flag
Emyrk Aug 18, 2025
1e3a810
force flag does not actuall work
Emyrk Aug 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ values which are defined [here](https://github.com/grafana/helm-charts/tree/main
| global.telemetry.profiling.scrape_interval | string | `"60s"` | how often the collector will scrape pprof endpoints |
| global.telemetry.profiling.scrape_timeout | string | `"60s"` | how long a request will be allowed to wait before being canceled |
| global.zone | string | `"svc"` | |
| grafana-agent.agent.clustering.enabled | bool | `false` | |
| grafana-agent.agent.clustering.enabled | bool | `true` | |
| grafana-agent.agent.configMap.create | bool | `false` | |
| grafana-agent.agent.configMap.key | string | `"config.river"` | |
| grafana-agent.agent.configMap.name | string | `"collector-config"` | |
Expand Down
9 changes: 8 additions & 1 deletion TESTING_KIND.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# Using kind to test observability

<details>
<summary>If using nix</summary>

`nix-shell -p kind kubernetes-helm kubectl`

</details>

To test the observability chart locally without a kubernetes cluster, you can use [`kind` (Kubernetes in Docker)](https://kind.sigs.k8s.io/). This allows you to create a local Kubernetes cluster that can be used for testing purposes.

```bash
Expand Down Expand Up @@ -86,7 +93,7 @@ helm install --namespace coder-observability --create-namespace observe .
## To update

```bash
helm upgrade --namespace coder-observability -force --create-namespace observe .
helm upgrade --namespace coder-observability --create-namespace observe .
```

Sometimes the config maps do not take. A quick way to reset the helm installation is just to uninstall and install.
Expand Down
4 changes: 4 additions & 0 deletions coder-observability/templates/_collector-config.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,10 @@ pyroscope.scrape "pods" {

scrape_interval = "{{ .Values.global.telemetry.profiling.scrape_interval }}"
scrape_timeout = "{{ .Values.global.telemetry.profiling.scrape_timeout }}"

clustering {
enabled = true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we enabling this by default with no means to disable it intentionally?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is set to true. When grafana agents are not running in cluster mode, this block is a no-op:

https://grafana.com/docs/agent/latest/flow/reference/components/pyroscope.scrape/#clustering-block

If Grafana Agent Flow is not running in clustered mode, this block is a no-op.

So you can disable this behavior by doing:

grafana-agent:
  agent:
    clustering:
      enabled: false # <- Now the clustering block in the agent config is a no-op

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I guess the question is, is there a use case to run agents in cluster mode, but not use cluster mode for the profiling.

As of now, I don't think there is. If we use more clustering features, then maybe we can add more knobs.

}
}

pyroscope.write "pods" {
Expand Down
2 changes: 1 addition & 1 deletion coder-observability/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ grafana-agent:
key: config.river
create: false
clustering:
enabled: false
enabled: true
extraArgs:
- --disable-reporting=true
mounts:
Expand Down
29 changes: 29 additions & 0 deletions compiled/resources.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10100,6 +10100,33 @@ subjects:
name: grafana
namespace: coder-observability
---
# Source: coder-observability/charts/grafana-agent/templates/cluster_service.yaml
apiVersion: v1
kind: Service
metadata:
name: grafana-agent-cluster
labels:
app.kubernetes.io/name: grafana-agent
app.kubernetes.io/instance: coder-observability
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
clusterIP: 'None'
selector:
app.kubernetes.io/name: grafana-agent
app.kubernetes.io/instance: coder-observability
ports:
# Do not include the -metrics suffix in the port name, otherwise metrics
# can be double-collected with the non-headless Service if it's also
# enabled.
#
# This service should only be used for clustering, and not metric
# collection.
- name: http
port: 80
targetPort: 80
protocol: "TCP"
---
# Source: coder-observability/charts/grafana-agent/templates/service.yaml
apiVersion: v1
kind: Service
Expand Down Expand Up @@ -10728,6 +10755,8 @@ spec:
- --storage.path=/tmp/agent
- --server.http.listen-addr=0.0.0.0:80
- --server.http.ui-path-prefix=/
- --cluster.enabled=true
- --cluster.join-addresses=grafana-agent-cluster
- --disable-reporting=true
env:
- name: AGENT_MODE
Expand Down