Skip to content

fix(observability): keep prometheus in profile set when enabling grafana sidecar - #239

Merged
zheli merged 1 commit into
mainfrom
fix/observability-grafana-prometheus-profile
Jul 10, 2026
Merged

fix(observability): keep prometheus in profile set when enabling grafana sidecar#239
zheli merged 1 commit into
mainfrom
fix/observability-grafana-prometheus-profile

Conversation

@srikanth-bitdynamics

Copy link
Copy Markdown
Collaborator

Supersedes #238 (same commit; the branch was renamed to remove a non-descriptive prefix — #238 auto-closed when its head branch was renamed).

Problem

dpm localnet observability enable --name <inst> (and the Web UI Metrics "Enable observability now" button, which shares the orchestration) fails on a running instance with:

observability toggle failed: enable grafana: docker compose up: exit status 1
output:
service "grafana" depends on undefined service "prometheus": invalid compose project

Reproduced on Splice 0.6.4, Compose v5.0.2, on a healthy instance created without --profile observability.

Root cause

SetObservability brings up prometheus and grafana in two separate docker compose up invocations, each activating only its own per-component profile. The grafana step ran --profile grafana up -d grafana, but the overlay (assets/compose/observability.yaml) gates prometheus behind profiles: ["observability","prometheus"] and grafana declares depends_on: prometheus. With only the grafana profile active, Compose filters prometheus out of the project model, so grafana's depends_on references an undefined service and the whole project is rejected. This bit even the enable-both path: prometheus comes up fine, then the separate grafana invocation fails.

Fix

Whenever grafana is started, activate both --profile prometheus --profile grafana so prometheus is a defined service (satisfying depends_on), and scope the up with --no-deps so activating that profile does not drag a prometheus container up in the legitimate grafana-only / external-scrape case (which the existing warning documents).

  • New observabilityUpProfiles + observabilityUpArgs helpers build the argv; enableObservabilitySidecar drops its now-redundant profile param.
  • Kept the overlay's depends_on — it still orders startup in the umbrella up.go path.
  • Both the CLI verb and the Web UI Metrics toggle go through the shared SetObservability, so CLI ↔ Web UI parity holds with no per-surface docker drift.

Verified scenarios: grafana onto an already-running prometheus, both from cold, and grafana-only (prometheus defined-but-not-started, warning still surfaced).

Test

TestObservabilityUpArgs_GrafanaKeepsPrometheusInProfileSet materializes the real overlay (injecting a bytes.Buffer for the drift writer per the repo testing rule) and asserts the generated grafana up argv keeps prometheus in the active profile set and passes --no-deps, while a prometheus-only up does not activate the grafana profile.

make test and make lint are green.

Related (not fixed here)

The same root cause also affects cold-start dpm localnet up --profile grafana (grafana-only): up.go runs a whole-project up under --profile grafana with prometheus's profile inactive, producing the identical rejection. That path can't use the toggle's --no-deps trick (no explicit service list), so the clean fix there is to drop grafana's depends_on and rely on runtime datasource discovery — a separate change to the overlay's dependency contract, left out of this scoped fix.

…ana sidecar

Enabling grafana via the observability toggle ran `docker compose
--profile grafana up -d grafana`, which filtered prometheus out of the
project model and broke grafana's `depends_on: prometheus`
("depends on undefined service prometheus: invalid compose project").
This failed even the enable-both path, where prometheus comes up fine
and the separate grafana invocation then rejects the project.

Activate the prometheus profile alongside grafana whenever grafana is
started, and scope the up with --no-deps so activating that profile
makes the dependency target defined without dragging a prometheus
container up in a grafana-only (external-scrape) setup. Both the CLI
verb and the Web UI Metrics toggle go through the shared
SetObservability, so parity holds.
@zheli
zheli merged commit e3a7a65 into main Jul 10, 2026
8 checks passed
@zheli
zheli deleted the fix/observability-grafana-prometheus-profile branch July 10, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants