Duplicate metrics collector registration attempted #5414

marioanton · 2023-10-09T09:08:21Z

What's wrong?

Getting following errors randomly in our grafana k8s monitoring based setup.

duplicate metrics collector registration attempted

This turns to restart the pods which is not a problem but is far from ideal.

Steps to reproduce

Metric collection using Grafana k8s monitoring
https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring

This is using version 0.36.2 of the agent in our charts values.
FLOW MODE

System information

No response

Software version

0.36.2

Configuration

Default config to collect metrics from pods.

Logs

grafana-agent ts=2023-10-09T08:22:53.841697396Z component=prometheus.remote_write.grafana_cloud_prometheus subcomponent=rw level=info remote_name=0dd43f url=https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push msg="Remote storage stopped."
grafana-agent ts=2023-10-09T08:22:53.841723316Z level=info component=prometheus.remote_write.grafana_cloud_prometheus msg="component exited"
grafana-agent ts=2023-10-09T08:22:53.842856766Z level=info component=loki.source.kubernetes_events.cluster_events msg="component exited"
grafana-agent panic: duplicate metrics collector registration attempted [recovered]
grafana-agent     panic: duplicate metrics collector registration attempted [recovered]
grafana-agent     panic: duplicate metrics collector registration attempted
grafana-agent 
grafana-agent goroutine 264 [running]:
grafana-agent go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1()
grafana-agent     /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/span.go:383 +0x2a
grafana-agent go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc00a208c00, {0x0, 0x0, 0xbe42180?})
grafana-agent     /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/span.go:421 +0xa29
grafana-agent panic({0x67f6000, 0xc003b2eaa0})
grafana-agent     /usr/local/go/src/runtime/panic.go:884 +0x213
grafana-agent go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1()
grafana-agent     /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/span.go:383 +0x2a
grafana-agent go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc00a208f00, {0x0, 0x0, 0xbe42180?})
grafana-agent     /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/span.go:421 +0xa29
grafana-agent panic({0x67f6000, 0xc003b2eaa0})
grafana-agent     /usr/local/go/src/runtime/panic.go:884 +0x213
grafana-agent github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc00262a630, {0xc01d575430?, 0x1, 0xb?})
grafana-agent     /go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/wrap.go:104 +0x151
grafana-agent github.com/grafana/agent/component/common/loki/wal.NewWatcherMetrics({0x81bf9f0, 0xc00262a630})
grafana-agent     /src/agent/component/common/loki/wal/watcher_metrics.go:73 +0xaac
grafana-agent github.com/grafana/agent/component/common/loki/client.NewManager(0xc0026f0190?, {0x8170560, 0xc002626690}, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, ...)
grafana-agent     /src/agent/component/common/loki/client/manager.go:61 +0x8f
grafana-agent github.com/grafana/agent/component/loki/write.(*Component).Update(0xc0026ed8c0, {0x6b6c260?, 0xc003955c70?})
grafana-agent     /src/agent/component/loki/write/write.go:172 +0x5bd
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*ComponentNode).evaluate(0xc00262c000, 0x2?)
grafana-agent     /src/agent/pkg/flow/internal/controller/node_component.go:301 +0x37e
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*ComponentNode).Evaluate(0xc000593e00?, 0x0?)
grafana-agent     /src/agent/pkg/flow/internal/controller/node_component.go:252 +0x25
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*Loader).evaluate(0xc000a4edc0, {0x8170560, 0xc003955b80}, {0x81bb490, 0xc00262c000})
grafana-agent     /src/agent/pkg/flow/internal/controller/loader.go:647 +0x76
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*Loader).EvaluateDependencies.func2({0x8173720?, 0xc00262c000})
grafana-agent     /src/agent/pkg/flow/internal/controller/loader.go:621 +0x466
grafana-agent github.com/grafana/agent/pkg/flow/internal/dag.WalkReverse(0xc0026152a0, {0xc023f3fb80, 0x1, 0x3?}, 0xc023f3fd58)
grafana-agent     /src/agent/pkg/flow/internal/dag/walk.go:69 +0x1d9
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*Loader).EvaluateDependencies(0xc000a4edc0, 0xc002625d40)
grafana-agent     /src/agent/pkg/flow/internal/controller/loader.go:600 +0x826
grafana-agent github.com/grafana/agent/pkg/flow.(*Flow).Run(0xc002463880, {0x81d7ee8, 0xc000593630})
grafana-agent     /src/agent/pkg/flow/flow.go:237 +0x70f
grafana-agent github.com/grafana/agent/cmd/internal/flowmode.(*flowRun).Run.func5()
grafana-agent     /src/agent/cmd/internal/flowmode/cmd_run.go:254 +0x65
grafana-agent created by github.com/grafana/agent/cmd/internal/flowmode.(*flowRun).Run
grafana-agent     /src/agent/cmd/internal/flowmode/cmd_run.go:252 +0xcfe
Stream closed EOF for grafana/grafana-agent-flow-2 (grafana-agent)

The text was updated successfully, but these errors were encountered:

tpaschalis · 2023-10-09T11:20:22Z

Hey there @marioanton. It'd be great if you could provide the configuration that the Agents are running (you might be able to kubectl describe the mounted ConfigMaps there).

marioanton · 2023-10-09T12:56:06Z

Sure thing @tpaschalis
Bear in mind this is failing even when there is no prom config there, only logs, for metrics config still same, config is much bigger.
Generated via k8s monitoring chart


discovery.kubernetes "pods" {
  role = "pod"
}

// Pod Logs
discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"[]
    action = "replace"
    target_label = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"[]
    action = "replace"
    target_label = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"[]
    action = "replace"
    target_label = "container"
  }
  rule {
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_name"[]
    separator = "/"
    action = "replace"
    replacement = "$1"
    target_label = "job"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_node_name"[]
    action = "keep"
    regex = env("HOSTNAME")
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"[]
    separator = "/"
    action = "replace"
    replacement = "/var/log/pods/*$1/*.log"
    target_label = "__path__"
  }

  // set the container runtime as a label
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_container_id"[]
    regex = "^(\\w+):\\/\\/.+$"
    replacement = "$1"
    target_label = "tmp_container_runtime"
  }
  
  rule {
    regex = "__meta_kubernetes_pod_label_(.+)"
    action = "labelmap"
  }
}

local.file_match "pod_logs" {
  path_targets = discovery.relabel.pod_logs.output
}

loki.source.file "pod_logs" {
  targets    = local.file_match.pod_logs.targets
  forward_to = [loki.process.pod_logs.receiver[]
}

loki.process "pod_logs" {
  stage.match {
    selector = "{tmp_container_runtime=\"containerd\"}"
    // the cri processing stage extracts the following k/v pairs: log, stream, time, flags
    stage.cri {}

    // Set the extract flags and stream values as labels
    stage.labels {
      values = {
        flags  = "",
        stream  = "",
      }
    }
  }

  // if the label tmp_container_runtime from above is docker parse using docker
  stage.match {
    selector = "{tmp_container_runtime=\"docker\"}"
    // the docker processing stage extracts the following k/v pairs: log, stream, time
    stage.docker {}

    // Set the extract stream value as a label
    stage.labels {
      values = {
        stream  = "",
      }
    }
  }

  // drop the temporary container runtime label as it is no longer needed
  stage.label_drop {
    values = ["tmp_container_runtime"[]
  }
  stage.match {
    selector = "{cluster=~\".+\", traces!=\"root\"}"
    stage.drop {
      expression = "(?:\"level\"|\"severity\")?.*([=|:\"]info|\\[info\\]|INFO|debug|DEBUG).*"
    }
  
  }
  stage.label_drop {
      values = ["flags", "app_kubernetes_io_managed_by", "pod_template_hash", "stream", "pod_template_generation", "filename"[]
  }

  forward_to = [loki.write.grafana_cloud_loki.receiver[]
}

// Grafana Cloud Loki
local.file "loki_host" {
  filename  = "xxxxx"
}

local.file "loki_username" {
  filename  = "xxxx"
}

local.file "loki_password" {
  filename  = "xxxx"
  is_secret = true
}

loki.write "grafana_cloud_loki" {
  endpoint {
    url = nonsensitive(local.file.loki_host.content) + "/loki/api/v1/push"

    basic_auth {
      username = local.file.loki_username.content
      password = local.file.loki_password.content
    }
  }
  external_labels = {
    cluster = "flow-cluster",
  }
}

marioanton · 2023-10-09T12:58:00Z

Adding another trace for this logs daemonset.

grafana-agent ts=2023-10-09T12:54:57.289714391Z level=info boringcryptoenabled=false
grafana-agent ts=2023-10-09T12:54:57.290514511Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="starting complete graph evaluation"
grafana-agent ts=2023-10-09T12:54:57.290686001Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=local.file.loki_host duration=143.75µs
grafana-agent ts=2023-10-09T12:54:57.290722801Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=tracing duration=8.1µs
grafana-agent ts=2023-10-09T12:54:57.290845091Z level=info component=discovery.kubernetes.pods msg="Using pod service account via in-cluster config"
grafana-agent ts=2023-10-09T12:54:57.291137211Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=discovery.kubernetes.pods duration=403.62µs
grafana-agent ts=2023-10-09T12:54:57.291480891Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=discovery.relabel.pod_logs duration=321.08µs
grafana-agent ts=2023-10-09T12:54:57.291591511Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=local.file_match.pod_logs duration=86.26µs
grafana-agent ts=2023-10-09T12:54:57.291624281Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=logging duration=13.34µs
grafana-agent ts=2023-10-09T12:54:57.291639361Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=http duration=5.99µs
grafana-agent ts=2023-10-09T12:54:57.291660411Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=cluster duration=6.13µs
grafana-agent ts=2023-10-09T12:54:57.291678301Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=ui duration=9.71µs
grafana-agent ts=2023-10-09T12:54:57.291795551Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=local.file.loki_password duration=103.69µs
grafana-agent ts=2023-10-09T12:54:57.291912451Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=local.file.loki_username duration=97.39µs
grafana-agent ts=2023-10-09T12:54:57.292661841Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=loki.write.grafana_cloud_loki duration=640.65µs
grafana-agent ts=2023-10-09T12:54:57.293340861Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=loki.process.pod_logs duration=650.15µs
grafana-agent ts=2023-10-09T12:54:57.293674071Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished node evaluation" node_id=loki.source.file.pod_logs duration=307.58µs
grafana-agent ts=2023-10-09T12:54:57.293695801Z level=info controller_id= trace_id=fba49324ff9bc57b3cab9a3e01c9059a msg="finished complete graph evaluation" duration=3.35056ms
grafana-agent ts=2023-10-09T12:54:57.293795931Z level=info msg="scheduling loaded components and services"
grafana-agent ts=2023-10-09T12:54:57.293937881Z level=info msg="starting cluster node" peers= advertise_addr=127.0.0.1:80
grafana-agent ts=2023-10-09T12:54:57.294091831Z level=info controller_id= trace_id=cd7d670e98cd217278efdae27993e858 msg="starting partial graph evaluation"
grafana-agent ts=2023-10-09T12:54:57.294179961Z level=info msg="peers changed" new_peers=grafana-agent-flow-grafana-agent-logs-8n45x
grafana-agent ts=2023-10-09T12:54:57.294256481Z level=info controller_id= trace_id=cd7d670e98cd217278efdae27993e858 msg="finished node evaluation" node_id=loki.source.file.pod_logs duration=116.88µs
grafana-agent ts=2023-10-09T12:54:57.294282661Z level=info controller_id= trace_id=cd7d670e98cd217278efdae27993e858 msg="finished partial graph evaluation" duration=226.24µs
grafana-agent ts=2023-10-09T12:54:57.294368861Z level=info service=http msg="now listening for http traffic" addr=0.0.0.0:80
grafana-agent ts=2023-10-09T12:54:57.357622596Z level=info controller_id= trace_id=f3ae16949c57867cfbf2c1413c012519 msg="starting partial graph evaluation"
grafana-agent ts=2023-10-09T12:54:57.357850926Z level=info controller_id= trace_id=f3ae16949c57867cfbf2c1413c012519 msg="finished node evaluation" node_id=loki.write.grafana_cloud_loki duration=191.62µs
grafana-agent ts=2023-10-09T12:54:57.357885496Z level=info controller_id= trace_id=f3ae16949c57867cfbf2c1413c012519 msg="finished partial graph evaluation" duration=270.37µs
grafana-agent ts=2023-10-09T12:54:57.357951036Z level=info service=http msg="http server closed" addr=:80 err="http: Server closed"
grafana-agent ts=2023-10-09T12:54:57.357968876Z level=info component=discovery.relabel.pod_logs msg="component exited"
grafana-agent ts=2023-10-09T12:54:57.357960926Z level=info component=loki.write.grafana_cloud_loki msg="component exited"
grafana-agent ts=2023-10-09T12:54:57.357989896Z level=info component=loki.process.pod_logs msg="component exited"
grafana-agent ts=2023-10-09T12:54:57.357994676Z level=info component=local.file_match.pod_logs msg="component exited"
grafana-agent ts=2023-10-09T12:54:57.358022006Z level=info service=http msg="http server closed" addr=memory err="http: Server closed"
grafana-agent ts=2023-10-09T12:54:57.358040256Z level=info component=loki.source.file.pod_logs msg="loki.source.file component shutting down, stopping readers and positions file"
grafana-agent ts=2023-10-09T12:54:57.358041826Z level=info component=discovery.kubernetes.pods msg="component exited"
grafana-agent ts=2023-10-09T12:54:57.361138266Z level=info component=loki.source.file.pod_logs msg="component exited"
grafana-agent ts=2023-10-09T12:54:57.388291274Z level=info component=local.file.loki_password msg="component exited"
grafana-agent ts=2023-10-09T12:54:57.388334544Z level=info component=local.file.loki_username msg="component exited"
grafana-agent ts=2023-10-09T12:54:57.388350474Z level=info component=local.file.loki_host msg="component exited"
grafana-agent panic: duplicate metrics collector registration attempted
grafana-agent 
grafana-agent goroutine 246 [running]:
grafana-agent github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc002716c90, {0xc0006a0b70?, 0x1, 0xb?})
grafana-agent     /go/pkg/mod/github.com/prometheus/client_golang@v1.16.0/prometheus/wrap.go:104 +0x151
grafana-agent github.com/grafana/agent/component/common/loki/wal.NewWatcherMetrics({0x81bf9f0, 0xc002716c90})
grafana-agent     /src/agent/component/common/loki/wal/watcher_metrics.go:73 +0xaac
grafana-agent github.com/grafana/agent/component/common/loki/client.NewManager(0xc00274e320?, {0x8170560, 0xc00270ff90}, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, ...)
grafana-agent     /src/agent/component/common/loki/client/manager.go:61 +0x8f
grafana-agent github.com/grafana/agent/component/loki/write.(*Component).Update(0xc002756120, {0x6b6c260?, 0xc0012976d0?})
grafana-agent     /src/agent/component/loki/write/write.go:172 +0x5bd
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*ComponentNode).evaluate(0xc000b14b40, 0x0?)
grafana-agent     /src/agent/pkg/flow/internal/controller/node_component.go:301 +0x37e
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*ComponentNode).Evaluate(0xc0012abcc0?, 0xc0026e35d8?)
grafana-agent     /src/agent/pkg/flow/internal/controller/node_component.go:252 +0x25
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*Loader).evaluate(0xc0013754a0, {0x8170560, 0xc001297590}, {0x81bb490, 0xc000b14b40})
grafana-agent     /src/agent/pkg/flow/internal/controller/loader.go:647 +0x76
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*Loader).EvaluateDependencies.func2({0x8173720?, 0xc000b14b40})
grafana-agent     /src/agent/pkg/flow/internal/controller/loader.go:621 +0x466
grafana-agent github.com/grafana/agent/pkg/flow/internal/dag.WalkReverse(0xc0013f3c00, {0xc0026e3b80, 0x1, 0x3?}, 0xc0005f5d58)
grafana-agent     /src/agent/pkg/flow/internal/dag/walk.go:69 +0x1d9
grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*Loader).EvaluateDependencies(0xc0013754a0, 0xc000b14480)
grafana-agent     /src/agent/pkg/flow/internal/controller/loader.go:600 +0x826
grafana-agent github.com/grafana/agent/pkg/flow.(*Flow).Run(0xc0012d82a0, {0x81d7ee8, 0xc0012ab680})
grafana-agent     /src/agent/pkg/flow/flow.go:237 +0x70f
grafana-agent github.com/grafana/agent/cmd/internal/flowmode.(*flowRun).Run.func5()
grafana-agent     /src/agent/cmd/internal/flowmode/cmd_run.go:254 +0x65
grafana-agent created by github.com/grafana/agent/cmd/internal/flowmode.(*flowRun).Run
grafana-agent     /src/agent/cmd/internal/flowmode/cmd_run.go:252 +0xcfe
Stream closed EOF for grafana/grafana-agent-flow-grafana-agent-logs-8n45x (grafana-agent)
stream logs failed previous terminated container "config-reloader" in pod "grafana-agent-flow-grafana-agent-logs-8n45x" not found for grafana/grafana-agent-flow-grafana-agent-logs-8n45x (config-reloader)
stream logs failed previous terminated container "config-reloader" in pod "grafana-agent-flow-grafana-agent-logs-8n45x" not found for grafana/grafana-agent-flow-grafana-agent-logs-8n45x (config-reloader)
stream logs failed previous terminated container "config-reloader" in pod "grafana-agent-flow-grafana-agent-logs-8n45x" not found for grafana/grafana-agent-flow-grafana-agent-logs-8n45x (config-reloader)

marioanton · 2023-10-16T08:39:01Z

Is there any news around this?

marioanton · 2023-10-31T12:51:20Z

Anyone?

marioanton · 2023-10-31T13:44:51Z

again same i believe:
using 0.27.1

│ grafana-agent ts=2023-10-31T13:40:23.170060849Z level=info msg="finished node evaluation" controller_id="" node_id=loki.write.grafana_cloud_loki duration=27.984938ms                                              │
│ grafana-agent panic: duplicate metrics collector registration attempted                                                                                                                                            │
│ grafana-agent                                                                                                                                                                                                      │
│ grafana-agent goroutine 230 [running]:                                                                                                                                                                             │
│ grafana-agent github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc0029c5950, {0xc01271f100?, 0x1, 0xb?})                                                                          │
│ grafana-agent     /go/pkg/mod/github.com/prometheus/client_golang@v1.17.0/prometheus/wrap.go:104 +0x145                                                                                                            │
│ grafana-agent github.com/grafana/agent/component/common/loki/wal.NewWatcherMetrics({0x86ff2e0, 0xc0029c5950})                                                                                                      │
│ grafana-agent     /src/agent/component/common/loki/wal/watcher_metrics.go:73 +0x8ed                                                                                                                                │
│ grafana-agent github.com/grafana/agent/component/common/loki/client.NewManager(0x41b008?, {0x86b0340, 0xc0029a7cc0}, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, ...)                                                     │
│ grafana-agent     /src/agent/component/common/loki/client/manager.go:61 +0x85                                                                                                                                      │
│ grafana-agent github.com/grafana/agent/component/loki/write.(*Component).Update(0xc001e1f0e0, {0x6f7ede0?, 0xc003f0a6e0?})                                                                                         │
│ grafana-agent     /src/agent/component/loki/write/write.go:172 +0x589                                                                                                                                              │
│ grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*ComponentNode).evaluate(0xc0029c38c0, 0x2?)                                                                                                  │
│ grafana-agent     /src/agent/pkg/flow/internal/controller/node_component.go:297 +0x2b4                                                                                                                             │
│ grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*ComponentNode).Evaluate(0xc001cedcc0?, 0x86b3ea0?)                                                                                           │
│ grafana-agent     /src/agent/pkg/flow/internal/controller/node_component.go:251 +0x1c                                                                                                                              │
│ grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*Loader).concurrentEvalFn(0xc001f5a900, {0x86b3ea0?, 0xc0029c38c0}, {0x8723290, 0xc00ef700c0}, {0x86b3ec0, 0xc00ef70030}, 0xc0029c3440)       │
│ grafana-agent     /src/agent/pkg/flow/internal/controller/loader.go:663 +0x53e                                                                                                                                     │
│ grafana-agent github.com/grafana/agent/pkg/flow/internal/controller.(*Loader).EvaluateDependencies.func2()                                                                                                         │
│ grafana-agent     /src/agent/pkg/flow/internal/controller/loader.go:615 +0x33                                                                                                                                      │
│ grafana-agent github.com/grafana/agent/pkg/flow/internal/worker.(*shardedWorkerPool).SubmitWithKey.func1()                                                                                                         │
│ grafana-agent     /src/agent/pkg/flow/internal/worker/worker_pool.go:78 +0xa7                                                                                                                                      │
│ grafana-agent github.com/grafana/agent/pkg/flow/internal/worker.(*shardedWorkerPool).start.func1()                                                                                                                 │
│ grafana-agent     /src/agent/pkg/flow/internal/worker/worker_pool.go:119 +0x77                                                                                                                                     │
│ grafana-agent created by github.com/grafana/agent/pkg/flow/internal/worker.(*shardedWorkerPool).start in goroutine 1                                                                                               │
│ grafana-agent     /src/agent/pkg/flow/internal/worker/worker_pool.go:112 +0x28

marioanton added the bug Something isn't working label Oct 9, 2023

tpaschalis mentioned this issue Oct 9, 2023

Avoid using promauto for metrics registration grafana/dskit#394

Open

tpaschalis added the type/signals label Oct 9, 2023

rfratto removed the type/signals label Nov 2, 2023

tpaschalis self-assigned this Nov 2, 2023

tpaschalis mentioned this issue Nov 3, 2023

loki/wal: fix panic for duplicate registered metrics #5705

Merged

3 tasks

tpaschalis closed this as completed in #5705 Nov 3, 2023

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Feb 21, 2024

github-actions bot locked as resolved and limited conversation to collaborators Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Duplicate metrics collector registration attempted #5414

Duplicate metrics collector registration attempted #5414

marioanton commented Oct 9, 2023 •

edited

Loading

tpaschalis commented Oct 9, 2023

marioanton commented Oct 9, 2023 •

edited

Loading

marioanton commented Oct 9, 2023

marioanton commented Oct 16, 2023

marioanton commented Oct 31, 2023

marioanton commented Oct 31, 2023 •

edited

Loading

Duplicate metrics collector registration attempted #5414

Duplicate metrics collector registration attempted #5414

Comments

marioanton commented Oct 9, 2023 • edited Loading

What's wrong?

Steps to reproduce

System information

Software version

Configuration

Logs

tpaschalis commented Oct 9, 2023

marioanton commented Oct 9, 2023 • edited Loading

marioanton commented Oct 9, 2023

marioanton commented Oct 16, 2023

marioanton commented Oct 31, 2023

marioanton commented Oct 31, 2023 • edited Loading

marioanton commented Oct 9, 2023 •

edited

Loading

marioanton commented Oct 9, 2023 •

edited

Loading

marioanton commented Oct 31, 2023 •

edited

Loading