Can't get custom profiles (goroutines,etc)?

[rlex]

Not sure if my misconfig or bug. Installed pyroscope to kubernetes with helm chart, added profiles and stuff:
(pyroscope 0.8.0)

      - pyroscopeConfigs:
          log-level: debug
          scrape-configs:
            - job-name: 'kubernetes-pods'
              enabled-profiles: [cpu, mem, goroutines, mutex]
              profiles:
                goroutines:
                  path: /debug/pprof/goroutine
                  params:
                    seconds: ['10']
                  sample-types:
                    goroutines:
                      display-name: goroutines
                      units: objects
                      aggregation: average
                      cumulative: false
                      sampled: false
                mutex:
                  path: /debug/pprof/mutex
                  sample-types:
                    mutex:
                      units: objects
                      aggregation: average
                      cumulative: true
                cpu:
                  path: /debug/pprof/profile
                  params:
                    seconds: ['10']
                  sample-types:
                    samples:
                      display-name: cpu
                      units: samples
                      sampled: true
                mem:
                  path: /debug/pprof/heap
                  sample-types:
                    inuse_objects:
                      units: objects
                      aggregation: avg
                    inuse_space:
                      units: bytes
                      aggregation: avg
                    alloc_objects:
                      units: objects
                      cumulative: true
                    alloc_space:
                      units: bytes
                      cumulative: true
              kubernetes-sd-configs:
                - role: pod
              relabel-configs:
                - source-labels: [__meta_kubernetes_pod_annotation_pyroscope_io_scrape]
                  action: keep
                  regex: true
                - source-labels:
                    [__meta_kubernetes_pod_annotation_pyroscope_io_application_name]
                  action: replace
                  target-label: __name__
                - source-labels: [__meta_kubernetes_pod_annotation_pyroscope_io_scheme]
                  action: replace
                  regex: (https?)
                  target-label: __scheme__
                - source-labels:
                    [__address__, __meta_kubernetes_pod_annotation_pyroscope_io_port]
                  action: replace
                  regex: ([^:]+)(?::\d+)?;(\d+)
                  replacement: $1:$2
                  target-label: __address__
                - action: labelmap
                  regex: __meta_kubernetes_pod_label_(.+)
                - source-labels: [__meta_kubernetes_namespace]
                  action: replace
                  target-label: kubernetes_namespace
                - source-labels: [__meta_kubernetes_pod_name]
                  action: replace
                  target-label: kubernetes_pod_name
                - source-labels: [__meta_kubernetes_pod_phase]
                  regex: Pending|Succeeded|Failed|Completed
                  action: drop
                - action: labelmap
                  regex: __meta_kubernetes_pod_annotation_pyroscope_io_profile_(.+)
                  replacement: __profile_$1

checked /tmp/config.yaml in pod:

log-level: debug
scrape-configs:
- enabled-profiles:
  - cpu
  - mem
  - goroutines
  - mutex
  job-name: kubernetes-pods
  kubernetes-sd-configs:
  - role: pod
  profiles:
    cpu:
      params:
        seconds:
        - "10"
      path: /debug/pprof/profile
      sample-types:
        samples:
          display-name: cpu
          sampled: true
          units: samples
    goroutines:
      params:
        seconds:
        - "10"
      path: /debug/pprof/goroutine
      sample-types:
        goroutines:
          aggregation: average
          cumulative: false
          display-name: goroutines
          sampled: false
          units: objects
    mem:
      path: /debug/pprof/heap
      sample-types:
        alloc_objects:
          cumulative: true
          units: objects
        alloc_space:
          cumulative: true
          units: bytes
        inuse_objects:
          aggregation: avg
          units: objects
        inuse_space:
          aggregation: avg
          units: bytes
    mutex:
      path: /debug/pprof/mutex
      sample-types:
        mutex:
          aggregation: average
          cumulative: true
          units: objects
  relabel-configs:
  - action: keep
    regex: true
    source-labels:
    - __meta_kubernetes_pod_annotation_pyroscope_io_scrape
  - action: replace
    source-labels:
    - __meta_kubernetes_pod_annotation_pyroscope_io_application_name
    target-label: __name__
  - action: replace
    regex: (https?)
    source-labels:
    - __meta_kubernetes_pod_annotation_pyroscope_io_scheme
    target-label: __scheme__
  - action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    source-labels:
    - __address__
    - __meta_kubernetes_pod_annotation_pyroscope_io_port
    target-label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - action: replace
    source-labels:
    - __meta_kubernetes_namespace
    target-label: kubernetes_namespace
  - action: replace
    source-labels:
    - __meta_kubernetes_pod_name
    target-label: kubernetes_pod_name
  - action: drop
    regex: Pending|Succeeded|Failed|Completed
    source-labels:
    - __meta_kubernetes_pod_phase
  - action: labelmap
    regex: __meta_kubernetes_pod_annotation_pyroscope_io_profile_(.+)
    replacement: __profile_$1

For tests i chose prometheus, adding following to annotations:

              annotations:
                pyroscope.io/scrape: 'true'
                pyroscope.io/application-name: prometheus
                pyroscope.io/port: "9090"

also tried adding (should be redundant since i already specified them as default scrape job?)

                pyroscope.io/profile-cpu-enabled: "true"
                pyroscope.io/profile-mem-enabled: "true"
                pyroscope.io/profile-goroutines-enabled: "true"
                pyroscope.io/profile-mutex-enabled: "true"

However, i see no goroutines or mutex profiles available - only default 5 (cpu, objects/mem alloc/inuse).

[kolesnikovae]

@rlex Thank you for reporting this!

• With regard to /goroutine profile, I guess I know the reason - sample-types keys must match sample types specified in the profile: for /goroutine it should be goroutine (singular).
• Both /mutex and /block profiles contain samples of contentions and delay types. Typically, when there is no mutex/block contention, the profiles are empty and won't be shown in the UI: I'd check what /debug/pprof/mutex and /debug/pprof/block show. If they aren't empty let's see what can be wrong.
• /threadcreate profile is broken and won't show any meaningful data.

The configuration should look like this:

    enabled-profiles: [cpu, mem, goroutines, mutex, block]
    profiles:
      goroutines:
        path: /debug/pprof/goroutine
        sample-types:
          goroutine:
            display-name: goroutines
            units: objects
            aggregation: average
      mutex:
        path: /debug/pprof/mutex
        sample-types:
          contentions:
            display-name: mutex-contentions
            units: objects
            aggregation: average
          delay: # Not fully supported: needs handling nanoseconds unit properly.
            display-name: mutex-delay
            units: nanoseconds
            aggregation: average
      block:
        path: /debug/pprof/block
        sample-types:
          contentions:
            display-name: block-contentions
            units: objects
            aggregation: average
          delay:
            display-name: block-delay
            units: nanoseconds
            aggregation: average

Note that in UI you can see misleading description - we're planing to release support for all profile types in Go soon and this will be fixed.

also tried adding (should be redundant since i already specified them as default scrape job?)

That's right, the configuration takes precedence.

[rlex]

Yep, seems to be working now, thanks!
I pulled part of that config from here: https://pyroscope.io/docs/pull-mode/
Might be worth an update?

[kolesnikovae]

Uh, this a leftover, sorry! We plan to slightly redesign this custom profiles thing in the nearest future (in a backward-compatible way), therefore this part was removed from the documentation (but links are still accessible, yeah).

[kolesnikovae]

@rlex may I close the issue, given we have #775?

[rlex]

Yes, seems like everything is solved here. Thanks!