[prometheusreceiver] Panic on shutdown

[povilasv]

Component(s)

receiver/prometheus

What happened?

Description

Sometimes collector panics on shutdown with the following:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3e29dd0]

goroutine 1 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).Shutdown(0x4000df5d40, {0x0, 0x0})
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.66.0/metrics_receiver.go:315 +0x20
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0x4001370730, {0x6bd7500, 0x400018c000})
        go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:121 +0x38c
go.opentelemetry.io/collector/service.(*service).Shutdown(0x4000810400, {0x6bd7500, 0x400018c000})
        go.opentelemetry.io/collector@v0.66.0/service/service.go:121 +0xd0
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0x4000ef3a70, {0x6bd7500?, 0x400018c000?})

Steps to Reproduce

Run collector v0.66 with my config

Expected Result

Actual Result

Collector version

v0.66.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

exporters:
      logging: {}
    extensions:
      file_storage:
        directory: /var/log/pods/
      health_check: {}
      memory_ballast:
        size_in_percentage: 40
    processors:
      batch: {}
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
    receivers:
      filelog:
        exclude:
        - /var/log/pods/sys-mon_chart-1669291265-opentelemetry-collector*_*/opentelemetry-collector/*.log
        include:
        - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
        operators:
        - id: get-format
          routes:
          - expr: body matches "^\\{"
            output: parser-docker
          - expr: body matches "^[^ Z]+ "
            output: parser-crio
          - expr: body matches "^[^ Z]+Z"
            output: parser-containerd
          type: router
        - id: parser-crio
          output: extract_metadata_from_filepath
          regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
          timestamp:
            layout: "2006-01-02T15:04:05.000000000-07:00"
            layout_type: gotime
            parse_from: attributes.time
          type: regex_parser
        - id: parser-containerd
          output: extract_metadata_from_filepath
          regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
          timestamp:
            layout: '%Y-%m-%dT%H:%M:%S.%LZ'
            parse_from: attributes.time
          type: regex_parser
        - id: parser-docker
          output: extract_metadata_from_filepath
          timestamp:
            layout: '%Y-%m-%dT%H:%M:%S.%LZ'
            parse_from: attributes.time
          type: json_parser
        - id: extract_metadata_from_filepath
          parse_from: attributes["log.file.path"]
          regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
          type: regex_parser
        - from: attributes.stream
          to: attributes["log.iostream"]
          type: move
        - from: attributes.container_name
          to: resource["k8s.container.name"]
          type: move
        - from: attributes.namespace
          to: resource["k8s.namespace.name"]
          type: move
        - from: attributes.pod_name
          to: resource["k8s.pod.name"]
          type: move
        - from: attributes.restart_count
          to: resource["k8s.container.restart_count"]
          type: move
        - from: attributes.uid
          to: resource["k8s.pod.uid"]
          type: move
        - from: attributes.log
          to: body
          type: move
        start_at: beginning
        storage: file_storage
      jaeger:
        protocols:
          grpc:
            endpoint: 0.0.0.0:14250
          thrift_compact:
            endpoint: 0.0.0.0:6831
          thrift_http:
            endpoint: 0.0.0.0:14268
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      prometheus:
        config:
          scrape_configs:
          - job_name: opentelemetry-collector
            scrape_interval: 10s
            static_configs:
            - targets:
              - ${MY_POD_IP}:8888
      zipkin:
        endpoint: 0.0.0.0:9411
    service:
      extensions:
      - health_check
      - memory_ballast
      - file_storage
      pipelines:
        logs:
          exporters:
          - logging
          processors:
          - memory_limiter
          - batch
          receivers:
          - otlp
          - filelog
        metrics:
          exporters:
          - logging
          processors:
          - memory_limiter
          - batch
          receivers:
          - otlp
          - prometheus
        traces:
          exporters:
          - logging
          processors:
          - memory_limiter
          - batch
          receivers:
          - otlp
          - jaeger
          - zipkin
      telemetry:
        logs:
          level: debug
        metrics:
          address: 0.0.0.0:8888

Log output

``
2022-11-24T12:04:34.496Z        info    pipelines/pipelines.go:102      Receiver is starting... {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-11-24T12:04:34.496Z        info    adapter/receiver.go:55  Starting stanza receiver        {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-11-24T12:04:34.497Z        info    zapgrpc/zapgrpc.go:174  [core] [Server #3 ListenSocket #4] ListenSocket created {"grpc_log": true}
2022-11-24T12:04:34.586Z        info    service/service.go:115  Starting shutdown...
2022-11-24T12:04:34.586Z        info    healthcheck/handler.go:129      Health Check state change       {"kind": "extension", "name": "health_check", "status": "unavailable"}
2022-11-24T12:04:34.587Z        info    pipelines/pipelines.go:118      Stopping receivers...
2022-11-24T12:04:34.588Z        info    zapgrpc/zapgrpc.go:174  [core] [Server #1 ListenSocket #2] ListenSocket deleted {"grpc_log": true}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3e29dd0]

goroutine 1 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).Shutdown(0x4000df5d40, {0x0, 0x0})
        github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.66.0/metrics_receiver.go:315 +0x20
go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0x4001370730, {0x6bd7500, 0x400018c000})
        go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:121 +0x38c
go.opentelemetry.io/collector/service.(*service).Shutdown(0x4000810400, {0x6bd7500, 0x400018c000})
        go.opentelemetry.io/collector@v0.66.0/service/service.go:121 +0xd0
go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0x4000ef3a70, {0x6bd7500?, 0x400018c000?})
        go.opentelemetry.io/collector@v0.66.0/service/collector.go:264 +0x30
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0x4000ef3a70, {0x6bd7500, 0x400018c000})
        go.opentelemetry.io/collector@v0.66.0/service/collector.go:166 +0x1fc
go.opentelemetry.io/collector/service.(*Collector).Run(0x4000ef3a70, {0x6bd7500, 0x400018c000})
        go.opentelemetry.io/collector@v0.66.0/service/collector.go:190 +0x30
go.opentelemetry.io/collector/service.NewCommand.func1(0x40001cb800, {0x5f1280e?, 0x1?, 0x1?})
        go.opentelemetry.io/collector@v0.66.0/service/command.go:53 +0x3b8
github.com/spf13/cobra.(*Command).execute(0x40001cb800, {0x40001a6010, 0x1, 0x1})
        github.com/spf13/cobra@v1.6.1/command.go:916 +0x5e0
github.com/spf13/cobra.(*Command).ExecuteC(0x40001cb800)
        github.com/spf13/cobra@v1.6.1/command.go:1044 +0x368
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.6.1/command.go:968
main.runInteractive({{0x400086c2d0, 0x400086d4a0, 0x400086c6f0, 0x40007f5f50}, {{0x5f3b360, 0xf}, {0x5fb82c8, 0x1f}, {0x5f0c283, 0x6}}, ...})
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32 +0x40
main.run(...)
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
        github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25 +0x124

Additional context

I believe I have some bug in other extension, but the error is masked by this panic.

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.