Enabling New Relic on v5.8.0 causing 'panic: runtime error: invalid memory address or nil pointer dereference' #5801

skreddy6673 · 2020-06-22T20:26:40Z

skreddy6673
Jun 22, 2020

Web nodes failed to start with following error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x191a9d1]

goroutine 12 [running]:
github.com/concourse/concourse/atc/metric/emitter.(*NewRelicEmitter).Emit(0xc0006cad20, 0x37277e0, 0xc00113a540, 0x2d50cf0, 0xe, 0x2953d60, 0x49ec6a0, 0x2d3d0ff, 0x2, 0xc000717650, ...)
        /tmp/build/1c3187db/concourse/atc/metric/emitter/newrelic.go:109 +0x891
github.com/concourse/concourse/atc/metric.emitLoop()
        /tmp/build/1c3187db/concourse/atc/metric/emit.go:148 +0x133
created by github.com/concourse/concourse/atc/metric.Initialize
        /tmp/build/1c3187db/concourse/atc/metric/emit.go:108 +0x3c8

Any work around or recommendations?

Answered by taylorsilva

Jul 2, 2020

This was fixed in 6.3.0 😄 #5697

I looked at the stack trace again. Between 5.7.0 and 5.8.0 the newrelic emitter was updated to also emit check containers.

concourse/atc/metric/emitter/newrelic.go

Line 29 in 4adfcb1

checks *stats

When we instantiate the emitter we forgot to instantiate the stat object for checks.

concourse/atc/metric/emitter/newrelic.go

Lines 66 to 78 in 4adfcb1

     return &NewRelicEmitter{  
   Client: client,  
   Url: "https://insights-collector.newrelic.com/v1/accounts/" + config.AccountID + "/events",  
   apikey: config.APIKey,  
   prefix: config.ServicePrefix,  
   containers: new(stats),  
   volumes: new(stats),  
   BatchSize: int(config.…

View full answer

skreddy6673 · 2020-06-22T21:33:52Z

skreddy6673
Jun 22, 2020
Author

I tried with v6.0.0 and got the same error

$ kubectl logs concourse-dev-web-6cb4d7545b-dp866 -n concourse-test
{"timestamp":"2020-06-22T21:31:58.559867203Z","level":"info","source":"atc","message":"atc.cmd.start","data":{"session":"1"}}
{"timestamp":"2020-06-22T21:31:58.559983958Z","level":"debug","source":"atc","message":"atc.metrics.metric-initialize","data":{"attributes":{},"buffer-size":1000,"host":"concourse-dev-web-6cb4d7545b-dp866","session":"2"}}
{"timestamp":"2020-06-22T21:31:58.772364724Z","level":"info","source":"atc","message":"atc.credential-manager.configured credentials manager","data":{"name":"kubernetes","session":"6"}}
{"timestamp":"2020-06-22T21:31:58.922276411Z","level":"info","source":"atc","message":"atc.cmd.finish","data":{"duration":64966,"session":"1"}}
{"timestamp":"2020-06-22T21:31:58.922643866Z","level":"info","source":"atc","message":"atc.lidar.start","data":{"session":"17"}}
{"timestamp":"2020-06-22T21:31:58.922700070Z","level":"info","source":"tsa","message":"tsa.listening","data":{}}
{"timestamp":"2020-06-22T21:31:58.922689374Z","level":"info","source":"atc","message":"atc.lidar.start","data":{"session":"17"}}
{"timestamp":"2020-06-22T21:31:58.922713568Z","level":"info","source":"atc","message":"atc.scheduler-interval-runner.start","data":{"interval":"10s","session":"20"}}
{"timestamp":"2020-06-22T21:31:58.922897753Z","level":"info","source":"atc","message":"atc.listening","data":{"debug":"127.0.0.1:8079","http":"0.0.0.0:8080"}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x199091c]

goroutine 11 [running]:
github.com/concourse/concourse/atc/metric/emitter.(*NewRelicEmitter).Emit(0xc000baf2c0, 0x39eed80, 0xc001198960, 0x2dcd5d8, 0xe, 0x0, 0xc0006ec450, 0xc000d29200, 0x22, 0xbfb466b63701625b, ...)
        /tmp/build/8cf986d9/concourse/atc/metric/emitter/newrelic.go:108 +0x81c
github.com/concourse/concourse/atc/metric.emitLoop()
        /tmp/build/8cf986d9/concourse/atc/metric/emit.go:138 +0x170
created by github.com/concourse/concourse/atc/metric.Initialize
        /tmp/build/8cf986d9/concourse/atc/metric/emit.go:98 +0x3c8

7 replies

taylorsilva Jun 24, 2020
Maintainer

I tried using your values.yaml but it had various small errors. Instead I modified my newrelic field looks like this:

    newrelic:
      enabled: true
      accountId: "ACC_ID"
      apiKey: "fakeapikey"
      servicePrefix: concourse_dev_

I was able to install Concourse on my GKE cluster without error:

$ k get pods -n new-relic
NAME                             READY   STATUS    RESTARTS   AGE
new-relic-postgresql-0           1/1     Running   0          44s
new-relic-web-78c7c577bf-wsxpz   1/1     Running   0          44s
new-relic-worker-0               1/1     Running   0          44s
new-relic-worker-1               1/1     Running   0          44s

If you do a kubectl describe pod and look in the Containers.Environment section do you see anything unexpected or malformed? Specifically the CONCOURSE_NEWRELIC_ flags? Mine looked like this (literally, I don't have an account or API key to use 😅 ):

      CONCOURSE_NEWRELIC_ACCOUNT_ID:                      ACC_ID
      CONCOURSE_NEWRELIC_API_KEY:                         fakeapikey
      CONCOURSE_NEWRELIC_SERVICE_PREFIX:                  concourse_dev_

skreddy6673 Jun 29, 2020
Author

Hi @taylorsilva
Sorry to disturb you.

$ kubectl get pods -n concourse-test
NAME                                 READY   STATUS    RESTARTS   AGE
concourse-dev-web-68549b7dc6-lvx75   0/1     Error     1          35s
concourse-dev-web-68549b7dc6-t8c4w   0/1     Error     0          35s

When i do describe pod i don't see anything unexpected.
Mine looks exactly same as your's except difference in values. I just copied everything just in case something else is causing the error.

Environment:
      CONCOURSE_CLUSTER_NAME:                          Concourse_Dev
      CONCOURSE_ENABLE_GLOBAL_RESOURCES:               true
      CONCOURSE_ENABLE_BUILD_AUDITING:                 true
      CONCOURSE_ENABLE_CONTAINER_AUDITING:             true
      CONCOURSE_ENABLE_JOB_AUDITING:                   true
      CONCOURSE_ENABLE_PIPELINE_AUDITING:              true
      CONCOURSE_ENABLE_RESOURCE_AUDITING:              true
      CONCOURSE_ENABLE_SYSTEM_AUDITING:                true
      CONCOURSE_ENABLE_TEAM_AUDITING:                  true
      CONCOURSE_ENABLE_WORKER_AUDITING:                true
      CONCOURSE_ENABLE_VOLUME_AUDITING:                true
      CONCOURSE_ENABLE_REDACT_SECRETS:                 true
      CONCOURSE_SECRET_RETRY_ATTEMPTS:                 5
      CONCOURSE_SECRET_RETRY_INTERVAL:                 1s
      CONCOURSE_LOG_LEVEL:                             debug
      CONCOURSE_BIND_PORT:                             8080
      CONCOURSE_BIND_IP:                               0.0.0.0
      CONCOURSE_ADD_LOCAL_USER:                        <set to the key 'local-users' in secret 'concourse-dev-web'>  Optional: false
      CONCOURSE_EXTERNAL_URL:                          URL
      CONCOURSE_DEBUG_BIND_IP:                         127.0.0.1
      CONCOURSE_DEBUG_BIND_PORT:                       8079
      CONCOURSE_INTERCEPT_IDLE_TIMEOUT:                0m
      CONCOURSE_GLOBAL_RESOURCE_CHECK_TIMEOUT:         24h
      CONCOURSE_RESOURCE_CHECKING_INTERVAL:            24h
      CONCOURSE_CONTAINER_PLACEMENT_STRATEGY:          fewest-build-containers
      CONCOURSE_BAGGAGECLAIM_RESPONSE_HEADER_TIMEOUT:  1m
      CONCOURSE_LOG_CLUSTER_NAME:                      Concourse_Dev
      CONCOURSE_BUILD_TRACKER_INTERVAL:                10s
      CONCOURSE_DEFAULT_BUILD_LOGS_TO_RETAIN:          10
      CONCOURSE_MAX_BUILD_LOGS_TO_RETAIN:              10
      CONCOURSE_DEFAULT_DAYS_TO_RETAIN_BUILD_LOGS:     7
      CONCOURSE_MAX_DAYS_TO_RETAIN_BUILD_LOGS:         7
      CONCOURSE_POSTGRES_HOST:                         HOST_URL
      CONCOURSE_POSTGRES_PORT:                         5432
      CONCOURSE_POSTGRES_USER:                         <set to the key 'postgresql-user' in secret 'concourse-dev-web'>      Optional: false
      CONCOURSE_POSTGRES_PASSWORD:                     <set to the key 'postgresql-password' in secret 'concourse-dev-web'>  Optional: false
      CONCOURSE_POSTGRES_SSLMODE:                      disable
      CONCOURSE_POSTGRES_CONNECT_TIMEOUT:              5m
      CONCOURSE_POSTGRES_DATABASE:                     atc
      CONCOURSE_KUBERNETES_IN_CLUSTER:                 true
      CONCOURSE_KUBERNETES_NAMESPACE_PREFIX:           concourse-
      CONCOURSE_NEWRELIC_ACCOUNT_ID:                   NEW_RELIC_ID
      CONCOURSE_NEWRELIC_SERVICE_PREFIX:               concourse_dev_
      CONCOURSE_GC_INTERVAL:                           30s
      CONCOURSE_GC_ONE_OFF_GRACE_PERIOD:               5m
      CONCOURSE_GC_MISSING_GRACE_PERIOD:               1m
      CONCOURSE_COOKIE_SECURE:                         true
      CONCOURSE_AUTH_DURATION:                         8760h
      CONCOURSE_SESSION_SIGNING_KEY:                   /concourse-keys/session_signing_key
      CONCOURSE_MAIN_TEAM_LOCAL_USER:                  concourse
      CONCOURSE_MAIN_TEAM_GITHUB_TEAM:                 MY_ORG:concourse-admins
      CONCOURSE_GITHUB_CLIENT_ID:                      <set to the key 'github-client-id' in secret 'concourse-dev-web'>      Optional: false
      CONCOURSE_GITHUB_CLIENT_SECRET:                  <set to the key 'github-client-secret' in secret 'concourse-dev-web'>  Optional: false
      CONCOURSE_GITHUB_HOST:                           github.my-org.net
      POD_IP:                                           (v1:status.podIP)
      CONCOURSE_PEER_ADDRESS:                          $(POD_IP)
      CONCOURSE_TSA_LOG_LEVEL:                         debug
      CONCOURSE_TSA_BIND_IP:                           0.0.0.0
      CONCOURSE_TSA_BIND_PORT:                         2222
      CONCOURSE_TSA_DEBUG_BIND_IP:                     127.0.0.1
      CONCOURSE_TSA_DEBUG_BIND_PORT:                   2221
      CONCOURSE_TSA_HOST_KEY:                          /concourse-keys/host_key
      CONCOURSE_TSA_AUTHORIZED_KEYS:                   /concourse-keys/worker_key.pub
      CONCOURSE_TSA_HEARTBEAT_INTERVAL:                30s
      CONCOURSE_NEWRELIC_API_KEY:                      <set to the key 'newrelic-api-key' in secret 'concourse-dev-web'>  Optional: false

Pod logs

$ kubectl logs concourse-dev-web-68549b7dc6-lvx75 -n concourse-test
{"timestamp":"2020-06-29T18:56:54.474811244Z","level":"info","source":"atc","message":"atc.cmd.start","data":{"session":"1"}}
{"timestamp":"2020-06-29T18:56:54.474983725Z","level":"debug","source":"atc","message":"atc.metrics.metric-initialize","data":{"attributes":{},"buffer-size":1000,"host":"concourse-dev-web-68549b7dc6-lvx75","session":"2"}}
{"timestamp":"2020-06-29T18:56:54.696085892Z","level":"info","source":"atc","message":"atc.credential-manager.configured credentials manager","data":{"name":"kubernetes","session":"6"}}
{"timestamp":"2020-06-29T18:56:54.830292933Z","level":"info","source":"atc","message":"atc.cmd.finish","data":{"duration":103309,"session":"1"}}
{"timestamp":"2020-06-29T18:56:54.830684590Z","level":"info","source":"atc","message":"atc.lidar.start","data":{"session":"17"}}
{"timestamp":"2020-06-29T18:56:54.830699679Z","level":"info","source":"tsa","message":"tsa.listening","data":{}}
{"timestamp":"2020-06-29T18:56:54.830684366Z","level":"info","source":"atc","message":"atc.scheduler-interval-runner.start","data":{"interval":"10s","session":"20"}}
{"timestamp":"2020-06-29T18:56:54.830694316Z","level":"info","source":"atc","message":"atc.lidar.start","data":{"session":"17"}}
{"timestamp":"2020-06-29T18:56:54.830891062Z","level":"info","source":"atc","message":"atc.listening","data":{"debug":"127.0.0.1:8079","http":"0.0.0.0:8080"}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x199091c]

goroutine 15 [running]:
github.com/concourse/concourse/atc/metric/emitter.(*NewRelicEmitter).Emit(0xc000b752c0, 0x39eed80, 0xc000132cc0, 0x2dcd5d8, 0xe, 0x0, 0xc0012c6f00, 0xc000da71d0, 0x22, 0xbfb6ac403185d35e, ...)
        /tmp/build/8cf986d9/concourse/atc/metric/emitter/newrelic.go:108 +0x81c
github.com/concourse/concourse/atc/metric.emitLoop()
        /tmp/build/8cf986d9/concourse/atc/metric/emit.go:138 +0x170
created by github.com/concourse/concourse/atc/metric.Initialize
        /tmp/build/8cf986d9/concourse/atc/metric/emit.go:98 +0x3c8

You mentioned my values file had various small errors. If you don't mind can you tell if they are related to my account details just like new relic ID API keys, username and passwords or something else?

Which version of helm chart did you used? If possible can you share your values files?

taylorsilva Jun 30, 2020
Maintainer

You mentioned my values file had various small errors.
They were unrelated to the NewRelic values. It was random stuff like: indention off by a space or two on various lines, parent keys deleted, new lines deleted making two lines into one line. Helm was catching all of these.

Helm: 2.16.9
GKE: 1.15.9
chart: latest version

I used the values.yaml on the master branch with this diff applied to it:

diff --git a/values.yaml b/values.yaml
index 61f4fa1..f598a31 100644
--- a/values.yaml
+++ b/values.yaml
@@ -661,7 +661,7 @@ concourse:
     emitToLogs: false

     newrelic:
-      enabled: false
+      enabled: true

       ## New Relic Insights Insert API base Url
       ## For configuring EU data centers refer to:
@@ -672,15 +672,15 @@ concourse:

       ## New Relic Account ID
       ##
-      accountId:
+      accountId: ACCOUNT_ID_HERE

       ## New Relic Insights API Key
       ##
-      apiKey:
+      apiKey: API_KEY_HERE

       ## An optional prefix for emitted New Relic events
       ##
-      servicePrefix:
+      servicePrefix: concourse_dev_

       ## Number of events to batch together before emitting
       ##

I deployed by running:

$ helm install ./ -n test --namespace test

and it work.

I noticed that your newrelic api key is in the web secret.

CONCOURSE_NEWRELIC_API_KEY:                      <set to the key 'newrelic-api-key' in secret 'concourse-dev-web'>  Optional: fals

Is this something you added to a fork of the chart? I noticed it's not part of our chart: https://github.com/concourse/concourse-chart/blob/master/templates/web-secrets.yaml
I'm wondering if the api key is being stored in a format not readable by Concourse?

skreddy6673 Jul 1, 2020
Author

I added the key in the chart itself but still got the same error,
yes, i don't want to expose this API key in the chart so stored it in web secret and calling it from here https://github.com/skreddy6673/concourse-chart/blob/07eb46797734b26cce9bbc89495dac192fa2670c/v6.0.0.yaml#L1446

I have another Instance of concourse v5.7.1 Helm chart version 8.2.13 running fine. Even this instance used to work with concourse v5.7.1 Helm chart version 8.2.13. When started to upgrade to v5.8.0 started to seeing this issue.

skreddy6673 Jul 1, 2020
Author

Helm v2.14.0
Concourse v5.8.0 and v6.0.0
kubernetes Client v1.14.8
Kubernetes Server v1.14.8-docker-1

taylorsilva · 2020-07-02T16:12:33Z

taylorsilva
Jul 2, 2020
Maintainer

This was fixed in 6.3.0 😄 #5697

I looked at the stack trace again. Between 5.7.0 and 5.8.0 the newrelic emitter was updated to also emit check containers.

concourse/atc/metric/emitter/newrelic.go

Line 29 in 4adfcb1

checks *stats

When we instantiate the emitter we forgot to instantiate the stat object for checks.

concourse/atc/metric/emitter/newrelic.go

Lines 66 to 78 in 4adfcb1

    
           return &NewRelicEmitter{ 
        
           	Client:             client, 
        
           	Url:                "https://insights-collector.newrelic.com/v1/accounts/" + config.AccountID + "/events", 
        
           	apikey:             config.APIKey, 
        
           	prefix:             config.ServicePrefix, 
        
           	containers:         new(stats), 
        
           	volumes:            new(stats), 
        
           	BatchSize:          int(config.BatchSize), 
        
           	BatchDuration:      config.BatchDuration, 
        
           	DisableCompression: config.DisableCompression, 
        
           	LastEmitTime:       time.Now(), 
        
           	NewRelicBatch:      make([]NewRelicEvent, 0), 
        
           }, nil

Concourse crashes when it tries to access the deleted field on the stat object that we never instantiated.

I was able to replicate this on 6.0.0 by:

Deploying Concourse using the helm config defined above, which ensures concourse tries to emit metrics via the newrelic emitter
Setting a pipeline with with resources to create check containers. I use the set-pipeline here to create multiple pipelines at once: https://github.com/concourse/examples/blob/master/pipelines/set-pipelines.yml
I deleted most of the pipelines to make concourse delete the check containers, thus it should add checks deleted events

concourse/atc/metric/emitter/newrelic.go

Lines 108 to 109 in 4adfcb1

case "checks deleted":

emitter.checks.deleted = event.Value

Concourse then crashed! Tried replicating on 6.3.0 and got no error

0 replies

skreddy6673 · 2020-07-02T18:33:23Z

skreddy6673
Jul 2, 2020
Author

Thank You @taylorsilva, Maybe it's time to upgrade. Will update you once it's done.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabling New Relic on v5.8.0 causing 'panic: runtime error: invalid memory address or nil pointer dereference' #5801

{{title}}

Replies: 3 comments 7 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

	return &NewRelicEmitter{
	Client: client,
	Url: "https://insights-collector.newrelic.com/v1/accounts/" + config.AccountID + "/events",
	apikey: config.APIKey,
	prefix: config.ServicePrefix,
	containers: new(stats),
	volumes: new(stats),
	BatchSize: int(config.…

Enabling New Relic on v5.8.0 causing 'panic: runtime error: invalid memory address or nil pointer dereference' #5801

skreddy6673 Jun 22, 2020

Replies: 3 comments · 7 replies

skreddy6673 Jun 22, 2020 Author

taylorsilva Jun 24, 2020 Maintainer

skreddy6673 Jun 29, 2020 Author

taylorsilva Jun 30, 2020 Maintainer

skreddy6673 Jul 1, 2020 Author

skreddy6673 Jul 1, 2020 Author

taylorsilva Jul 2, 2020 Maintainer

skreddy6673 Jul 2, 2020 Author

skreddy6673
Jun 22, 2020

Replies: 3 comments 7 replies

skreddy6673
Jun 22, 2020
Author

taylorsilva Jun 24, 2020
Maintainer

skreddy6673 Jun 29, 2020
Author

taylorsilva Jun 30, 2020
Maintainer

skreddy6673 Jul 1, 2020
Author

skreddy6673 Jul 1, 2020
Author

taylorsilva
Jul 2, 2020
Maintainer

skreddy6673
Jul 2, 2020
Author