Memory issue(?) with remote_write on ARMv7 with

[vpetersson]

(Somewhat of a follow-up on #2663)

What did you do?

I'm trying to send logs remotely using remote_write on ARMv7.

What did you expect to see?

I expected Prometheus to send a payload to the URL specified in remote_write.

What did you see instead? Under which circumstances?

All. It works fine if I remove the remote_write stanza.

Environment

A Raspberry Pi 3 Model B running ResinOS.

In case the memory allocation is a concern, here is a dump of that:

# free -m
             total       used       free     shared    buffers     cached
Mem:           972        541        431         49         60        264
-/+ buffers/cache:        215        756
Swap:            0          0          0

• System information:

Linux 4.4.48 armv7l

• Prometheus version:

prometheus, version 1.6.1 (branch: master, revision: 4666df502c0e239ed4aa1d80abbbfb54f61b23c3)
  build user:       root@ca1b452514b7
  build date:       20170419-13:53:08
  go version:       go1.8.1

• Prometheus configuration file:

# cat /etc/prometheus.yml
# my global config
global:
  scrape_interval:     30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'tmp-resin'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'tmp'
    static_configs:
      - targets: ['localhost:8000']

remote_write:
    - url: https://requestb.in/abc123

• Logs:

# prometheus -config.file=/etc/prometheus.yml
INFO[0000] Starting prometheus (version=1.6.1, branch=master, revision=4666df502c0e239ed4aa1d80abbbfb54f61b23c3)  source=main.go:88
INFO[0000] Build context (go=go1.8.1, user=root@ca1b452514b7, date=20170419-13:53:08)  source=main.go:89
INFO[0000] Loading configuration file /etc/prometheus.yml  source=main.go:251
INFO[0000] Loading series map and head chunks...         source=storage.go:421
INFO[0000] 0 series loaded.                              source=storage.go:432
INFO[0000] Starting target manager...                    source=targetmanager.go:61
INFO[0000] Listening on :9090                            source=web.go:259
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11580]

goroutine 182 [running]:
sync/atomic.addUint64(0x12448474, 0x1, 0x0, 0x1035a2c, 0x1244844c)
        /usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
github.com/prometheus/prometheus/storage/remote.(*ewmaRate).incr(0x12448474, 0x1, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/remote/ewma.go:65 +0x30
github.com/prometheus/prometheus/storage/remote.(*shards).enqueue(0x123175c0, 0x1250f060, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:427 +0x38
github.com/prometheus/prometheus/storage/remote.(*QueueManager).Append(0x12448400, 0x1250ed20, 0x1265e960, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:239 +0x190
github.com/prometheus/prometheus/storage/remote.(*Writer).Append(0x1265e960, 0x1250ed20, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/remote/write.go:79 +0x8c
github.com/prometheus/prometheus/storage.Fanout.Append(0x124c6c80, 0x2, 0x2, 0x1250ed20, 0x123902f0, 0x125d9338)
        /go/src/github.com/prometheus/prometheus/storage/storage.go:60 +0x58
github.com/prometheus/prometheus/storage.(*Fanout).Append(0x124c71a0, 0x1250ed20, 0x14047d14, 0x123902d3)
        <autogenerated>:3 +0x64
github.com/prometheus/prometheus/retrieval.(*countingAppender).Append(0x126da0a0, 0x1250ed20, 0x14047cfc, 0x125d9378)
        /go/src/github.com/prometheus/prometheus/retrieval/target.go:261 +0x3c
github.com/prometheus/prometheus/retrieval.ruleLabelsAppender.Append(0x1e053b8, 0x126da0a0, 0x1254c140, 0x1250ed20, 0x126da0c0, 0x1265340)
        /go/src/github.com/prometheus/prometheus/retrieval/target.go:201 +0x138
github.com/prometheus/prometheus/retrieval.(*ruleLabelsAppender).Append(0x126da0c0, 0x1250ed20, 0x1e06ad8, 0x126da0c0)
        <autogenerated>:35 +0x64
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0x12328140, 0x124be700, 0xb, 0xc8, 0xe, 0x256ae0f6, 0x1eabaa0)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:519 +0x224
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0x12328140, 0xfc23ac00, 0x6, 0x540be400, 0x2, 0x0)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:429 +0x4e0
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:258 +0x330

[vpetersson]

Just FYI, if I comment out the remote_write stanza, it loads and I get the following output:

# prometheus -config.file=/etc/prometheus.yml
INFO[0000] Starting prometheus (version=1.6.1, branch=master, revision=4666df502c0e239ed4aa1d80abbbfb54f61b23c3)  source=main.go:88
INFO[0000] Build context (go=go1.8.1, user=root@ca1b452514b7, date=20170419-13:53:08)  source=main.go:89
INFO[0000] Loading configuration file /etc/prometheus.yml  source=main.go:251
INFO[0001] Loading series map and head chunks...         source=storage.go:421
WARN[0001] Persistence layer appears dirty.              source=persistence.go:846
WARN[0001] Starting crash recovery. Prometheus is inoperational until complete.  source=crashrecovery.go:40
WARN[0001] To avoid crash recovery in the future, shut down Prometheus with SIGTERM or a HTTP POST to /-/quit.  source=crashrecovery.go:41
INFO[0001] Scanning files.                               source=crashrecovery.go:55
INFO[0001] File scan complete. 0 series found.           source=crashrecovery.go:83
INFO[0001] Checking for series without series file.      source=crashrecovery.go:85
INFO[0001] Check for series without series file complete.  source=crashrecovery.go:131
INFO[0001] Cleaning up archive indexes.                  source=crashrecovery.go:411
INFO[0001] Clean-up of archive indexes complete.         source=crashrecovery.go:504
INFO[0001] Rebuilding label indexes.                     source=crashrecovery.go:512
INFO[0001] Indexing metrics in memory.                   source=crashrecovery.go:513
INFO[0001] Indexing archived metrics.                    source=crashrecovery.go:521
INFO[0001] All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)  source=crashrecovery.go:540
WARN[0001] Crash recovery complete.                      source=crashrecovery.go:153
INFO[0001] 0 series loaded.                              source=storage.go:432
INFO[0001] Starting target manager...                    source=targetmanager.go:61
INFO[0001] Listening on :9090                            source=web.go:259
[...]

No crash as far as I can tell.

[juliusv]

It doesn't look like a nil pointer exception, but some other segmentation fault that happens in the arm64 implementation of the sync/atomic package.

I suspected an alignment issue like in elastic/beats#3273, but the field we atomically increment is already the first in its struct, so it should be aligned properly: https://github.com/prometheus/prometheus/blob/master/storage/remote/ewma.go#L24

From https://golang.org/pkg/sync/atomic/:

"On both ARM and x86-32, it is the caller's responsibility to arrange for 64-bit alignment of 64-bit words accessed atomically. The first word in a global variable or in an allocated struct or slice can be relied upon to be 64-bit aligned."

So now I don't know if this is a Go bug or what... I don't have an ARM laying around.

I wonder if you could try compiling the following Go program for your rpi and see if it crashes?

https://play.golang.org/p/Wz42HYfwT2

[vpetersson]

@juliusv

It doesn't seem like it crashes from what I can tell:

root@3c7a719:~# go build test.go
root@3c7a719:~# ls -l
total 412
-rwxr-xr-x 1 root root 414536 May  1 18:33 test
-rw-r--r-- 1 root root    139 May  1 18:32 test.go
root@3c7a719:~# ./test
root@3c7a719:~# echo $?
0

[juliusv]

@vpetersson Thanks! Strange. Did you also use Go 1.8.1 for this, which was the Go version from which your broken Prometheus was built?

Maybe I should get a Raspberry Pi :) @discordianfish Can you reproduce this on yours, perhaps?

[vpetersson]

@juliusv No, this was not 1.8.1. I simply installed the golang apt package available for Raspbian:

$ go version
go version go1.3.3 linux/arm

[juliusv]

Ah. Since I'm suspecting a Go bug, it would be super helpful if someone could try it out with Go 1.8.1.

[vpetersson]

@juliusv Pulled 1.8.1 from upstream and ran it again. Same result:

root@3c7a719:~# /usr/local/go/bin/go build test.go
root@3c7a719:~# ./test
root@3c7a719:~# echo $?
0
root@3c7a719:~# /usr/local/go/bin/go version
go version go1.8.1 linux/arm

[juliusv]

Thanks! Hm ok. Anyone want to donate an rpi to me? :) Or rather, looks like someone who has one has to dig deeper into this...

[vpetersson]

I can send you one. I've followed you on Twitter. Follow me back and DM me your address.

[grobie]

You can find the address in https://github.com/prometheus/prometheus/blob/master/MAINTAINERS.md

…

On Wed, May 3, 2017 at 11:50 AM Viktor Petersson ***@***.***> wrote: I can send you one. I've followed you on Twitter. Follow me back and DM me your address. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2666 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAANaJigiDQAqmxasv2wMM5-GuasWMk0ks5r2E3KgaJpZM4NL5_Q> .

[vpetersson]

@grobie It's kinda hard to FedEx a parcel to an email address. ;)

[grobie]

Wooops, sorry for the noise. The message before was collapsed in my client ;-)

…

On Wed, May 3, 2017 at 12:07 PM Viktor Petersson ***@***.***> wrote: @grobie <https://github.com/grobie> It's kinda hard to FedEx a parcel to an email address. ;) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2666 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAANaERXEI46nGZTXVKlAb9UJ3gnjIcEks5r2FHIgaJpZM4NL5_Q> .

[juliusv]

@vpetersson What a scheme to get followers! Hehe. But this is awesome, thanks! Done.

[vpetersson]

@juliusv Probably the most expensive scheme ever invented. :)

[tomwilkie]

@juliusv it could be that the embedded ewmaRates are not aligned: https://github.com/prometheus/prometheus/blob/master/storage/remote/queue_manager.go#L188.

Go seems to offer very few guarantees re: alignment - https://golang.org/ref/spec#Size_and_alignment_guarantees

@SuperQ was running the remote-write code on a Pi at KubeCon, but that may have been a Pi3 (64bit) or with an earlier version that didn't include the dynamic sharding code.

I have a Pi3 here so can give this a quick go.

[vpetersson]

@tomwilkie Yeah I've had it running in the past too. Please note that this was running ResinOS (which is basically running Docker on the Pi). Not sure if that changes anything.

[SuperQ]

I was running on a pine64 .

I have a pi3 and pi1 I can test on.

[tomwilkie]

Oh man, even the unit test reproduce the issue:

$ go test
--- FAIL: TestSampleDelivery (4.83s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x126b4]

goroutine 28 [running]:
testing.tRunner.func1(0x1076a180)
	/usr/local/go/src/testing/testing.go:622 +0x254
panic(0x28ebd0, 0x425250)
	/usr/local/go/src/runtime/panic.go:489 +0x288
sync/atomic.addUint64(0x10794074, 0x1, 0x0, 0x243150, 0x1079404c)
	/usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
github.com/prometheus/prometheus/storage/remote.(*ewmaRate).incr(0x10794074, 0x1, 0x0)
	/home/pi/go/src/github.com/prometheus/prometheus/storage/remote/ewma.go:65 +0x30
github.com/prometheus/prometheus/storage/remote.(*shards).enqueue(0x107f4150, 0x136fd900, 0x0)
	/home/pi/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:427 +0x38
github.com/prometheus/prometheus/storage/remote.(*QueueManager).Append(0x10794000, 0x107f2720, 0x64, 0x2a05f200)
	/home/pi/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:239 +0x190
github.com/prometheus/prometheus/storage/remote.TestSampleDelivery(0x1076a180)
	/home/pi/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager_test.go:106 +0x3e0
testing.tRunner(0x1076a180, 0x2dcac8)
	/usr/local/go/src/testing/testing.go:657 +0x88
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:697 +0x25c
exit status 2
FAIL	github.com/prometheus/prometheus/storage/remote	4.923s

Bring on the ARM CI...

[tomwilkie]

Okay, its embedded struct alignment:

package main

import (
        "sync/atomic"
)

type a struct {
        counter int64
}

type b struct {
        b uint32
        a a
}

func main() {
        foo := &b{}
        atomic.AddInt64(&foo.a.counter, 1)
}

$ go run test.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11204]

goroutine 1 [running]:
sync/atomic.addUint64(0x1030a0a4, 0x1, 0x0, 0x103000f0, 0x103000f0)
	/usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
main.main()
	/home/pi/test.go:18 +0x50
exit status 2

[tomwilkie]

And it seems like multiple embedded structs are tightly-packed:

package main

import (
        "sync/atomic"
)

type a struct {
        counter int64
        a uint32
}

type b struct {
        a1 a
        a2 a
        b uint32
}

func main() {
        foo := &b{}
        atomic.AddInt64(&foo.a1.counter, 1)
        atomic.AddInt64(&foo.a2.counter, 1)
}

$ go run test.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11204]

goroutine 1 [running]:
sync/atomic.addUint64(0x1036600c, 0x1, 0x0, 0x1, 0x0)
	/usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
main.main()
	/home/pi/test.go:21 +0x60
exit status 2

TBH I don't see a want of guaranteeing alignment that isn't super fragile!

[discordianfish]

Wat? Ah finally context :) I just now got the notification.. If you still have something to test, let me know. Otherwise there is also scaleway.

[tomwilkie]

thanks @discordianfish; I've got a pi at home and tested the above code. TLDR; atomicly-accessed int64s must be 64-bit aligned. Can't see a way to guarantee this in go that isn't super fragile.

Some more reading:

• golang/go#599
• golang/go#18955
• https://groups.google.com/forum/#!topic/golang-nuts/_nHK6P8_lhw
• golang/go#11891

[tomwilkie]

@vpetersson mind testing out the 266-alignment-on-arm branch? Its work for me.

[juliusv]

@tomwilkie I guess you mean this branch in your fork: https://github.com/tomwilkie/prometheus/tree/2666-alignment-on-arm

Ok, so it's an alignment problem after all. Interesting that the statement about first words in structs being aligned from the sync/atomic docs is apparently not true.

We also use atomic accesses in other places, such as for counter increments in the Prometheus client library. So it's probably pure luck that those places aren't crashing on ARM right now. We did have the same problem with x86 32-bit earlier, but solved that by having the atomically accessed variable as the first member in the struct (which doesn't seem to help this time).

[tomwilkie]

Ok, so it's an alignment problem after all. Interesting that the statement about first words in structs being aligned from the sync/atomic docs is apparently not true.

You need to be careful about how to parse the phrase, as what it means is "The first word ... in an allocated struct ... can be relied upon to be 64-bit aligned." Embedded structs are not allocated structs.

[juliusv]

@tomwilkie Man, that's a gotcha. Thanks. How about just making them pointers to independently allocated structs then (with a big fat comment that it should stay this way)? Saves us all the extra padding fields in tomwilkie@f1806c3. AFAIU there is no guarantee about those padding fields creating alignment either?

[tomwilkie]

Sure, happy with that.

[vpetersson]

I can confirm that the problem is still present in 1.6.2:

May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Loading configuration file /etc/prometheus.yml" source="main.go:251"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Loading series map and head chunks..." source="storage.go:421"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=warning msg="Persistence layer appears dirty." source="persistence.go:846"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=warning msg="Starting crash recovery. Prometheus is inoperational until complete." source="crashrecovery.go:40"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=warning msg="To avoid crash recovery in the future, shut down Prometheus with SIGTERM or a HTTP POST to /-/quit." source="crashrecovery.go:41"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Scanning files." source="crashrecovery.go:55"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="File scan complete. 228 series found." source="crashrecovery.go:83"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Checking for series without series file." source="crashrecovery.go:85"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Check for series without series file complete." source="crashrecovery.go:131"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Cleaning up archive indexes." source="crashrecovery.go:411"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Clean-up of archive indexes complete." source="crashrecovery.go:504"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Rebuilding label indexes." source="crashrecovery.go:512"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Indexing metrics in memory." source="crashrecovery.go:513"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Indexing archived metrics." source="crashrecovery.go:521"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)" source="crashrecovery.go:540"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=warning msg="Crash recovery complete." source="crashrecovery.go:153"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="579 series loaded." source="storage.go:432"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Starting target manager..." source="targetmanager.go:61"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Listening on :9090" source="web.go:259"
May 14 11:59:05 3c7a719 prometheus[255]: panic: runtime error: invalid memory address or nil pointer dereference
May 14 11:59:05 3c7a719 prometheus[255]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11580]
May 14 11:59:05 3c7a719 prometheus[255]: goroutine 209 [running]:
May 14 11:59:05 3c7a719 prometheus[255]: sync/atomic.addUint64(0x12230174, 0x1, 0x0, 0xf51a38, 0x1223014c)
May 14 11:59:05 3c7a719 prometheus[255]: /usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage/remote.(*ewmaRate).incr(0x12230174, 0x1, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/remote/ewma.go:65 +0x30
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage/remote.(*shards).enqueue(0x122a6690, 0x124d7020, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:427 +0x38
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage/remote.(*QueueManager).Append(0x12230100, 0x124d6d60, 0x12330990, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:239 +0x190
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage/remote.(*Writer).Append(0x12330990, 0x124d6d60, 0x0, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/remote/write.go:79 +0x8c
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage.Fanout.Append(0x1247a5a0, 0x2, 0x2, 0x124d6d60, 0x134abb00, 0x13568e28)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/storage.go:60 +0x58
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage.(*Fanout).Append(0x1247a7a0, 0x124d6d60, 0x1350bd14, 0x13568e10)
May 14 11:59:05 3c7a719 prometheus[255]: <autogenerated>:3 +0x64
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.(*countingAppender).Append(0x134ee5d0, 0x124d6d60, 0x1350bcfc, 0x13568e68)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/target.go:261 +0x3c
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.ruleLabelsAppender.Append(0x1cb54e0, 0x134ee5d0, 0x12488540, 0x124d6d60, 0x134ee5e0, 0x11761f0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/target.go:201 +0x138
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.(*ruleLabelsAppender).Append(0x134ee5e0, 0x124d6d60, 0x1cb6b60, 0x134ee5e0)
May 14 11:59:05 3c7a719 prometheus[255]: <autogenerated>:35 +0x64
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0x121f2900, 0x12267500, 0xb, 0xc8, 0xe, 0x33e0cd1, 0x1d5e298)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:519 +0x224
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0x121f2900, 0xfc23ac00, 0x6, 0x540be400, 0x2, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:429 +0x4e0
May 14 11:59:05 3c7a719 systemd[1]: prometheus.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 14 11:59:05 3c7a719 systemd[1]: Unit prometheus.service entered failed state.
May 14 11:59:05 3c7a719 prometheus[255]: created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:258 +0x330
May 14 11:59:05 3c7a719 systemd[1]: scanbt-worker.service holdoff time over, scheduling restart.

[juliusv]

@tomwilkie Could you check the latest release version (1.6.2) on your rpi 3 again? I wonder if there is a difference between your rpi and @vpetersson's, but at least they both seem to be an rpi 3. I doubt that the exact OS should make a difference, as it's an architectural memory access issue that should be independent of the OS?

[tomwilkie]

I think the issue is that the fix[1] didn't make it onto the 1.6 branch[2]. Since its been merged already, should I make a cherry pick for 1.6.3?

[1] 3141a6b
[2] https://github.com/prometheus/prometheus/commits/release-1.6

[grobie]

I think there were plans to release 1.7 soon, are we sure we'll have a 1.6.3 release? We need to get better on marking changes to be included in the next patch release in general.

[brian-brazil]

This fix should be out now.

[vpetersson]

I can confirm that this is now working. Thanks, guys!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.