Skip to content

promtail: journal scrape not working for aarch64 / rpi4 #3522

New issue
New issue
Closed
Closed
promtail: journal scrape not working for aarch64 / rpi4#3522

Description

@AngellusMortis

AngellusMortis
openedon Mar 21, 2021

Describe the bug
I am unable to enable the journal scrape on an aarch64 install of Linux on a RPI (archlinuxarm).

Cross-posted to archlinuxarm forum as well since I am not sure where the fix would exist at:
https://archlinuxarm.org/forum/viewtopic.php?f=15&t=15235

To Reproduce
Steps to reproduce the behavior:

  1. Install archlinuxarm (aarch64) on RPI4: https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-4
  2. Install promtail: pacman -S promtail
  3. Add journal scrape config
  4. Started Promtail (v2.2.0 / a27c799)

Expected behavior
Promtail start as expected and sends logs for journald.

Environment:

  • RPI4 + archlinuxarm (aarch64)

Screenshots, Promtail config, or terminal output

Mar 20 15:29:16 alarm promtail[243566]: level=info ts=2021-03-20T19:29:16.163114189Z caller=main.go:110 msg="Starting Promtail" version="(version=2.1.0, branch=, revision=)"
Mar 20 15:29:17 alarm promtail[243566]: unexpected fault address 0x6c62767862ea9f
Mar 20 15:29:17 alarm promtail[243566]: fatal error: fault
Mar 20 15:29:17 alarm promtail[243566]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x6c62767862ea9f pc=0xaaaaaec746d0]
Mar 20 15:29:17 alarm promtail[243566]: goroutine 225 [running]:
Mar 20 15:29:17 alarm promtail[243566]: runtime.throw(0xaaaaaffa7175, 0x5)
Mar 20 15:29:17 alarm promtail[243566]:         runtime/panic.go:1117 +0x54 fp=0x40008f38d0 sp=0x40008f38a0 pc=0xaaaaae848b24
Mar 20 15:29:17 alarm promtail[243566]: runtime.sigpanic()
Mar 20 15:29:17 alarm promtail[243566]:         runtime/signal_unix.go:741 +0x230 fp=0x40008f3910 sp=0x40008f38d0 pc=0xaaaaae8603a0
Mar 20 15:29:17 alarm promtail[243566]: github.com/golang/snappy.encodeBlock(0x4000f80003, 0x6b614, 0x6b614, 0x4000db8000, 0x10000, 0x5c08b, 0x400089c138)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/golang/snappy@v0.0.2/encode_arm64.s:666 +0x360 fp=0x40008fb9b0 sp=0x40008f3920 pc=0xaaaaaec746d0
Mar 20 15:29:17 alarm promtail[243566]: github.com/golang/snappy.Encode(0x4000f80000, 0x6b617, 0x6b617, 0x4000dc8000, 0x4c08b, 0x4c08b, 0x0, 0xaaaaaea2075c, 0x40009a48f0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/golang/snappy@v0.0.2/encode.go:39 +0x17c fp=0x40008fba00 sp=0x40008fb9b0 pc=0xaaaaaec7336c
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*batch).encode(0x400095e3c0, 0x1, 0x10, 0x4c, 0x4000af6730, 0x40009a48f0, 0x400007dbd8)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/client/batch.go:79 +0x94 fp=0x40008fba60 sp=0x40008fba00 pc=0xaaaaaefaecc4
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*client).sendBatch(0x400044c900, 0x0, 0x0, 0x400095e3c0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/client/client.go:241 +0x30 fp=0x40008fbc90 sp=0x40008fba60 pc=0xaaaaaefafe40
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*client).run(0x400044c900)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/client/client.go:233 +0x46c fp=0x40008fbfd0 sp=0x40008fbc90 pc=0xaaaaaefafdac
Mar 20 15:29:17 alarm promtail[243566]: runtime.goexit()
Mar 20 15:29:17 alarm promtail[243566]:         runtime/asm_arm64.s:1130 +0x4 fp=0x40008fbfd0 sp=0x40008fbfd0 pc=0xaaaaae87fda4
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/client.New
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/client/client.go:170 +0x320
Mar 20 15:29:17 alarm promtail[243566]: goroutine 1 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/server.(*noopServer).Run(0x40009bc2b8, 0x6, 0x400098c3c0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/server/server.go:260 +0x8c
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail.(*Promtail).Run(0x40003569b0, 0x4000998b00, 0x4)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/promtail.go:98 +0xe8
Mar 20 15:29:17 alarm promtail[243566]: main.main()
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/cmd/promtail/main.go:113 +0x48c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 19 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog.(*loggingT).flushDaemon(0xaaaab1a35920)
Mar 20 15:29:17 alarm promtail[243566]:         k8s.io/klog@v1.0.0/klog.go:1010 +0x70
Mar 20 15:29:17 alarm promtail[243566]: created by k8s.io/klog.init.0
Mar 20 15:29:17 alarm promtail[243566]:         k8s.io/klog@v1.0.0/klog.go:411 +0x11c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 69 [select]:
Mar 20 15:29:17 alarm promtail[243566]: go.opencensus.io/stats/view.(*worker).start(0x400049e100)
Mar 20 15:29:17 alarm promtail[243566]:         go.opencensus.io@v0.22.4/stats/view/worker.go:276 +0x98
Mar 20 15:29:17 alarm promtail[243566]: created by go.opencensus.io/stats/view.init.0
Mar 20 15:29:17 alarm promtail[243566]:         go.opencensus.io@v0.22.4/stats/view/worker.go:34 +0x70
Mar 20 15:29:17 alarm promtail[243566]: goroutine 70 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xaaaab1a35a00)
Mar 20 15:29:17 alarm promtail[243566]:         k8s.io/klog/v2@v2.3.0/klog.go:1131 +0x70
Mar 20 15:29:17 alarm promtail[243566]: created by k8s.io/klog/v2.init.0
Mar 20 15:29:17 alarm promtail[243566]:         k8s.io/klog/v2@v2.3.0/klog.go:416 +0x11c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 148 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x4000111728)
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 187 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x400000e840)
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 190 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x400000eab0)
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 258 [select]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/positions.(*positions).run(0x4000356a00)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/positions/positions.go:153 +0xbc
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/positions.New
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/positions/positions.go:95 +0x144
Mar 20 15:29:17 alarm promtail[243566]: goroutine 263 [select]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow(0x40009bc288, 0x4000984300, 0xaaaab0b1aaf8, 0xaaaab1a64c00, 0x0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:238 +0x130
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/targets/journal.journalTargetWithReader.func1(0x40001ae400, 0x4000984300)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/targets/journal/journaltarget.go:184 +0x54
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/targets/journal.journalTargetWithReader
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/targets/journal/journaltarget.go:182 +0x310
Mar 20 15:29:17 alarm promtail[243566]: goroutine 275 [syscall]:
Mar 20 15:29:17 alarm promtail[243566]: os/signal.signal_recv(0x0)
Mar 20 15:29:17 alarm promtail[243566]:         runtime/sigqueue.go:168 +0xcc
Mar 20 15:29:17 alarm promtail[243566]: os/signal.loop()
Mar 20 15:29:17 alarm promtail[243566]:         os/signal/signal_unix.go:23 +0x20
Mar 20 15:29:17 alarm promtail[243566]: created by os/signal.Notify.func1.1
Mar 20 15:29:17 alarm promtail[243566]:         os/signal/signal.go:151 +0x44
Mar 20 15:29:17 alarm promtail[243566]: goroutine 203 [syscall]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal._Cfunc_my_sd_journal_wait(0xffff5af0eb80, 0xaaaadb8b6200, 0x186a0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]:         _cgo_gotypes.go:554 +0x44
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*Journal).Wait.func1(0xffff5af0eb80, 0x40001b47d0, 0x186a0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/journal.go:1014 +0x80
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*Journal).Wait(0x40001b47d0, 0x5f5e100, 0x0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/journal.go:1014 +0xa8
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow.func1(0x40009bc288, 0x4000982150, 0x40001b4830)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:233 +0x38
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:232 +0xe8
Mar 20 15:29:17 alarm systemd[1]: promtail.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions