Traefik shuts down when SystemD watchdog run #1353

jskarpe · 2017-03-28T10:23:26Z

What version of Traefik are you using (`traefik version`)?

1.2.1

What did you do?

Install service file from: https://github.com/containous/traefik/blob/master/contrib/systemd/traefik.service

What did you expect to see?

Running service

What did you see instead?

Crashing service every 2s, with SystemD starting it again. Adjusting watchdog timings correlate with time between crashes

The text was updated successfully, but these errors were encountered:

timoreimann · 2017-03-28T13:33:28Z

@yuav can you paste log outputs please?

q7r · 2017-04-19T20:59:37Z

@timoreimann I have the very same problem with v1.2.3. Which log do you need?
Here is systemctl status:

[root@traefik system]# systemctl status traefik.service
● traefik.service - Traefik
   Loaded: loaded (/etc/systemd/system/traefik.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-04-19 23:40:28 EEST; 418ms ago
 Main PID: 9890 (traefik)
   CGroup: /system.slice/traefik.service
           └─9890 /usr/bin/traefik --configFile=/etc/traefik.toml

Apr 19 23:40:28 traefik systemd[1]: traefik.service holdoff time over, scheduling restart.
Apr 19 23:40:28 traefik systemd[1]: Starting Traefik...
Apr 19 23:40:28 traefik systemd[1]: Started Traefik.
Apr 19 23:40:28 traefik traefik[9890]: 2017/04/19 23:40:28 structs.go:21: Connected to 10.10.10.10:2181
Apr 19 23:40:28 traefik traefik[9890]: 2017/04/19 23:40:28 structs.go:21: Authenticated: id=97820254626581384, timeout=40000
[root@traefik system]# systemctl status traefik.service
● traefik.service - Traefik
   Loaded: loaded (/etc/systemd/system/traefik.service; enabled; vendor preset: disabled)
   Active: deactivating (stop-sigabrt) (Result: watchdog) since Wed 2017-04-19 23:40:29 EEST; 3ms ago
 Main PID: 9890 (traefik)
   CGroup: /system.slice/traefik.service
           └─9890 /usr/bin/traefik --configFile=/etc/traefik.toml

Apr 19 23:40:28 traefik systemd[1]: traefik.service holdoff time over, scheduling restart.
Apr 19 23:40:28 traefik systemd[1]: Starting Traefik...
Apr 19 23:40:28 traefik systemd[1]: Started Traefik.
Apr 19 23:40:28 traefik traefik[9890]: 2017/04/19 23:40:28 structs.go:21: Connected to 10.10.10.10:2181
Apr 19 23:40:28 traefik traefik[9890]: 2017/04/19 23:40:28 structs.go:21: Authenticated: id=97820254626581384, timeout=40000
Apr 19 23:40:29 traefik systemd[1]: traefik.service watchdog timeout (limit 1s)!
[root@traefik system]# systemctl status traefik.service
● traefik.service - Traefik
   Loaded: loaded (/etc/systemd/system/traefik.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2017-04-19 23:40:31 EEST; 5ms ago
  Process: 9899 ExecStart=/usr/bin/traefik --configFile=/etc/traefik.toml (code=exited, status=2)
 Main PID: 9899 (code=exited, status=2)

Apr 19 23:40:31 traefik traefik[9899]: rip    0x477223
Apr 19 23:40:31 traefik traefik[9899]: rflags 0x246
Apr 19 23:40:31 traefik traefik[9899]: cs     0x33
Apr 19 23:40:31 traefik traefik[9899]: fs     0x0
Apr 19 23:40:31 traefik traefik[9899]: gs     0x0
Apr 19 23:40:31 traefik systemd[1]: traefik.service holdoff time over, scheduling restart.
Apr 19 23:40:31 traefik systemd[1]: start request repeated too quickly for traefik.service
Apr 19 23:40:31 traefik systemd[1]: Failed to start Traefik.
Apr 19 23:40:31 traefik systemd[1]: Unit traefik.service entered failed state.
Apr 19 23:40:31 traefik systemd[1]: traefik.service failed.

It works without WatchdogSec=1s line in systemd file.

emilevauge · 2017-04-19T21:23:39Z

@guilhem any idea on what's going on here ? 🤔

philiplb · 2017-04-26T10:01:34Z

It seems to work for me if I set the WatchdogSec to 10s.

Yggdrasil · 2017-04-29T15:55:36Z

Same issue for me with 1.2.3. @philiplb's suggestion doesn't work for me. Only disabling the Watchdog timer completely (i.e. removing the line WatchdogSec=1s) keeps Traefik running.

guilhem · 2017-04-30T00:25:44Z

I will look at it this week-end

Commit coreos/go-systemd@0c088e introduce cleaning environment. First usage of sdnotify (for type=notify) was clearing NOTIFY_SOCKET environment variable. sdnotify in watchdog was unable to ping back. Fix traefik#1353

nsteinmetz · 2018-02-13T09:55:04Z

Hi there,

Just had the issue with 1.5.2 whereas I didn't have it before. New thing in my context is that I use ubuntu 16.04 LTS at my client whereas I used to used Debian on servers or HypriotOS on arm boards.

I add to remove the Watchdog directive in the systemd service file to make it work again.

cat /etc/systemd/system/traefik.service
[Unit]
Description=Traefik

[Service]
Type=notify
ExecStart=/usr/bin/traefik --configFile=/etc/traefik/conf/traefik.toml
Restart=always

[Install]
WantedBy=multi-user.target

guilhem · 2018-02-13T10:07:49Z

@nsteinmetz do you have a log (journald) to provides?

nsteinmetz · 2018-02-13T10:11:36Z

Oh yes, I posted them on slack:

Feb 13 10:44:34  systemd[1]: traefik.service: Service hold-off time over, scheduling restart.
Feb 13 10:44:34  systemd[1]: Stopped Traefik.
Feb 13 10:44:34  systemd[1]: Starting Traefik...
Feb 13 10:44:35  systemd[1]: Started Traefik.
Feb 13 10:44:38  traefik[10341]: goroutine 71 [running]:
Feb 13 10:44:38  traefik[10341]: runtime/debug.Stack(0x74f40f, 0xc42000e370, 0x2128009)
Feb 13 10:44:38  traefik[10341]:         /usr/local/go/src/runtime/debug/stack.go:24 +0xa7
Feb 13 10:44:38  traefik[10341]: runtime/debug.PrintStack()
Feb 13 10:44:38  traefik[10341]:         /usr/local/go/src/runtime/debug/stack.go:16 +0x22
Feb 13 10:44:38  traefik[10341]: github.com/containous/traefik/safe.defaultRecoverGoroutine(0x1dd7b00, 0x32e5af0)
Feb 13 10:44:38  traefik[10341]:         /go/src/github.com/containous/traefik/safe/routine.go:148 +0x7d
Feb 13 10:44:38  traefik[10341]: github.com/containous/traefik/safe.GoWithRecover.func1.1(0x21aa728)
Feb 13 10:44:38  traefik[10341]:         /go/src/github.com/containous/traefik/safe/routine.go:139 +0x57
Feb 13 10:44:38  traefik[10341]: panic(0x1dd7b00, 0x32e5af0)
Feb 13 10:44:38  traefik[10341]:         /usr/local/go/src/runtime/panic.go:491 +0x283
Feb 13 10:44:38  traefik[10341]: main.healthCheck(0xc4205316a0, 0x0, 0x100, 0x0, 0x0, 0xc4202ee5c0, 0x0, 0x0, 0xc4202ee600, 0xc42035ad28, ...)
Feb 13 10:44:38  traefik[10341]:         /go/src/github.com/containous/traefik/cmd/traefik/healthcheck.go:53 +0x37
Feb 13 10:44:38  traefik[10341]: main.run.func1()
Feb 13 10:44:38  traefik[10341]:         /go/src/github.com/containous/traefik/cmd/traefik/traefik.go:180 +0xfd
Feb 13 10:44:38  traefik[10341]: github.com/containous/traefik/safe.GoWithRecover.func1(0x21aa728, 0xc4205a3360)
Feb 13 10:44:38  traefik[10341]:         /go/src/github.com/containous/traefik/safe/routine.go:142 +0x4d
Feb 13 10:44:38  traefik[10341]: created by github.com/containous/traefik/safe.GoWithRecover
Feb 13 10:44:38  traefik[10341]:         /go/src/github.com/containous/traefik/safe/routine.go:136 +0x49
...skipping...
Feb 13 10:44:34  traefik[10325]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Feb 13 10:44:34  traefik[10325]: net/http.(*conn).serve(0xc420260d20, 0x30b9f40, 0xc42065e300)
Feb 13 10:44:34  traefik[10325]:         /usr/local/go/src/net/http/server.go:1826 +0x88f
Feb 13 10:44:34  traefik[10325]: created by net/http.(*Server).Serve
Feb 13 10:44:34  traefik[10325]:         /usr/local/go/src/net/http/server.go:2720 +0x288
Feb 13 10:44:34  traefik[10325]: rax    0xca
Feb 13 10:44:34  traefik[10325]: rbx    0x32ff8c0
Feb 13 10:44:34  traefik[10325]: rcx    0x45eb23
Feb 13 10:44:34  traefik[10325]: rdx    0x0
Feb 13 10:44:34  traefik[10325]: rdi    0x32ff9f8
Feb 13 10:44:34  traefik[10325]: rsi    0x0
Feb 13 10:44:34  traefik[10325]: rbp    0x7ffe5bb89010
Feb 13 10:44:34  traefik[10325]: rsp    0x7ffe5bb88fc8
Feb 13 10:44:34  traefik[10325]: r8     0x0
Feb 13 10:44:34  traefik[10325]: r9     0x0
Feb 13 10:44:34  traefik[10325]: r10    0x0
Feb 13 10:44:34  traefik[10325]: r11    0x286
Feb 13 10:44:34  traefik[10325]: r12    0x0
Feb 13 10:44:34  traefik[10325]: r13    0x0
Feb 13 10:44:34  traefik[10325]: r14    0x45c360
Feb 13 10:44:34  traefik[10325]: r15    0x0
Feb 13 10:44:34  traefik[10325]: rip    0x45eb21
Feb 13 10:44:34  traefik[10325]: rflags 0x286
Feb 13 10:44:34  traefik[10325]: cs     0x33
Feb 13 10:44:34  traefik[10325]: fs     0x0
Feb 13 10:44:34  traefik[10325]: gs     0x0

guilhem · 2018-02-13T10:16:04Z

@nsteinmetz watchdog is sharing code with traefik healtcheck command.
Can you try to run it and see what happen?
It seems to have a problem with it.

ldez · 2018-02-13T10:18:31Z

@nsteinmetz could you open a new issue.

nsteinmetz · 2018-02-13T10:19:10Z

@ldez ok I will.

nsteinmetz · 2018-02-13T10:25:09Z

Following issue now on : #2851

ldez added the contributor/need-more-information label Apr 12, 2017

ldez added investigation-needed and removed contributor/need-more-information labels Apr 26, 2017

guilhem mentioned this issue May 2, 2017

Fix systemd watchdog feature #1525

Merged

ldez closed this as completed in #1525 May 3, 2017

ldez removed the investigation-needed label May 12, 2017

ldez added this to the 1.3 milestone Feb 13, 2018

ldez added the kind/bug/confirmed a confirmed bug (reproducible). label Feb 13, 2018

nsteinmetz mentioned this issue Feb 13, 2018

Traefik 1.5.2 & systemd watchdog issue #2851

Closed

traefik locked and limited conversation to collaborators Sep 1, 2019

traefiker added the status/5-frozen-due-to-age label Sep 1, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Traefik shuts down when SystemD watchdog run #1353

Traefik shuts down when SystemD watchdog run #1353

jskarpe commented Mar 28, 2017

timoreimann commented Mar 28, 2017

q7r commented Apr 19, 2017 •

edited by ldez

emilevauge commented Apr 19, 2017

philiplb commented Apr 26, 2017

Yggdrasil commented Apr 29, 2017

guilhem commented Apr 30, 2017

nsteinmetz commented Feb 13, 2018 •

edited

guilhem commented Feb 13, 2018

nsteinmetz commented Feb 13, 2018

guilhem commented Feb 13, 2018

ldez commented Feb 13, 2018

nsteinmetz commented Feb 13, 2018

nsteinmetz commented Feb 13, 2018

Traefik shuts down when SystemD watchdog run #1353

Traefik shuts down when SystemD watchdog run #1353

Comments

jskarpe commented Mar 28, 2017

What version of Traefik are you using (traefik version)?

What did you do?

What did you expect to see?

What did you see instead?

timoreimann commented Mar 28, 2017

q7r commented Apr 19, 2017 • edited by ldez

emilevauge commented Apr 19, 2017

philiplb commented Apr 26, 2017

Yggdrasil commented Apr 29, 2017

guilhem commented Apr 30, 2017

nsteinmetz commented Feb 13, 2018 • edited

guilhem commented Feb 13, 2018

nsteinmetz commented Feb 13, 2018

guilhem commented Feb 13, 2018

ldez commented Feb 13, 2018

nsteinmetz commented Feb 13, 2018

nsteinmetz commented Feb 13, 2018

What version of Traefik are you using (`traefik version`)?

q7r commented Apr 19, 2017 •

edited by ldez

nsteinmetz commented Feb 13, 2018 •

edited