Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kernel Panic on Ubuntu 18.04.03 with kernel 4.15.0-60 #2904

Closed
FingerlessGlov3s opened this issue Sep 3, 2019 · 71 comments

Comments

@FingerlessGlov3s
Copy link

commented Sep 3, 2019

Prior to placing the issue, please check following: (fill out each checkbox with a X once done)

  • I understand that not following below instructions might result in immediate closing and deletion of my issue.
  • I have understood that answers are voluntary and community-driven, and not commercial support.
  • I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description of the bug: What kind of issue have you exactly come across?
When using the latest kernel on Ubuntu 18.04 "kernel 4.15.0-60" soon as you start the mailcow docker enviroment which 10-30 seconds the machine kernels panics.
I've been sitting in the IRC last 24 hours and someone tried to setup mailcow on Vultr and Hetzner and got kernel panics, he left before I could ask which kernel but I bet they would of been fully up to date images,

IRC message from the guy before he left, he also joined after I message the room about mine kernel panicing asking if anyone else has had the issue.
": any current known issues around spinning up mailcow instances? have had two fail consecutively today. ubuntu 18.04, fresh docker/compose install - everything seems to go ok, then lose connectivity. kvm shows kernel panic. have tried two vps's - vultr and hetzner, plenty of ram, vcpus, etc. have installed on vultr before. i'm probably being dumb somewhere - but wanted to check in/ask before i beat my head into it for a while "

At the time of writing this Roger1 on IRC is having the same issue on AWS, who use their own kernel but I bet probably bases on 4.15.0-60, theirs is 4.15.0-1047-aws

Roger1 started up each service by hand and it crashed after starting mailcow/dockerapi

I will post more information later, and Roger1 said he would post logs too.

Further information (where applicable):

Question Answer
My operating system Ubuntu 18.04.3
Is Apparmor, SELinux or similar active? shall check later
Virtualization technlogy (KVM, VMware, Xen, etc) I believe Scaleway use VMware
Server/VM specifications (Memory, CPU Cores) 3 cores, 4GB Ram
Docker Version (docker version) 19.03
Docker-Compose Version (docker-compose version) v1.24.1
Reverse proxy (custom solution) NOPE
@RogerSik

This comment has been minimized.

Copy link

commented Sep 3, 2019

Switch the linux kernel from 4.15.0-1047-aws x86_64 to 5.0.0-1014-aws solved my issue. It was a mailcow instances which was longer and in use and the problem for the first time today began.

When starting all docker container from mailcow no TCP/HTTP/Ping Communition was more possible. I assumed the ip blocker was going in rage but the crash/black happens when I started all container manually and than started "mailcow/dockerapi:1.32"

@patschi

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

I'd more likely guess that this is an issue from Docker, and not from mailcow. Mailcow is "just" a collection out of various containers to build a mailserver - it doesn't have enough kernel privileges to let the host kernel panic.

@c0by85

This comment has been minimized.

Copy link

commented Sep 3, 2019

Got the same Problem, had to go one kernel back.

Problem kernel was Linux mail 4.15.0-60

currently working kernel:

uname -a Linux mail 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

Got the same Problem, had to go one kernel back.

Problem kernel was Linux mail 4.15.0-60

uname -a Linux mail 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic

There's something with this this latest kernel.

Anyone running any other docker contains with this kernel on different host?
Web servers, databases or something?

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

Checking it in a bit.

What's the panic message?

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

Is AppArmor active?

@RogerSik

This comment has been minimized.

Copy link

commented Sep 3, 2019

AppArmor not active. And no else container running.

@Shineson1001

This comment has been minimized.

Copy link

commented Sep 3, 2019

the last syslog entries:

`root@smtprelay02:~# tail -f /var/log/syslog
Sep 3 12:23:26 smtprelay02 kernel: [ 16.850172] eth0: renamed from vethe4dfad1
Sep 3 12:23:27 smtprelay02 systemd-networkd[1000]: br-mailcow: Gained IPv6LL
Sep 3 12:23:27 smtprelay02 kernel: [ 17.083272] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:23:27 smtprelay02 systemd-networkd[1000]: vethfb645ce: Gained carrier
Sep 3 12:23:27 smtprelay02 kernel: [ 17.089237] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:23:27 smtprelay02 kernel: [ 17.089383] br-mailcow: port 16(vethce06c3c) entered disabled state
Sep 3 12:23:27 smtprelay02 kernel: [ 17.089494] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 3 12:23:27 smtprelay02 kernel: [ 17.089546] IPv6: ADDRCONF(NETDEV_CHANGE): vethfb645ce: link becomes ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.089575] br-mailcow: port 3(vethfb645ce) entered blocking state

Sep 3 12:23:27 smtprelay02 kernel: [ 17.089576] br-mailcow: port 3(vethfb645ce) entered forwarding state

Sep 3 12:23:27 smtprelay02 kernel: [ 17.318859] eth0: renamed from veth2c9e04f

Sep 3 12:23:27 smtprelay02 systemd-networkd[1000]: veth2cf0374: Gained carrier

Sep 3 12:23:27 smtprelay02 kernel: [ 17.396235] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398722] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398730] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398774] IPv6: ADDRCONF(NETDEV_CHANGE): veth2cf0374: link becomes ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398872] br-mailcow: port 5(veth2cf0374) entered blocking state

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398875] br-mailcow: port 5(veth2cf0374) entered forwarding state

Sep 3 12:23:27 smtprelay02 containerd[1335]: time="2019-09-03T12:23:27.836730866+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/8e927e58c109d4ae3b255521b792bb213e8a089db04149765bb8d0c2fef41c89/shim.sock" debug=false pid=4181

Sep 3 12:23:27 smtprelay02 systemd-networkd[1000]: vethd87bee6: Gained IPv6LL

Sep 3 12:23:27 smtprelay02 containerd[1335]: time="2019-09-03T12:23:27.920245956+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/9ea6823344f4d8cda454f2281b33cace2d471cb362577293eafb7dd44bd502fa/shim.sock" debug=false pid=4225

Sep 3 12:23:27 smtprelay02 containerd[1335]: time="2019-09-03T12:23:27.985258651+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/1558911a3f198704efdf17c9260b8ce3ef4dc1898edc42cca992c6f5b6738711/shim.sock" debug=false pid=4253

Sep 3 12:23:27 smtprelay02 containerd[1335]: time="2019-09-03T12:23:27.987555563+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/261a1fa502c54e630d9d1831e6afe42653228e950f4daf97959025e200688990/shim.sock" debug=false pid=4257

Sep 3 12:23:28 smtprelay02 kernel: [ 18.014537] eth0: renamed from veth8c2f06e

Sep 3 12:23:28 smtprelay02 kernel: [ 18.151543] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:28 smtprelay02 systemd-networkd[1000]: veth3cdfa48: Gained carrier

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156377] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156389] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156463] IPv6: ADDRCONF(NETDEV_CHANGE): veth3cdfa48: link becomes ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156538] br-mailcow: port 4(veth3cdfa48) entered blocking state

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156553] br-mailcow: port 4(veth3cdfa48) entered forwarding state

Sep 3 12:23:28 smtprelay02 kernel: [ 18.501021] eth0: renamed from veth17b6626

Sep 3 12:23:28 smtprelay02 kernel: [ 18.574858] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:28 smtprelay02 systemd-networkd[1000]: veth9eddcff: Gained carrier

Sep 3 12:23:28 smtprelay02 kernel: [ 18.581918] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.581930] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.582221] IPv6: ADDRCONF(NETDEV_CHANGE): veth9eddcff: link becomes ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.582416] br-mailcow: port 8(veth9eddcff) entered blocking state

Sep 3 12:23:28 smtprelay02 kernel: [ 18.582419] br-mailcow: port 8(veth9eddcff) entered forwarding state

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: vethfb645ce: Gained IPv6LL

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: veth2cf0374: Gained IPv6LL

Sep 3 12:23:29 smtprelay02 kernel: [ 19.070157] eth0: renamed from vetha9b3771

Sep 3 12:23:29 smtprelay02 kernel: [ 19.134323] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: vethe7fa3e7: Gained carrier

Sep 3 12:23:29 smtprelay02 kernel: [ 19.136915] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.136927] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.136988] IPv6: ADDRCONF(NETDEV_CHANGE): vethe7fa3e7: link becomes ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.137027] br-mailcow: port 2(vethe7fa3e7) entered blocking state

Sep 3 12:23:29 smtprelay02 kernel: [ 19.137031] br-mailcow: port 2(vethe7fa3e7) entered forwarding state

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: veth3cdfa48: Gained IPv6LL

Sep 3 12:23:29 smtprelay02 kernel: [ 19.414803] eth0: renamed from veth7fb8648

Sep 3 12:23:29 smtprelay02 kernel: [ 19.483509] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: vetha939816: Gained carrier

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486320] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486339] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486380] IPv6: ADDRCONF(NETDEV_CHANGE): vetha939816: link becomes ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486489] br-mailcow: port 10(vetha939816) entered blocking state

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486491] br-mailcow: port 10(vetha939816) entered forwarding state

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: vethe7fa3e7: Gained IPv6LL

Sep 3 12:23:30 smtprelay02 kernel: [ 20.274464] eth0: renamed from veth0a9fbfc

Sep 3 12:23:30 smtprelay02 kernel: [ 20.342475] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344759] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344779] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344825] IPv6: ADDRCONF(NETDEV_CHANGE): vethf7a040a: link becomes ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344933] br-mailcow: port 7(vethf7a040a) entered blocking state

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344937] br-mailcow: port 7(vethf7a040a) entered forwarding state

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: vethf7a040a: Gained carrier

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: veth9eddcff: Gained IPv6LL

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: vetha939816: Gained IPv6LL

Sep 3 12:23:30 smtprelay02 kernel: [ 20.683061] eth0: renamed from veth7950e6f

Sep 3 12:23:30 smtprelay02 kernel: [ 20.759578] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: veth70a186e: Gained carrier

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762101] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762192] IPv6: ADDRCONF(NETDEV_CHANGE): veth70a186e: link becomes ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762353] br-mailcow: port 14(veth70a186e) entered blocking state

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762354] br-mailcow: port 14(veth70a186e) entered forwarding state

Sep 3 12:23:31 smtprelay02 kernel: [ 21.784033] eth0: renamed from vethfeb6cd8

Sep 3 12:23:31 smtprelay02 kernel: [ 21.847287] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849767] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849787] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849828] IPv6: ADDRCONF(NETDEV_CHANGE): vethaf5af12: link becomes ready

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849913] br-mailcow: port 13(vethaf5af12) entered blocking state

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849915] br-mailcow: port 13(vethaf5af12) entered forwarding state

Sep 3 12:23:31 smtprelay02 systemd-networkd[1000]: vethaf5af12: Gained carrier

Sep 3 12:23:32 smtprelay02 systemd-networkd[1000]: vethf7a040a: Gained IPv6LL

Sep 3 12:23:32 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:32+02:00" level=warning msg="xtables contention detected while running [--wait -t nat -I POSTROUTING -d 127.0.0.11 -j DOCKER_POSTROUTING]: Waited for 4.01 seconds and received """

Sep 3 12:23:32 smtprelay02 kernel: [ 22.351044] eth0: renamed from vethd203b78

Sep 3 12:23:32 smtprelay02 systemd-networkd[1000]: veth70a186e: Gained IPv6LL

Sep 3 12:23:32 smtprelay02 kernel: [ 22.426306] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:32 smtprelay02 systemd-networkd[1000]: veth2958443: Gained carrier

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429602] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429623] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429695] IPv6: ADDRCONF(NETDEV_CHANGE): veth2958443: link becomes ready

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429771] br-mailcow: port 6(veth2958443) entered blocking state

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429777] br-mailcow: port 6(veth2958443) entered forwarding state

Sep 3 12:23:33 smtprelay02 systemd-networkd[1000]: vethaf5af12: Gained IPv6LL

Sep 3 12:23:33 smtprelay02 kernel: [ 23.540965] eth0: renamed from vethbfda38c

Sep 3 12:23:33 smtprelay02 kernel: [ 23.626221] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:33 smtprelay02 kernel: [ 23.634258] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:33 smtprelay02 systemd-networkd[1000]: veth1b3a578: Gained carrier

Sep 3 12:23:33 smtprelay02 kernel: [ 23.639386] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:33 smtprelay02 kernel: [ 23.639522] IPv6: ADDRCONF(NETDEV_CHANGE): veth1b3a578: link becomes ready

Sep 3 12:23:33 smtprelay02 kernel: [ 23.639603] br-mailcow: port 9(veth1b3a578) entered blocking state

Sep 3 12:23:33 smtprelay02 kernel: [ 23.639608] br-mailcow: port 9(veth1b3a578) entered forwarding state

Sep 3 12:23:34 smtprelay02 systemd-networkd[1000]: veth2958443: Gained IPv6LL

Sep 3 12:23:34 smtprelay02 kernel: [ 24.603020] eth0: renamed from veth5fb449d

Sep 3 12:23:34 smtprelay02 kernel: [ 24.687067] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:34 smtprelay02 kernel: [ 24.690037] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:34 smtprelay02 kernel: [ 24.691929] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:34 smtprelay02 kernel: [ 24.692031] IPv6: ADDRCONF(NETDEV_CHANGE): veth58f4bcc: link becomes ready

Sep 3 12:23:34 smtprelay02 kernel: [ 24.692104] br-mailcow: port 12(veth58f4bcc) entered blocking state

Sep 3 12:23:34 smtprelay02 kernel: [ 24.692108] br-mailcow: port 12(veth58f4bcc) entered forwarding state

Sep 3 12:23:34 smtprelay02 systemd-networkd[1000]: veth58f4bcc: Gained carrier

Sep 3 12:23:34 smtprelay02 systemd-networkd[1000]: veth1b3a578: Gained IPv6LL

Sep 3 12:23:35 smtprelay02 kernel: [ 25.645392] eth0: renamed from veth964a240

Sep 3 12:23:35 smtprelay02 systemd-networkd[1000]: veth58f4bcc: Gained IPv6LL

Sep 3 12:23:35 smtprelay02 kernel: [ 25.739583] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:35 smtprelay02 kernel: [ 25.742687] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:23:35 smtprelay02 kernel: [ 25.742697] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 3 12:23:35 smtprelay02 kernel: [ 25.742748] IPv6: ADDRCONF(NETDEV_CHANGE): vethdd28ca3: link becomes ready
Sep 3 12:23:35 smtprelay02 kernel: [ 25.742794] br-mailcow: port 11(vethdd28ca3) entered blocking state
Sep 3 12:23:35 smtprelay02 kernel: [ 25.742796] br-mailcow: port 11(vethdd28ca3) entered forwarding state
Sep 3 12:23:35 smtprelay02 systemd-networkd[1000]: vethdd28ca3: Gained carrier
Sep 3 12:23:36 smtprelay02 kernel: [ 26.491201] eth0: renamed from veth073401d
Sep 3 12:23:36 smtprelay02 kernel: [ 26.592834] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:23:36 smtprelay02 kernel: [ 26.598836] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:23:36 smtprelay02 kernel: [ 26.598846] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 3 12:23:36 smtprelay02 kernel: [ 26.598891] IPv6: ADDRCONF(NETDEV_CHANGE): vethe06f939: link becomes ready
Sep 3 12:23:36 smtprelay02 kernel: [ 26.598933] br-mailcow: port 15(vethe06f939) entered blocking state
Sep 3 12:23:36 smtprelay02 kernel: [ 26.598934] br-mailcow: port 15(vethe06f939) entered forwarding state
Sep 3 12:23:36 smtprelay02 systemd-networkd[1000]: vethe06f939: Gained carrier
Sep 3 12:23:37 smtprelay02 systemd-networkd[1000]: vethdd28ca3: Gained IPv6LL
Sep 3 12:23:37 smtprelay02 kernel: [ 27.668102] eth0: renamed from veth04ee854
Sep 3 12:23:37 smtprelay02 kernel: [ 27.770486] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:23:37 smtprelay02 systemd-networkd[1000]: vethce06c3c: Gained carrier
Sep 3 12:23:37 smtprelay02 kernel: [ 27.774643] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:23:37 smtprelay02 kernel: [ 27.774674] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 3 12:23:37 smtprelay02 kernel: [ 27.774746] IPv6: ADDRCONF(NETDEV_CHANGE): vethce06c3c: link becomes ready
Sep 3 12:23:37 smtprelay02 kernel: [ 27.774816] br-mailcow: port 16(vethce06c3c) entered blocking state
Sep 3 12:23:37 smtprelay02 kernel: [ 27.774816] br-mailcow: port 16(vethce06c3c) entered forwarding state
Sep 3 12:23:38 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:38.534496356+02:00" level=info msg="Loading containers: done."
Sep 3 12:23:38 smtprelay02 systemd-networkd[1000]: vethe06f939: Gained IPv6LL
Sep 3 12:23:38 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:38.735362740+02:00" level=info msg="Docker daemon" commit=74b1e89 graphdriver(s)=overlay2 version=19.03.1
Sep 3 12:23:38 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:38.735785526+02:00" level=info msg="Daemon has completed initialization"
Sep 3 12:23:38 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:38.806077067+02:00" level=info msg="API listen on /var/run/docker.sock"

Sep 3 12:23:38 smtprelay02 systemd[1]: Started Docker Application Container Engine.

Sep 3 12:23:38 smtprelay02 systemd[1]: Reached target Multi-User System.

Sep 3 12:23:38 smtprelay02 systemd[1]: Reached target Graphical Interface.
Sep 3 12:23:38 smtprelay02 systemd[1]: Starting Execute cloud user/final scripts...
Sep 3 12:23:38 smtprelay02 systemd[1]: Starting Update UTMP about System Runlevel Changes...
Sep 3 12:23:38 smtprelay02 systemd[1]: Started Update UTMP about System Runlevel Changes.
Sep 3 12:23:39 smtprelay02 cloud-init[6530]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu118.04.1 running 'modules:final' at Tue, 03 Sep 2019 10:23:39 +0000. Up 29.12 seconds.
Sep 3 12:23:39 smtprelay02 cloud-init[6530]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1
18.04.1 finished at Tue, 03 Sep 2019 10:23:39 +0000. Datasource DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]. Up 29.26 seconds
Sep 3 12:23:39 smtprelay02 systemd[1]: Started Execute cloud user/final scripts.
Sep 3 12:23:39 smtprelay02 systemd[1]: Reached target Cloud-init target.
Sep 3 12:23:39 smtprelay02 systemd[1]: Startup finished in 3.747s (kernel) + 25.605s (userspace) = 29.352s.
Sep 3 12:23:39 smtprelay02 systemd-networkd[1000]: vethce06c3c: Gained IPv6LL
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 6 br-mailcow 172.22.1.1:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 7 br-mailcow [fd4d:6169:6c63:6f77::1]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 8 br-mailcow [fe80::42:b2ff:fe4e:1f7b%3]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 9 br-mailcow [fe80::1%3]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 10 vethd87bee6 [fe80::fc96:cbff:fe0b:2f6d%6]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 11 vethe7fa3e7 [fe80::d81f:4fff:fe50:3357%8]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 12 vethfb645ce [fe80::d052:93ff:fe5a:83e6%10]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 13 veth3cdfa48 [fe80::10ba:ffff:fea3:71f1%12]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 14 veth2cf0374 [fe80::5082:36ff:feb9:6ca5%14]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 15 veth2958443 [fe80::7ce3:3dff:fe9b:8337%16]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 16 vethf7a040a [fe80::c4d2:77ff:febd:3af2%18]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 17 veth9eddcff [fe80::c03:5aff:feb4:b57a%20]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 18 veth1b3a578 [fe80::4c6f:dfff:fef6:956a%22]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 19 vetha939816 [fe80::d0ce:2cff:feea:113e%24]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 20 vethdd28ca3 [fe80::dcb4:15ff:fe68:b219%26]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 21 veth58f4bcc [fe80::10a0:39ff:fe51:e2db%28]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 22 vethaf5af12 [fe80::c49a:edff:fe0b:ced1%30]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 23 veth70a186e [fe80::6cbe:adff:fec2:2c76%32]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 24 vethe06f939 [fe80::78ab:caff:fe70:501d%34]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 25 vethce06c3c [fe80::1cad:79ff:fe3e:f3a8%36]:123
Sep 3 12:23:40 smtprelay02 ntpd[1258]: new interface(s) found: waking up resolver

Sep 3 12:23:48 smtprelay02 kernel: [ 38.658727] ------------[ cut here ]------------
Sep 3 12:23:48 smtprelay02 kernel: [ 38.658731] kernel BUG at /build/linux-5mCauq/linux-4.15.0/net/ipv4/ip_output.c:636!
Sep 3 12:23:48 smtprelay02 kernel: [ 38.658792] invalid opcode: 0000 [#1] SMP PTI
Sep 3 12:23:48 smtprelay02 kernel: [ 38.658817] Modules linked in: ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_security ip6table_raw ip6table_mangle iptable_security iptable_raw iptable_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables nf_tables_ipv4 nf_tables veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc aufs overlay vmw_vsock_vmci_transport vsock vmw_balloon snd_ens1371 intel_rapl_perf snd_ac97_codec input_leds joydev serio_raw gameport snd_rawmidi snd_seq_device ac97_bus snd_pcm snd_timer snd soundcore vmw_vmci shpchp mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp

Sep 3 12:23:48 smtprelay02 kernel: [ 38.659034] libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd psmouse vmwgfx ttm drm_kms_helper syscopyarea mptspi sysfillrect sysimgblt mptscsih fb_sys_fops i2c_piix4 mptbase ahci e1000 drm libahci scsi_transport_spi pata_acpi
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659170] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.15.0-60-generic #67-Ubuntu
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659185] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659210] RIP: 0010:ip_do_fragment+0x482/0x820
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659220] RSP: 0018:ffff9458796c3a18 EFLAGS: 00010202
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659231] RAX: 0000000000000001 RBX: ffff9457be2e3000 RCX: ffffffffa104cdf0
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659245] RDX: 0000000000000024 RSI: 00000000000005c8 RDI: ffff9457b7be1500
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659260] RBP: ffff9458796c3a80 R08: ffff945873cacb80 R09: 00000000000005dc
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659274] R10: 0000000000000000 R11: ffff94586b977400 R12: 0000000000000014
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659288] R13: ffff9457b7be0f00 R14: 0000000000000d17 R15: ffff945873cacbce
Sep 3 12:23:48 smtprelay02 kernel: [ 38.659303] FS: 0000000000000000(0000) GS:ffff9458796c0000(0000) knlGS:0000000000000000`

@Shineson1001

This comment has been minimized.

Copy link

commented Sep 3, 2019

I get the kernel error, if i start the container manually ... the rspamd container crashes the system.

Sep 3 12:59:30 smtprelay02 kernel: [ 173.123882] br-mailcow: port 14(veth9540b17) entered blocking state
Sep 3 12:59:30 smtprelay02 kernel: [ 173.123884] br-mailcow: port 14(veth9540b17) entered disabled state
Sep 3 12:59:30 smtprelay02 kernel: [ 173.123952] device veth9540b17 entered promiscuous mode
Sep 3 12:59:30 smtprelay02 kernel: [ 173.124056] IPv6: ADDRCONF(NETDEV_UP): veth9540b17: link is not ready
Sep 3 12:59:30 smtprelay02 systemd-udevd[12772]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 3 12:59:30 smtprelay02 systemd-udevd[12773]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 3 12:59:30 smtprelay02 systemd-udevd[12772]: Could not generate persistent MAC address for vethc7f56a5: No such file or directory
Sep 3 12:59:30 smtprelay02 systemd-udevd[12773]: Could not generate persistent MAC address for veth9540b17: No such file or directory
Sep 3 12:59:30 smtprelay02 networkd-dispatcher[1001]: WARNING:Unknown index 87 seen, reloading interface list
Sep 3 12:59:30 smtprelay02 containerd[1062]: time="2019-09-03T12:59:30.469138265+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/2cf2287d059960f91f96ebc7346a24c46f3244b60ede7a908a891f714a9bc47d/shim.sock" debug=false pid=12780
Sep 3 12:59:31 smtprelay02 kernel: [ 173.757655] eth0: renamed from vethc7f56a5
Sep 3 12:59:31 smtprelay02 systemd-networkd[790]: veth9540b17: Gained carrier
Sep 3 12:59:31 smtprelay02 kernel: [ 173.832872] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:59:31 smtprelay02 kernel: [ 173.834984] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 3 12:59:31 smtprelay02 kernel: [ 173.835007] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 3 12:59:31 smtprelay02 kernel: [ 173.835100] IPv6: ADDRCONF(NETDEV_CHANGE): veth9540b17: link becomes ready
Sep 3 12:59:31 smtprelay02 kernel: [ 173.835273] br-mailcow: port 14(veth9540b17) entered blocking state
Sep 3 12:59:31 smtprelay02 kernel: [ 173.835276] br-mailcow: port 14(veth9540b17) entered forwarding state
Sep 3 12:59:32 smtprelay02 systemd-networkd[790]: veth9540b17: Gained IPv6LL
Sep 3 12:59:33 smtprelay02 ntpd[1016]: Listen normally on 43 veth9540b17 [fe80::fc01:94ff:fe4b:fb21%87]:123
Sep 3 12:59:33 smtprelay02 ntpd[1016]: new interface(s) found: waking up resolver
Sep 3 12:59:45 smtprelay02 kernel: [ 187.775225] ------------[ cut here ]------------
Sep 3 12:59:45 smtprelay02 kernel: [ 187.775241] kernel BUG at /build/linux-5mCauq/linux-4.15.0/net/ipv4/ip_output.c:636!
Sep 3 12:59:45 smtprelay02 kernel: [ 187.776628] invalid opcode: 0000 [#1] SMP PTI

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

No, I removed it, it still crashes.

I removed almost all containers manually. It always crashes.

Someone should file a bug at Ubuntu or Moby (but I think it is a Ubuntu issue).

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

No, I removed it, it still crashes.

I removed almost all containers manually. It always crashes.

Someone should file a bug at Ubuntu or Moby (but I think it is a Ubuntu issue).

"almost all", which ones left?

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

You cannot just stop containers, some containers may depend on each other in scripts and don't start before service X is available.

I removed Redis and Unbound and it works. But this does not mean much, as Redis is a dependency in many start scripts.

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

Remember to stop watchdog...

I removed unbound and all dns: flags - no crash so far.

But we need to report that anyway... if it breaks with a kernel upgrade and runs fine on all other kernels on Ubuntu 18.04 and even other OSes, it is probably not our fault. :/

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

No you can't just stop containers, as some are dependent on each other MYSQL etc.

Just wondered if you had go through the compose and taken out the dependencies, although they might just keep restarting as they'll error to due other services not being there.

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

Shall I create a bug on Ubuntu Launch Pad Kernel? or you guys thinking something else.

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

I mean, it works on other Kernels within Ubuntu. And all other distros.

I recommend to update to Kernel 5 as long as it isn't addressed.

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

It works without the...

      dns:
        - ${IPV4_NETWORK:-172.22.1}.254

...parameters.

Can you confirm?

It is important to remove all existing containers before trying this.

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

It works without the...

      dns:
        - ${IPV4_NETWORK:-172.22.1}.254

...parameters.

Can you confirm?

It is important to remove all existing containers before trying this.

Whats the - before the IP address for?

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

It's yml. key => value. :)

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

It's yml. key => value. :)

Ah yes, silly me. Was seeing it like a foreign character.

I'll test it when I get home and build a test instance but wont be for like 4-5 hours. So if someone else can test it before the go for it!

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

Thanks. :) Just try with update.sh, please. :)

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

Thanks. :) Just try with update.sh, please. :)

It'll be a fresh install, so shall be fully up to date!

Ah so RSPAMD and postfix will only use unbound.

Why not reference the DNS server by the container hostname? or just its ip? without the {$

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

I just updated the compose file, you need to update again.

It does not work like that. You cannot resolve a name without a resolver.

@fbcd4f50ea5350f051450323b5abfd0b

This comment has been minimized.

Copy link

commented Sep 3, 2019

I can confirm that the updated docker-compose works with the latest kernel.
Thanks for the quick fix!

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

It is strange as hell... can someone test which container crashes with the dns flag?

@fbcd4f50ea5350f051450323b5abfd0b

This comment has been minimized.

Copy link

commented Sep 3, 2019

Sorry, after a few minutes it crashed again. Also with the updated compose file.

@Shineson1001

This comment has been minimized.

Copy link

commented Sep 3, 2019

it does not work with the new docker-compose

`Sep 3 15:43:40 smtprelay kernel: [ 106.505952] ------------[ cut here ]------------
Sep 3 15:43:40 smtprelay kernel: [ 106.506032] kernel BUG at /build/linux-5mCauq/linux-4.15.0/net/ipv4/ip_output.c:636!
Sep 3 15:43:40 smtprelay kernel: [ 106.506152] invalid opcode: 0000 [#1] SMP PTI
Sep 3 15:43:40 smtprelay kernel: [ 106.506231] Modules linked in: ip6t_MASQUERADE nf_nat_masquerade_ipv6 nf_tables_ipv4 nf_tables ip6table_security ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_raw ip6table_mangle iptable_security iptable_raw iptable_mangle ip6table_filter ip6_tables xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc ipt_REJECT nf_reject_ipv4 aufs xt_multiport iptable_filter overlay vmw_vsock_vmci_transport vsock coretemp ppdev intel_rapl_perf joydev input_leds serio_raw vmw_balloon parport_pc parport shpchp vmw_vmci mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
Sep 3 15:43:40 smtprelay kernel: [ 106.506794] ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc vmwgfx ttm aesni_intel aes_x86_64 drm_kms_helper crypto_simd glue_helper syscopyarea cryptd sysfillrect sysimgblt fb_sys_fops mptspi psmouse mptscsih mptbase drm pata_acpi vmxnet3 scsi_transport_spi i2c_piix4 floppy
Sep 3 15:43:40 smtprelay kernel: [ 106.507116] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.15.0-60-generic #67-Ubuntu
Sep 3 15:43:40 smtprelay kernel: [ 106.507175] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
Sep 3 15:43:40 smtprelay kernel: [ 106.507266] RIP: 0010:ip_do_fragment+0x482/0x820
Sep 3 15:43:40 smtprelay kernel: [ 106.507305] RSP: 0018:ffff93837fd83a98 EFLAGS: 00010202
Sep 3 15:43:40 smtprelay kernel: [ 106.507348] RAX: 0000000000000001 RBX: ffff93836efd4200 RCX: ffffffffa204cdf0
Sep 3 15:43:40 smtprelay kernel: [ 106.507403] RDX: 0000000000000024 RSI: 00000000000005c8 RDI: ffff93836fe4b500
Sep 3 15:43:40 smtprelay kernel: [ 106.507458] RBP: ffff93837fd83b00 R08: ffff93836fe9e800 R09: 00000000000005dc
Sep 3 15:43:40 smtprelay kernel: [ 106.507537] R10: 0000000000000000 R11: ffff93836a9bc000 R12: 0000000000000014
Sep 3 15:43:40 smtprelay kernel: [ 106.507634] R13: ffff93836fe4b900 R14: 0000000000000681 R15: ffff93836fe9e84e
Sep 3 15:43:40 smtprelay kernel: [ 106.507690] FS: 0000000000000000(0000) GS:ffff93837fd80000(0000) knlGS:0000000000000000
Sep 3 15:43:40 smtprelay kernel: [ 106.509527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 3 15:43:40 smtprelay kernel: [ 106.511380] CR2: 00007fb34b201c52 CR3: 00000001c6d92000 CR4: 00000000000406e0
Sep 3 15:43:40 smtprelay kernel: [ 106.513337] Call Trace:
Sep 3 15:43:40 smtprelay kernel: [ 106.515161]
Sep 3 15:43:40 smtprelay kernel: [ 106.516934] ? ip_copy_metadata+0x220/0x220
Sep 3 15:43:40 smtprelay kernel: [ 106.518794] ip_fragment.constprop.45+0x43/0x80
Sep 3 15:43:40 smtprelay kernel: [ 106.520622] ip_finish_output+0x182/0x270
Sep 3 15:43:40 smtprelay kernel: [ 106.522299] ? nf_hook_slow+0x48/0xc0
Sep 3 15:43:40 smtprelay kernel: [ 106.523922] ip_output+0x70/0xe0
Sep 3 15:43:40 smtprelay kernel: [ 106.525515] ? ip_fragment.constprop.45+0x80/0x80
Sep 3 15:43:40 smtprelay kernel: [ 106.527090] ip_forward_finish+0x51/0x80
Sep 3 15:43:40 smtprelay kernel: [ 106.528629] ip_forward+0x376/0x470
Sep 3 15:43:40 smtprelay kernel: [ 106.530138] ? ip4_key_hashfn+0xc0/0xc0
Sep 3 15:43:40 smtprelay kernel: [ 106.531614] ip_rcv_finish+0x129/0x430
Sep 3 15:43:40 smtprelay kernel: [ 106.533050] ip_rcv+0x296/0x360
Sep 3 15:43:40 smtprelay kernel: [ 106.534458] ? inet_del_offload+0x40/0x40
Sep 3 15:43:40 smtprelay kernel: [ 106.535862] __netif_receive_skb_core+0x432/0xb80
Sep 3 15:43:40 smtprelay kernel: [ 106.537316] __netif_receive_skb+0x18/0x60
Sep 3 15:43:40 smtprelay kernel: [ 106.538671] ? __netif_receive_skb+0x18/0x60
Sep 3 15:43:40 smtprelay kernel: [ 106.540096] netif_receive_skb_internal+0x45/0xe0
Sep 3 15:43:40 smtprelay kernel: [ 106.541353] napi_gro_receive+0xc5/0xf0
Sep 3 15:43:40 smtprelay kernel: [ 106.542560] vmxnet3_rq_rx_complete+0x85b/0xeb0 [vmxnet3]
Sep 3 15:43:40 smtprelay kernel: [ 106.543745] vmxnet3_poll_rx_only+0x36/0xa0 [vmxnet3]
Sep 3 15:43:40 smtprelay kernel: [ 106.544856] net_rx_action+0x140/0x3a0
Sep 3 15:43:40 smtprelay kernel: [ 106.545943] __do_softirq+0xe4/0x2d4
Sep 3 15:43:40 smtprelay kernel: [ 106.546973] irq_exit+0xc5/0xd0
Sep 3 15:43:40 smtprelay kernel: [ 106.547980] do_IRQ+0x86/0xe0
Sep 3 15:43:40 smtprelay kernel: [ 106.549301] common_interrupt+0x8c/0x8c
Sep 3 15:43:40 smtprelay kernel: [ 106.550222]
Sep 3 15:43:40 smtprelay kernel: [ 106.551135] RIP: 0010:native_safe_halt+0x12/0x20
Sep 3 15:43:40 smtprelay kernel: [ 106.552303] RSP: 0018:ffffb1fac0ccfe80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd3
Sep 3 15:43:40 smtprelay kernel: [ 106.553245] RAX: ffffffffa21bb0f0 RBX: 0000000000000003 RCX: 0000000000000000
Sep 3 15:43:40 smtprelay kernel: [ 106.554212] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 3 15:43:40 smtprelay kernel: [ 106.555143] RBP: ffffb1fac0ccfe80 R08: 00000018cf5bf95f R09: ffff9383712aa100
Sep 3 15:43:40 smtprelay kernel: [ 106.556083] R10: 0000000000000000 R11: 7fffffffffffffff R12: 0000000000000003
Sep 3 15:43:40 smtprelay kernel: [ 106.557046] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Sep 3 15:43:40 smtprelay kernel: [ 106.558169] ? __sched_text_end+0x1/0x1
Sep 3 15:43:40 smtprelay kernel: [ 106.559068] default_idle+0x20/0x100
Sep 3 15:43:40 smtprelay kernel: [ 106.560061] arch_cpu_idle+0x15/0x20
Sep 3 15:43:40 smtprelay kernel: [ 106.560912] default_idle_call+0x23/0x30
Sep 3 15:43:40 smtprelay kernel: [ 106.561789] do_idle+0x172/0x1f0
Sep 3 15:43:40 smtprelay kernel: [ 106.562668] cpu_startup_entry+0x73/0x80
Sep 3 15:43:40 smtprelay kernel: [ 106.563525] start_secondary+0x1ab/0x200
Sep 3 15:43:40 smtprelay kernel: [ 106.564393] secondary_startup_64+0xa5/0xb0
Sep 3 15:43:40 smtprelay kernel: [ 106.565246] Code: 8b 87 d8 00 00 00 48 2b 87 d0 00 00 00 39 c2 0f 87 f4 00 00 00 8b 87 e4 00 00 00 83 f8 01 0f 85 e5 00 00 00 48 83 7f 18 00 74 8e <0f> 0b 8b 0a 89 08 44 89 e1 8b 54 0a fc 89 54 08 fc e9 00 fd ff
Sep 3 15:43:40 smtprelay kernel: [ 106.568090] RIP: ip_do_fragment+0x482/0x820 RSP: ffff93837fd83a98
Sep 3 15:43:40 smtprelay kernel: [ 106.569165] ---[ end trace fc721464731e6cec ]---

`

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

You need to make sure it removed all old containers.

It works here.

Please report that to Ubuntu!! Don't use their new kernel, report it, upgrade to kernel 5 (within Ubuntu).

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

How come you've done

  • ${IPV4_NETWORK:-172.22.1}.254
    instead of
  • 172.22.1.254

Just wondering

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

IPV4_NETWORK is a var.

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 3, 2019

I've +1 the bug on Launchpad

Also posted my test results, plus crash log.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447/comments/3

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

I will add a check for that kernel, when I get home.

@tmowbray

This comment has been minimized.

Copy link

commented Sep 3, 2019

Experienced similar issue with 4.15.0-60. Server would crash ~5 after a boot with no indication as to why. Have reverted back to 4.15.0-58 for now to keep client happy.

EDIT: This was an existing mailcow deployment that started crashing after updating Ubuntu with the latest kernel. I have also confirmed the bug report over on Ubuntu's launchpad

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 3, 2019

Can you guys confirm the report?

@Saalbuerger

This comment has been minimized.

Copy link

commented Sep 3, 2019

Experienced similar issue with 4.15.0-60. Server would crash ~5 after a boot with no indication as to why. Have reverted back to 4.15.0-58 for now to keep client happy.

+1

Workaround

  1. make a backup copy of /etc/default/grub
    sudo cp /etc/default/grub /etc/default/grub.bak

  2. find 4.15.0-58 in /boot/grub/grub.cfg
    sudo view /boot/grub/grub.cfg
    e.g. line menuentry 'Ubuntu, mit Linux 4.15.0-58-generic' --> 'gnulinux-4.15.0-58-generic-advanced-0473a811-2081-47ab-ac48-DUMMY-ID'

  3. edit /etc/default/grub (backup first!) and change GRUB_DEFAULT=0 to GRUB_DEFAULT='gnulinux-advanced-0473a811-2081-47ab-ac48-DUMMY-ID>gnulinux-4.15.0-58-generic-advanced-0473a811-2081-47ab-ac48-DUMMY-ID'

  4. create and check grup config
    sudo update-grub

  5. sudo reboot now

  6. check kernel Version
    uname -r --> 4.15.0-58-generic

  7. start mailcow

https://askubuntu.com/questions/100232/how-do-i-change-the-grub-boot-order/110738#110738

@jeslen

This comment has been minimized.

Copy link

commented Sep 4, 2019

Experienced similar issue with 4.15.0-60. Server would crash ~5 after a boot with no indication as to why. Have reverted back to 4.15.0-58 for now to keep client happy.

+1

Workaround

  1. make a backup copy of /etc/default/grub
    sudo cp /etc/default/grub /etc/default/grub.bak
  2. find 4.15.0-58 in /boot/grub/grub.cfg
    sudo view /boot/grub/grub.cfg
    e.g. line menuentry 'Ubuntu, mit Linux 4.15.0-58-generic' --> 'gnulinux-4.15.0-58-generic-advanced-0473a811-2081-47ab-ac48-DUMMY-ID'
  3. edit /etc/default/grub (backup first!) and change GRUB_DEFAULT=0 to GRUB_DEFAULT='gnulinux-advanced-0473a811-2081-47ab-ac48-DUMMY-ID>gnulinux-4.15.0-58-generic-advanced-0473a811-2081-47ab-ac48-DUMMY-ID'
  4. create and check grup config
    sudo update-grub
  5. sudo reboot now
  6. check kernel Version
    uname -r --> 4.15.0-58-generic
  7. start mailcow

https://askubuntu.com/questions/100232/how-do-i-change-the-grub-boot-order/110738#110738

I wish I found this 24 hours ago. Thank you.

@patschi

This comment has been minimized.

Copy link
Member

commented Sep 5, 2019

@FingerlessGlov3s

This comment has been minimized.

Copy link
Author

commented Sep 5, 2019

Wonder when the new kernel will be released :-)

Will the docker-compose.yml be reverted back to how it waS?

@Adorfer

This comment has been minimized.

Copy link

commented Sep 5, 2019

Now it hit me too... thanks ansible autoupdater roles...

grafik

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 5, 2019

:( It is fixed in -62 via bionic-proposed

@Adorfer

This comment has been minimized.

Copy link

commented Sep 5, 2019

p.s.: Locking/pinning the kernel version is easier done via the following steps.
(Do not continue if you see relevant error after executing the individual commands. For the first commands you may need to be quick, since the crash will happen within 50 seconds after full boot.)

sudo systemctl disable docker 
sudo systemctl stop docker
echo linux-image-4.15.0-58-generic hold | sudo dpkg --set-selections;
echo linux-image-generic hold | sudo dpkg --set-selections; 
echo linux-generic-hwe hold | sudo dpkg --set-selections;
sudo update-grub
sudo touch /forcefsck
sudo systemctl enable docker 
sudo reboot
@unixfox

This comment has been minimized.

Copy link
Contributor

commented Sep 5, 2019

I just fixed the issue by installing the kernel 5.0 on Ubuntu 18.04 by executing this command :

sudo apt-get install --install-recommends linux-generic-hwe-18.04
@Adorfer

This comment has been minimized.

Copy link

commented Sep 6, 2019

just for notice: it happend to me yesterday on 2 (two) different Ubuntu16.04 with HWE (running 4.15.0-60) too, different physical machines.

both VMs without docker, without webserver. Just a "simple" BGP-routers (with GRE-tunnels and SNAT)
Sometimes it took 10-20 minutes for a crash, but it crashed systematically.
in other words: The issue seems to have more real life impact than docker.

grafik

@andryyy

This comment has been minimized.

Copy link
Member

commented Sep 6, 2019

Yes, it actually was a bigger issue and not related to Docker, mailcow or something like that.

[Impact]
Some fragmentation+NAT workloads will cause kernel BUG/Ooops.

[Test case]
sudo iptables -t nat -I POSTROUTING -j MASQUERADE
sudo hping3 192.168.122.1 -s 1000 -p 2000 -d 60000

[Regression potential]
This could make fragmented packets stop flowing. So, make sure fragmented pings still work.
ping 192.168.122.1 -s 60000 still works, even with the above nat rule.
@marcx28

This comment has been minimized.

Copy link

commented Sep 10, 2019

The new Kernel has been released :)

@Adorfer

This comment has been minimized.

Copy link

commented Sep 10, 2019

The new Kernel has been released
not for Ubuntu 16.04LTS so far.

@unixfox

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2019

The new Kernel has been released
not for Ubuntu 16.04LTS so far.

but this issue is about ubuntu 18.04

@Adorfer

This comment has been minimized.

Copy link

commented Sep 10, 2019

but this issue is about ubuntu 18.04

shall we make a new issue for 16.04LTS?

@tmowbray

This comment has been minimized.

Copy link

commented Sep 10, 2019

but this issue is about ubuntu 18.04

shall we make a new issue for 16.04LTS?

I guess the question is, does this affect the current 16.04 kernel? I don't believe there have been any reports, however I've been focused 18.04

@apintocr

This comment has been minimized.

Copy link

commented Sep 10, 2019

@Adorfer

This comment has been minimized.

Copy link

commented Sep 10, 2019

I guess the question is, does this affect the current 16.04 kernel?

No, this is not the question.

I don't believe there have been any reports,

i agree with your belief.

My questions is: Since the mailcow i run on U16.04LTS is affected as well and since for 18.04LTS it is said to be resolved by deployment of 4.15.0-62:

a) close this issue and make a new one for 16.04LTS
b) keep this one open and widen the subjet to 16.04LTS-HWE+18.04LTS-NonHEWE

@templis

This comment has been minimized.

Copy link

commented Sep 10, 2019

No problems on Ubuntu 16.04 for me so far...
Have around 3-4 machines still on Ubuntu 16.04 with Mailcow running and this machines are not affected by kernel freezes.

@svengo

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2019

I think that ubuntu 16.04 with HWE uses the same kernel as ubuntu 18.04 without HWE.

@Adorfer

This comment has been minimized.

Copy link

commented Sep 10, 2019

No problems on Ubuntu 16.04 for me so far...

I say "i have a mailcow on 16.04LTS affected". And you reply "you have not": What is your conclusion from that, in relation to my question?
a) there is no such problem, i am just making stuff up
b) i should try use my 16.04 smarter not to be affected by this issue, since i am the only unlucky person.

@templis

This comment has been minimized.

Copy link

commented Sep 10, 2019

I say "i have a mailcow on 16.04LTS affected". And you reply "you have not": What is your conclusion from that, in relation to my question?
a) there is no such problem, i am just making stuff up
b) i should try use my 16.04 smarter not to be affected by this issue, since i am the only unlucky person.

You don't have a standard installation, so when you are able to install a custom kernel, you should be able to check to where to get a newer custom kernel: https://packages.ubuntu.com/search?keywords=linux-generic
There you can see, you must enable the proposed repo.
And here you can read how you can do this: https://wiki.ubuntu.com/Testing/EnableProposed
Elsewhere you can go back to standard and have no problem or you just wait for HWE release in upstream repo.
So yes - b) seems the correct answer...

And - If you would be nicer, there might be a lot more people who are willing to help you with your problems :/

@svengo

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2019

A fix was released for Ubuntu 18.04 LTS and Ubuntu 16.04 LTS: USN-4115-2: Linux kernel regression.
I think this issue should be closed.

@mkuron mkuron closed this Sep 11, 2019

@MAGICCC MAGICCC unpinned this issue Sep 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.