Dispatcher unable to connect to DB after initially working

[ZummiGummi]

Behaviour

Steps to reproduce this issue

1. update docker-compose.yml for 1.66
2. docker-compose pull to update from 1.64 to latest (1.66)
3. docker-compose up
4. working
5. stop (ctrl+c)
6. docker-compose up
7. dispatcher_1 fails with database error

db_1          | 2020-08-12 14:29:13 35 [Warning] Aborted connection 35 to db: 'librenms' user: 'librenms' host: '172.26.7.10' (Got an error reading communication packets)
dispatcher_1  | MainThread(INFO):Configuring LibreNMS service
dispatcher_1  | MainThread(INFO):Attempting to load .env from '/opt/librenms/.env'
dispatcher_1  | MainThread(INFO):Using pure python SQL client
dispatcher_1  | 'poller_enabled'
db_1          | 2020-08-12 14:29:14 37 [Warning] Aborted connection 37 to db: 'librenms' user: 'librenms' host: '172.26.7.10' (Got an error reading communication packets)
dispatcher_1  | MainThread(INFO):Configuring LibreNMS service
dispatcher_1  | MainThread(INFO):Attempting to load .env from '/opt/librenms/.env'
dispatcher_1  | MainThread(INFO):Using pure python SQL client

Expected behaviour

Dispatcher continues to work after being stopped.

Actual behaviour

Dispatcher fails to connect to the database and there is no polling.

Configuration

• Docker version (type docker --version) : Docker version 19.03.11, build 42e35e61f3
• Docker compose version if applicable (type docker-compose --version) : docker-compose version 1.24.1, build 4667896b
• Platform (Debian 9, Ubuntu 18.04, ...) : Debian 10
• System info (type uname -a) : Linux 4.19.0-9-amd64 can not connect to librenms gui - 404 page not found #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux
• Include all necessary configuration files : docker-compose.yml, .env, ...

docker-compose.yml

version: "3.5"

services:
  db:
    image: mariadb:10.4
    container_name: librenms_db
    command:
      - "mysqld"
      - "--innodb-file-per-table=1"
      - "--lower-case-table-names=0"
      - "--character-set-server=utf8"
      - "--collation-server=utf8_unicode_ci"
    volumes:
      - "./db:/var/lib/mysql"
    environment:
      - "TZ=${TZ}"
      - "MYSQL_ALLOW_EMPTY_PASSWORD=yes"
      - "MYSQL_DATABASE=${MYSQL_DATABASE}"
      - "MYSQL_USER=${MYSQL_USER}"
      - "MYSQL_PASSWORD=${MYSQL_PASSWORD}"
    restart: always

  memcached:
    image: memcached:alpine
    container_name: librenms_memcached
    environment:
      - "TZ=${TZ}"
    restart: always

  redis:
    image: redis:5.0-alpine
    container_name: librenms_redis
    environment:
      - "TZ=${TZ}"
    restart: always

  rrdcached:
    image: crazymax/rrdcached
    container_name: librenms_rrdcached
    volumes:
      - "./librenms/rrd:/data/db"
      - "./rrd-journal:/data/journal"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "LOG_LEVEL=LOG_INFO"
      - "WRITE_TIMEOUT=1800"
      - "WRITE_JITTER=1800"
      - "WRITE_THREADS=4"
      - "FLUSH_DEAD_DATA_INTERVAL=3600"
    restart: always

  msmtpd:
    image: crazymax/msmtpd:latest
    container_name: librenms_msmtpd
    env_file:
      - "./msmtpd.env"
    restart: always

  librenms:
    image: librenms/librenms:latest
    container_name: librenms
    hostname: librenms
    depends_on:
      - db
      - memcached
      - rrdcached
      - msmtpd
    volumes:
      - "./librenms:/data"
    labels:
      - traefik.enable=true
      - traefik.http.routers.librenms.entrypoints=web
      - traefik.http.routers.librenms.rule=Host(`librenms.example.com`)
      - traefik.http.services.librenms.loadbalancer.server.port=8000
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=db"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
    env_file:
      - "./librenms.env"
    restart: always
    networks:
      - traefik
      - default

  dispatcher:
    image: librenms/librenms:latest
    container_name: librenms_dispatcher
    hostname: librenms-dispatcher
    depends_on:
      - librenms
      - redis
    volumes:
      - "./librenms:/data"
    env_file:
      - "./librenms.env"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=db"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
      - "DISPATCHER_NODE_ID=dispatcher1"
      - "REDIS_HOST=redis"
      - "REDIS_PORT=6379"
      - "REDIS_DB=0"
      - "SIDECAR_DISPATCHER=1"
    restart: always

  syslogng:
    image: librenms/librenms:latest
    container_name: librenms_syslog
    hostname: librenms-syslogng
    depends_on:
      - librenms
    ports:
      - target: 514
        published: 514
        protocol: tcp
      - target: 514
        published: 514
        protocol: udp
    volumes:
      - "./librenms:/data"
    environment:
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=db"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
      - "SIDECAR_SYSLOGNG=1"
    env_file:
      - "./librenms.env"
    restart: always

  oxidized:
    image: oxidized/oxidized:latest
    container_name: oxidized
    environment:
      - "CONFIG_RELOAD_INTERVAL=600"
    volumes:
      - ./oxidized:/root/.config/oxidized
    restart: always

networks:
  traefik:
    external: true

.env

MYSQL_DATABASE=librenms
MYSQL_USER=librenms
MYSQL_PASSWORD=redacted

TZ=NZ
PUID=1000
PGID=1000

librenms.env

MEMORY_LIMIT=256M
UPLOAD_MAX_SIZE=16M
OPCACHE_MEM_SIZE=128
REAL_IP_FROM=0.0.0.0/32
REAL_IP_HEADER=X-Forwarded-For
LOG_IP_VAR=http_x_forwarded_for

LIBRENMS_SNMP_COMMUNITY=public
MEMCACHED_HOST=memcached
MEMCACHED_PORT=11211
RRDCACHED_HOST=rrdcached
RRDCACHED_PORT=42217

LIBRENMS_WEATHERMAP=false
LIBRENMS_WEATHERMAP_SCHEDULE=*/5 * * * *

Docker info

Client:
 Debug Mode: false

Server:
 Containers: 14
  Running: 5
  Paused: 0
  Stopped: 9
 Images: 40
 Server Version: 19.03.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-9-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.832GiB
 Name: redacted
 ID: ESKN:QCAL:Y224:7N3F:5V6R:77UY:IPGK:WBBQ:34GE:NV5T:M7LZ:VBE5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Logs

> Container logs (set LOG_LEVEL to debug if applicable)

logs.txt

[ZummiGummi]

Just a note, that deleting the db mapped volume fixes the problem but I would prefer to not do that.

[crazy-max]

@ZummiGummi

Just a note, that deleting the db mapped volume fixes the problem but I would prefer to not do that.

So after deleting the db everything works fine now? Even after restarting your stack?

[ZummiGummi]

I can confirm that after deleting db and restarting multiple times, it continues to work.

[crazy-max]

@ZummiGummi Ok looks like the same issue as #80 (comment). Come back to me if this issue occurs again.

[ZummiGummi]

Very inconsistent behavior trying to pinpoint the cause, but I think this link fixed the problem.
https://community.librenms.org/t/your-database-is-out-of-date-fix-lnms-migrate-error/12275/8