RRDs saving to incorrect location

[bfarney]

Behaviour

Steps to reproduce this issue

1. build aws-services-nms server, docker up
2. build aws-master-nms server, docker up
3. login to aws-master-nms server, change libre password
4. Global Settings -> Poller -> Distributed Poller -> Enter Memcached Host: AWS-SERVICES-NMS.DOMAIN.NOC
5. Global Settings -> Poller -> Distributed Poller -> Toggle Enable Distributed Polling
6. Run validate.php & validate.php -g distributedpoller on aws-master-nms
7. Add first device
8. RRD's are then created on aws-master-nms instead of on aws-services-nms where rrdcached is running. They are created in /librenms/rrd/device-name-added.domain.noc

Expected behaviour

RRD files should be created on the server running rrdcached

Actual behaviour

RRD files are created on the server that the webui is running on when the device is added

Configuration

• Docker version (type docker --version) : Docker version 19.03.12, build 48a66213fe
• Docker compose version if applicable (type docker-compose --version) : docker-compose version 1.26.2, build eefe0d31
• Platform (Debian 9, Ubuntu 18.04, ...) : Ubuntu 18.04
• System info (type uname -a) : Linux ip-172-29-1-51 5.3.0-1023-aws /opt/librenms/snmp-scan.py does not work when running in Docker container #25~18.04.1-Ubuntu SMP Fri Jun 5 15:18:30 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Include all necessary configuration files : docker-compose.yml, .env, ...

SERVER: AWS-SERVICES-NMS.DOMAIN.NOC
FILE: .env

#System Config
TZ=America/Chicago
PUID=1000
PGID=1000

#Misc
LIBRENMS_SNMP_COMMUNITY=infotech-test

SERVER: AWS-SERVICES-NMS.DOMAIN.NOC
FILE: docker-compose.yml

version: "3.5"

services:
  memcached:
    image: memcached:alpine
    container_name: librenms_memcached
    ports:
      - target: 11211
        published: 11211
        protocol: tcp
    environment:
      - "TZ=${TZ}"
    restart: always

  redis:
    image: redis:5.0-alpine
    container_name: librenms_redis
    ports:
      - target: 6379
        published: 6379
        protocol: tcp
    environment:
      - "TZ=${TZ}"
    restart: always

  rrdcached:
    image: crazymax/rrdcached
    container_name: librenms_rrdcached
    ports:
      - target: 42217
        published: 42217
        protocol: tcp
    volumes:
      - "./librenms/rrd:/data/db"
      - "./rrd-journal:/data/journal"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "LOG_LEVEL=LOG_INFO"
      - "WRITE_TIMEOUT=1800"
      - "WRITE_JITTER=1800"
      - "WRITE_THREADS=4"
      - "FLUSH_DEAD_DATA_INTERVAL=3600"
    restart: always

  dozzle:
    image: amir20/dozzle:latest
    container_name: dozzle
    restart: unless-stopped
    ports:
      - "9002:8080"
    environment:
      DOZZLE_LEVEL: info
      DOZZLE_TAILSIZE: 300
      DOZZLE_FILTER: "status=running"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

SERVER: AWS-MASTER-NMS.DOMAIN.NOC
FILE: .env

#System Config
TZ=America/Chicago
PUID=1000
PGID=1000
librehostname=AWS-MASTER-NMS.DOMAIN.NOC
dispatcherhostname=AWS-MASTER-DISP
sysloghostname=AWS-MASTER-SYSLOG

#Poller Settings
MEMCACHED_HOST=AWS-SERVICES-NMS.DOMAIN.NOC
MEMCACHED_PORT=11211
RRDCACHED_HOST=AWS-SERVICES-NMS.DOMAIN.NOC
RRDCACHED_PORT=42217

#Dispatcher Settings
dispatcher_id=AWS-MASTER-DISP-1
redis_host=AWS-SERVICES-NMS.DOMAIN.NOC
redis_port=6379

#database settings
MYSQL_DATABASE=librenms
MYSQL_USER=librenms
MYSQL_PASSWORD=randompasswordhere
#db_host=AWS-MASTER-NMS.DOMAIN.NOC
db_host=db

#Misc
LIBRENMS_SNMP_COMMUNITY=infotech-test
LIBRENMS_WEATHERMAP=false
LIBRENMS_WEATHERMAP_SCHEDULE=*/5 * * * *

SERVER: AWS-MASTER-NMS.DOMAIN.NOC
FILE: docker-compose.yml

version: "3.5"

services:
  db:
    image: mariadb:10.4
    container_name: librenms_db
    ports:
      - target: 3306
        published: 3306
        protocol: tcp
    command:
      - "mysqld"
      - "--innodb-file-per-table=1"
      - "--lower-case-table-names=0"
      - "--character-set-server=utf8"
      - "--collation-server=utf8_unicode_ci"
    volumes:
      - "./db:/var/lib/mysql"
    environment:
      - "TZ=${TZ}"
      - "MYSQL_ALLOW_EMPTY_PASSWORD=yes"
      - "MYSQL_DATABASE=${MYSQL_DATABASE}"
      - "MYSQL_USER=${MYSQL_USER}"
      - "MYSQL_PASSWORD=${MYSQL_PASSWORD}"
    restart: always

  librenms:
    image: librenms/librenms:latest
    container_name: librenms
    hostname: "${librehostname}"
    ports:
      - target: 8000
        published: 8000
        protocol: tcp
    depends_on:
      - db
    volumes:
      - "./librenms:/data"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=${db_host}"
      - "MEMCACHED_HOST=${MEMCACHED_HOST}"
      - "MEMCACHED_PORT=${MEMCACHED_PORT}"
      - "RRDCACHED_HOST=${RRDCACHED_HOST}"
      - "RRDCACHED_PORT=${RRDCACHED_PORT}"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
    restart: always
  
  dispatcher:
    image: librenms/librenms:latest
    container_name: librenms_dispatcher
    hostname: "${dispatcherhostname}"
    depends_on:
      - librenms
    volumes:
       - "./librenms:/data"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=${db_host}"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DISPATCHER_NODE_ID=${dispatcher_id}"
      - "DB_TIMEOUT=60"
      - "REDIS_HOST=${redis_host}"
      - "REDIS_PORT=${redis_port}"
      - "REDIS_DB=0"
      - "SIDECAR_DISPATCHER=1"
    restart: always

  dozzle:
    image: amir20/dozzle:latest
    container_name: dozzle
    restart: unless-stopped
    ports:
      - "9002:8080"
    environment:
      DOZZLE_LEVEL: info
      DOZZLE_TAILSIZE: 300
      DOZZLE_FILTER: "status=running"
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock

  syslog-ng:
    image: librenms/librenms:latest
    container_name: librenms_syslog
    hostname: "${sysloghostname}"
    depends_on:
      - librenms
    ports:
      - target: 514
        published: 514
        protocol: tcp
      - target: 514
        published: 514
        protocol: udp
    volumes:
      - "./librenms:/data"
    environment:
      - "TZ=${TZ}"
      - "PUID=${PUID}"
      - "PGID=${PGID}"
      - "DB_HOST=${db_host}"
      - "DB_NAME=${MYSQL_DATABASE}"
      - "DB_USER=${MYSQL_USER}"
      - "DB_PASSWORD=${MYSQL_PASSWORD}"
      - "DB_TIMEOUT=60"
      - "SIDECAR_SYSLOGNG=1"
    restart: always

Docker info

Server: AWS-SERVICES-NMS.DOMAIN.NOC

Client:
 Debug Mode: false

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.3.0-1023-aws
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 978.8MiB
 Name: ip-172-29-1-50
 ID: TL2G:QOW2:PFGL:KVSQ:PCTJ:YBOH:RFSQ:SXWG:7CHM:AIUR:BDJM:FCDF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Server: AWS-MASTER-NMS.DOMAIN.NOC

Client:
 Debug Mode: false

Server:
 Containers: 7
  Running: 5
  Paused: 0
  Stopped: 2
 Images: 3
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.3.0-1023-aws
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 978.8MiB
 Name: ip-172-29-1-51
 ID: 6275:4JBM:4ILW:JEGU:RZOB:HA2R:2CE3:RNCR:FZJE:YGTI:ST6A:EOLM
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Logs

RRDTool Output

-1
ERROR: rrdcached@AWS-SERVICES-NMS.DOMAIN.NOC:42217: RRD Error: opening '/data/db/device-name-added.domain.noc/port-id14.rrd': No such file or directory

Server: aws-master-nms
Output of command: sudo docker-compose run librenms ./validate.php

====================================
Component | Version
--------- | -------
LibreNMS  | 1.65.1
DB Schema | 2020_06_23_00522_alter_availability_perc_column (170)
PHP       | 7.3.19
Python    | 3.8.3
MySQL     | 10.4.13-MariaDB-1:10.4.13+maria~focal
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.8
====================================

[OK]    Installed from the official Docker image; no Composer required
[WARN]  You have no devices.
        [FIX]: 
        Consider adding a device such as localhost: /addhost
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  IPv6 is disabled on your server, you will not be able to add IPv6 devices.
[WARN]  Updates are managed through the official Docker image
[cmd] ./validate.php exited 0

Server: aws-master-nms
Output of command: sudo docker-compose run librenms ./validate.php -g distributedpoller

====================================
Component | Version
--------- | -------
LibreNMS  | 1.65.1
DB Schema | 2020_06_23_00522_alter_availability_perc_column (170)
PHP       | 7.3.19
Python    | 3.8.3
MySQL     | 10.4.13-MariaDB-1:10.4.13+maria~focal
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.8
====================================

[OK]    Installed from the official Docker image; no Composer required
Checking distributedpoller: OK
[OK]    Connection to memcached is ok
[cmd] ./validate.php exited 0

Server: aws-master-nms
Output of command: sudo docker ps -sa

ubuntu@ip-172-29-1-51:~/librenms$ sudo docker ps -sa
sudo: unable to resolve host ip-172-29-1-51
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS                         PORTS                                                  NAMES                                SIZE
6fc04415473a        librenms/librenms:latest   "/init ./validate.ph…"   About an hour ago   Exited (0) About an hour ago                                                          librenms_librenms_run_4d0b71c4dcf0   923kB (virtual 854MB)
ba474683b813        librenms/librenms:latest   "/init ./validate.php"   About an hour ago   Exited (0) About an hour ago                                                          librenms_librenms_run_a02dfd438364   923kB (virtual 854MB)
1b2c8cc6b1f6        librenms/librenms:latest   "/init"                  2 hours ago         Up 2 hours                     514/tcp, 8000/tcp, 514/udp                             librenms_dispatcher                  893kB (virtual 854MB)
ee27b58b9534        librenms/librenms:latest   "/init"                  2 hours ago         Up 2 hours                     0.0.0.0:514->514/tcp, 0.0.0.0:514->514/udp, 8000/tcp   librenms_syslog                      909kB (virtual 854MB)
1aa4059fb074        librenms/librenms:latest   "/init"                  2 hours ago         Up 2 hours                     514/tcp, 514/udp, 0.0.0.0:8000->8000/tcp               librenms                             1.22MB (virtual 854MB)
38a01131fd24        mariadb:10.4               "docker-entrypoint.s…"   2 hours ago         Up 2 hours                     0.0.0.0:3306->3306/tcp                                 librenms_db                          2B (virtual 400MB)
d5d6ae34e293        amir20/dozzle:latest       "/dozzle"                2 hours ago         Up 2 hours                     0.0.0.0:9002->8080/tcp                                 dozzle                               0B (virtual 12.7MB)

[bfarney]

I've been looking through the code. And I'm wondering if this behavior is being set by this line.

docker/rootfs/etc/cont-init.d/03-config.sh

Line 117 in ee2d7dd

\$config['rrd_dir'] = '/data/rrd';

I'm not confident in this but it's my best idea at this point? It could just as easily be something in my config.

Thanks!!
Brian

[crazy-max]

@bfarney Looking at your composition, following env vars are missing for the dispatcher service:

      - "MEMCACHED_HOST=${MEMCACHED_HOST}"
      - "MEMCACHED_PORT=${MEMCACHED_PORT}"
      - "RRDCACHED_HOST=${RRDCACHED_HOST}"
      - "RRDCACHED_PORT=${RRDCACHED_PORT}"

Please review your configuration following our example. Here we use a common librenms.env file across our containers.

[bfarney]

@crazy-max Thank you so much. I did see that librenms.env file as well, and I thought for sure I had tested that too. but clearly I did not! I added those variables and confirmed everything is working great. Again thank you for your help and time!!