all devices are offline after first scan and NBTSCAN + AVAHISCAN stuck if enabled

[tradexsrl]

What installation are you running?

Production (netalertx) 📦

Did I research?

• I have searched the docs https://docs.netalertx.com/
• I have searched the existing open and closed issues
• I confirm my SCAN_SUBNETS is configured and tested as per https://docs.netalertx.com/SUBNETS

The issue occurs in the following browsers. Select at least 2.

• Firefox
• Chrome
• Other (unsupported) - PRs welcome
• N/A - This is an issue with the backend

What I want to do

Hi , i'm using for a while netalertx v26.1.17 with docker-compose on a rocky linux 9.5 with selinux disabled "paired" with a remote node . at the beginning device in this node appear correctly online and offline , now all devices are offline , instead the device pulled from the node are correctly online and this is not possible because are for example production servers and client that are running
i tried to do some maintenance using the maintenance section and the tool section , then after the first scan device reappear online but then again after a while again all offline. i stop the stack in portainer , delete all file from db directory and start again the stack , again the same issue and i notice the NBTSCAN + AVAHISCAN stuck. so i decide to disable these plugin by first disabling the schedule and then delete them from the loaded plugins. in this way scanning is not stuck but again all devices are offline
please see docker-compose , app.conf , logs attached for app and from stdout when trace is enabled
the persmission on db and config subdir are 20211:20211 0775

Relevant settings you changed

appconf.txt

docker-compose.yml

services:
  netalertx:
    image: jokobsk/netalertx:latest
    container_name: netalertx
    network_mode: "host"
    restart: always

    environment:
      - TZ=Europe/Rome
      - PUID=20211
      - PGID=20211
      - PORT=20211
      - LISTEN_ADDR=0.0.0.0
      - GRAPHQL_PORT=20212
    cap_drop:
      - ALL
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - NET_BIND_SERVICE
      - CHOWN
      - SETUID
      - SETGID

    read_only: true

    tmpfs:
      - "/tmp:uid=20211,gid=20211,mode=1700,rw,noexec,nosuid,nodev,async,noatime,nodiratime"

    volumes:
      - /docker/netalertx/data:/data
      - /etc/localtime:/etc/localtime:ro

app.log

app.zip

stdout.zip

Debug enabled

• I have read and followed the steps in the wiki link above and provided the required debug logs and the log section covers the time when the issue occurs.

[jokob-sk]

Hi @tradexsrl ,

Thanks for the logs. This helps a lot.

Looks like your SYNC plugin schedule is different from the other scanners, that's why you see that behavior. See how [Scheduler] run for SYNC: YES but other plugins are not executed because schedules differ:

5:11:02 [Scheduler] run for ARPSCAN: NO 👈
15:11:02 [Scheduler] run for INTRNT: NO
15:11:02 [Scheduler] run for NMAPDEV: NO
15:11:02 [Scheduler] run for ICMP: NO
15:11:02 [Scheduler] run for NMAP: NO
15:11:02 [Scheduler] run for CSVBCKP: NO
15:11:02 [Scheduler] run for DBCLNP: NO
15:11:02 [Scheduler] run for MAINT: NO 
15:11:02 [Scheduler] run for SYNC: YES 👈
15:11:02 [SSE] Broadcasted event: state_update
... and then later Disconnections jump 👇
15:11:03 [Process Scan]  Exclude ignored devices
15:11:03 [New Devices] Excluding Ignored Devices Query: DELETE FROM CurrentScan WHERE 1=1 AND 1=0
15:11:03 [Process Scan]  Processing scan results
15:11:03 [Save Devices] Saving this IP into the CurrentScan table: 192.168.10.202
15:11:03 [Process Scan] Print Stats
15:11:03 [Scan Stats] Devices Detected.......: 8
15:11:03 [Scan Stats] New Devices............: 0
15:11:03 [Scan Stats] Down Alerts............: 0
15:11:03 [Scan Stats] New Down Alerts........: 0
15:11:03 [Scan Stats] New Connections........: 3
15:11:03 [Scan Stats] Disconnections.........: 145 👈
15:11:03 [Scan Stats] IP Changes.............: 0

You can also see it in your config:

SYNC_RUN_SCHD='*/5 * * * *'
...
ARPSCAN_RUN_SCHD='*/60 * * * *'

All devices have to be detected in the same scan to be considered online. Different schedules = different scans. Make sure all scans (including SYNC) have the same schedule and the issues should disappear. More info can be found here: https://docs.netalertx.com/NOTIFICATIONS/?h=same+schedule+interval

[tradexsrl]

Hi , i immediately setup sync to same schedule of other plugin and let run the system with trace activated over the weekend but up to now all devices in this node are offline except itself and the devices from other node
strange is the last seen is updated

logs.zip

i attached the app.conf , logs unfortunately are over 1gb and trying to download from the gui it say internal server error. so i just restarted the stack and attached are the stdout.log - app.log of at least 3 hours

appconf.txt

[jokob-sk]

Hi,

Can you try to set your SYNC plugin to run on a schedule:

SYNC_RUN='always_after_scan' => SYNC_RUN='schedule'

Also maybe instead of once every hour try first minute of every hour so teh schedules don't drift:

*/60 * * * * -> 0 * * * *

Replace all */60 * * * * with 0 * * * *. This will execute scans at exactly 1:00, 2:00, etc.

EDIT: *0 * * * * corrected to 0 * * * *

[tradexsrl]

hi , thank's for the tip , i setup sync to schedule and the other scan to 0 * * * * and now it shows device online.
very strange , on the other node i have */60 * * * * , sync = always after scan and it works
this is a rocky linux vm
side note: notification on ntfy also doesn't work anymore , i have setup to be notified on new events but i do not receive anymore notification from netalertx. I have read other discussion that there are problem with ntfy and to use -dev image or wait the next release , when there will be the new release ?

[jokob-sk]

Glad to hear the issue is resolved.

Please open a separate issue if you believe NTFY notifications don't work or try the -dev image if possible.

Closing as original setup issue is solved.