Argus Out of Memory

[keomabrun]

Argus was not reachable this morning:

nc -zv argus.paris.inria.fr 1883
nc: connect to argus.paris.inria.fr port 1883 (tcp) failed: Connection refused

The service showed errors:

kbrunlag@argus:~$ systemctl status mosquitto.service 
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)
   Active: active (exited) since Fri 2018-02-09 00:44:43 CET; 6 months 16 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/mosquitto.service

May 12 10:58:49 argus mosquitto[552]: warning: can't get client address: Connection reset by peer
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

The journal showed "Out of memory" issue:

kbrunlag@argus:~$ journalctl -xe | grep mosquitto
Aug 24 21:54:10 argus kernel: [  552]   123   552  2940887   927210    5750      15  2004857             0 mosquitto
Aug 24 21:54:10 argus kernel: Out of memory: Kill process 552 (mosquitto) score 965 or sacrifice child
Aug 24 21:54:10 argus kernel: Killed process 552 (mosquitto) total-vm:11763548kB, anon-rss:3708840kB, file-rss:0kB, shmem-rss:0kB

Restarting the service made Mosquitto work again:

nc -zv argus.paris.inria.fr 1883 
Connection to argus.paris.inria.fr 1883 port [tcp/*] succeeded!

This issue is not solved and is likely to happen again.
In the long term we need a monitoring tool for our servers.