clamav - outdated version breaks email delivery

[mwerle]

Describe the bug

Outdated clamav causes exim4 to reject all incoming emails.

THIS IS NOT ACCEPTABLE! Yes, for sure, there must be a warning, but it MUST NEVER break the entire email delivery system.

Relevant exim config:

av_scanner = clamd:/var/run/clamav/clamd.ctl
...
deny
malware = *
message = This message was detected as possible malware ($malware_name).

Exim log:

2024-02-28 22:46:43 1rfRlC-00F71v-Vp malware acl condition: clamd /var/run/clamav/clamd.ctl : unable to connect to UNIX socket (/var/run/clamav/clamd.ctl): No such file or directory
2024-02-28 22:46:43 1rfRlC-00F71v-Vp H=mta4.XXX.com (bp141.mta.XXX.com) [XXX.XXX.125.141] X=TLS1.2:ECDHE_SECP256R1__RSA_SHA512__AES_256_GCM:256 CV=no F=bounce-19_HTML-138758325-593165-6151033-13051@bounce.email.XXX.com temporarily rejected after DATA

Freshclam log:

Wed Feb 28 21:36:27 2024 -> --------------------------------------
Wed Feb 28 22:36:27 2024 -> Received signal: wake up
Wed Feb 28 22:36:27 2024 -> ClamAV update process started at Wed Feb 28 22:36:27 2024
Wed Feb 28 22:36:27 2024 -> WARNING: Your ClamAV installation is OUTDATED!
Wed Feb 28 22:36:27 2024 -> WARNING: Local version: 0.103.10 Recommended version: 0.103.11
Wed Feb 28 22:36:27 2024 -> DON'T PANIC! Read https://docs.clamav.net/manual/Installing.html
Wed Feb 28 22:36:27 2024 -> daily.cld database is up-to-date (version: 27199, sigs: 2054066, f-level: 90, builder: raynman)
Wed Feb 28 22:36:27 2024 -> main.cld database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Wed Feb 28 22:36:27 2024 -> bytecode.cld database is up-to-date (version: 335, sigs: 86, f-level: 90, builder: raynman)

How to reproduce the problem

No particular steps required; just have a long-running server.

root@boa:/var/log/clamav# clamconf -n
Checking configuration files in /etc/clamav

Config file: clamd.conf

PreludeAnalyzerName = "ClamAV"
LogFile = "/var/log/clamav/clamav.log"
LogFileMaxSize = "4294967295"
LogTime = "yes"
LogRotate = "yes"
ExtendedDetectionInfo = "yes"
LocalSocket = "/var/run/clamav/clamd.ctl"
LocalSocketGroup = "clamav"
LocalSocketMode = "666"
MaxConnectionQueueLength = "15"
MaxThreads = "12"
ReadTimeout = "180"
CommandReadTimeout = "5"
SendBufTimeout = "200"
SelfCheck = "3600"
User = "clamav"
BytecodeTimeout = "60000"
MaxScanTime = "120000"
MaxRecursion = "16"
PCREMatchLimit = "10000"
PCRERecMatchLimit = "5000"

Config file: freshclam.conf

LogFileMaxSize = "4294967295"
LogTime = "yes"
LogRotate = "yes"
UpdateLogFile = "/var/log/clamav/freshclam.log"
Checks = "24"
DatabaseMirror = "db.local.clamav.net", "database.clamav.net"
MaxAttempts = "5"
ReceiveTimeout = "30"

clamav-milter.conf not found

Software settings

Version: 0.103.10
Optional features supported: MEMPOOL IPv6 FRESHCLAM_DNS_FIX AUTOIT_EA06 BZIP2 LIBXML2 PCRE2 ICONV JSON

Database information

Database directory: /var/lib/clamav
main.cld: version 62, sigs: 6647427, built on Thu Sep 16 14:32:42 2021
bytecode.cld: version 335, sigs: 86, built on Tue Feb 27 16:37:24 2024
daily.cld: version 27199, sigs: 2054066, built on Wed Feb 28 10:31:56 2024
Total number of signatures: 8701579

Platform information

uname: Linux 5.10.0-18-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64
OS: linux-gnu, ARCH: x86_64, CPU: x86_64
Full OS version: Debian GNU/Linux 11 (bullseye)
zlib version: 1.2.11 (1.2.11), compile flags: a9
platform id: 0x0a21818108000000000a0201

Build information

GNU C: 10.2.1 20210110 (10.2.1)
CPPFLAGS: -Wdate-time -D_FORTIFY_SOURCE=2
CFLAGS: -g -O2 -ffile-prefix-map=/build/reproducible-path/clamav-0.103.10+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
CXXFLAGS: -g -O2 -ffile-prefix-map=/build/reproducible-path/clamav-0.103.10+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64
LDFLAGS: -Wl,-z,relro -Wl,-z,now -Wl,--as-needed
Configure: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--runstatedir=/run' '--disable-maintainer-mode' '--disable-dependency-tracking' 'CFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/clamav-0.103.10+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/clamav-0.103.10+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now -Wl,--as-needed' '--with-dbdir=/var/lib/clamav' '--sysconfdir=/etc/clamav' '--disable-clamav' '--disable-unrar' '--enable-milter' '--enable-dns-fix' '--with-libjson' '--with-system-libmspack' '--with-libcurl=/usr' '--with-gnu-ld' '--with-systemdsystemunitdir=/lib/systemd/system' 'build_alias=x86_64-linux-gnu' 'OBJCFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/clamav-0.103.10+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security'
sizeof(void*) = 8
Engine flevel: 129, dconf: 129

[micahsnyder]

Wed Feb 28 22:36:27 2024 -> WARNING: Your ClamAV installation is OUTDATED!
Wed Feb 28 22:36:27 2024 -> WARNING: Local version: 0.103.10 Recommended version: 0.103.11

These are just warning messages. It is non-fatal.

I cannot tell from your logs why your clamd service is not running and listening to the /var/run/clamav/clamd.ctl socket.

[mwerle]

Sorry, but it's the only reason I can find in the logs. Restarting clamav and recreating the socket causes the same issue; exim4 unable to connect to the clamav socket.

(ED; thank you for your fast reply, btw)

[micahsnyder]

The warnings are from freshclam, the signature database updater.
What happens in your logs when you try starting clamd?

Edit: That is, please check /var/log/clamav/clamav.log

[mwerle]

What happens in your logs when you try starting clamd?

clamav starts normally except for this warning and (re)creates its socket in /var/run/clamav (which I manually deleted to test it)

[micahsnyder]

So... it's working now?

[mwerle]

clamav is running, but exim4 cannot connect to the clamav socket. (I restarted exim4 as well). Hence I assumed it was due to the outdated version as I can't see anything else wrong.

[micahsnyder]

(which I manually deleted to test it)
...
clamav is running, but exim4 cannot connect to the clamav socket.

Have you since restarted clamd without deleting the socket file so that exim can open it?

[micahsnyder]

If /var/run/clamav/clamd.ctl exists and exim4 can't open it, then perhaps it is a permissions issue with the socket file, or the user/groups that exim4 is running with?

Can you tell if the exim4 user/process has read/write permissions for /var/run/clamav/clamd.ctl?

[mwerle]

Yes, my first steps were to restart clamav, freshclam, and exim4.
(I did this several times, including manually deleting the clamav socket to ensure clamav would recreate it)

Since that didn't fix the situation I started looking into log files; the only issue I saw was the OUTDATED warning. My apologies if I jumped the gun with this issue, but it's the only reason I could see for it breaking. My server currently has an uptime of over 500 days, and I haven't modified anything for many months. It does run daily updates, although is running Debian 11.. on my backlog to make some time to upgrade it.

Can you tell if the exim4 user/process has read/write permissions for /var/run/clamav/clamd.ctl?
Everybody does:

root@boa:/var/log/clamav# ls -la /var/run/clamav/
total 0
drwxr-xr-x 2 clamav root 60 Feb 28 23:29 .
drwxr-xr-x 37 root root 1120 Feb 28 10:26 ..
srw-rw-rw- 1 clamav clamav 0 Feb 28 23:29 clamd.ctl

[micahsnyder]

Do you see any errors or warnings or anything in /var/log/clamav/clamav.log? I'm not sure where else to look for clues.

How much RAM does your system have? We really ought to add this to our clamconf -n output so I wouldn't have to ask. Perhaps there's some issue with clamd running out of memory. The signature database appears to require ~1.3GB these days just to start, which means it would use at least 2.6GB on reload + any memory used during a scan, on top of RAM requirements for other software.

[mwerle]

Plenty of RAM, using about 4/12GB); system has never used swap that I'm aware of.

As for the OUTDATED, I am terribly sorry, but it appears that I've jumped the gun on this issue - I've now looked further back in the logs and see that freshclam has been reporting this warning for a while now, so it's certainly not the root cause of my current issue. So I'll close this bug as it's not actually the cause.

If I track down the cause I'll add a note here.

Thank you for your help and suggestions of where else to look for the issue. I've disabled clamav for now just so I can get emails flowing again and will see about updating the OS; it's been on my TODO for a while now anyway.

[micahsnyder]

So sorry we couldn't nail down the root cause. Best of luck.