clamonacc fatal error

[goshansp]

issue

clamonacc dev/104 crashes with below ERROR when a folder like i.e. clamav.git-repo is copied and we aproach of about ~10000 files opened. this issue occurs on all kernel versions. the default ulimit=1024 needs to be raised for this to occur.

ERROR: Clamonacc: clamonacc has experienced a fatal error, if you continue to see this error, please run clamonacc with --verbose and report the issue and crash report to the developers

reproduction steps

# raise root ulimit
[vagrant@rhel8 ~]$ sudo vim /etc/security/limits.conf # add following two lines 
root    soft    nofile  100000
root    hard    nofile  100000

# verify root ulimit
[vagrant@centos7 git]$ sudo su -
[root@centos7 ~]# ulimit -n
100000


# starting clamonacc
[vagrant@rhel8 ~]$ sudo git/clamav-devel/build/clamonacc/clamonacc -F --config-file=/etc/clamd.d/clamd.conf --stream --verbose

# wrecking havoc:
[vagrant@rhel8 ~]$ git clone https://github.com/Cisco-Talos/clamav.git
[vagrant@rhel8 ~]$ cp clamav-devel target_git_folder -r
[vagrant@rhel8 ~]$ cat /proc/sys/fs/file-nr
# repeat above folder copy until it `signal 11` around 11k

conf

TCPSocket 3310
TCPAddr 127.0.0.1
TemporaryDirectory /tmp/clamav
OnAccessExcludeUname clamscan
OnAccessIncludePath /home
OnAccessIncludePath /usr
OnAccessIncludePath /etc
DatabaseDirectory /var/lib/clamav

furher observations

• red hat 8 + Fedora 34 (ulimit 100k / max_user_watches = 500k). this issue can be reproduced
• cannot reproduce when clamav 103.2/103.3 installed from rpm on centos 8 or fedora 34.
• $ git clone https://github.com/Cisco-Talos/clamav.git does not cause crash (tried: 4x no crash)
• after a crash, proper termination (ctrl-c) sometimes impossible, maybe related to https://gitlab.com/goshansp/ansible-role-clamav/-/issues/5
• possibly related to no such file or directory clamonacc / ClamInotif: could not watch path #186 ?
• after the crash sometimes one can ctrl-c out, sometimes until segfault ... sometimes a kill -9 from other terminal needed.

log

ClamFanotif: attempting to feed consumer queue
Clamonacc: onas_clamonacc_exit(), signal 11
ERROR: Clamonacc: clamonacc has experienced a fatal error, if you continue to see this error, please run clamonacc with --verbose and report the issue and crash report to the developers            
Clamonacc: attempting to stop ddd thread ...
ClamInotif: onas_ddd_exit()
ClamInotif: stopped
Clamonacc: attempting to stop event consumer thread ...
ClamScanQueue: onas_scan_queue_exit()

Please let me know if there's anything for me to test. Any advice appreciated. Thanks.

[goshansp]

confirmed as well on dev/0.103.3 and rel/0.103. how do my binaries (fatal error) differ from the packaged rpms 0.103.2 (working)? what is different in the build of https://src.fedoraproject.org/rpms/clamav/blob/rawhide/f/clamav.spec ?

[micahsnyder]

I don't see anything that would explain it. 🤷‍♂️ 🙁

[goshansp]

still occurs without OnAccessExtraScanning yes so maybe originating rather from fanotify side ... issue reproduces reliably on refrence rhel 8 and fedora 34 (kernel 5.11) as well.

[goshansp]

cat /proc/sys/fs/file-nr will rise until ~11000 when we get Clamonacc: onas_clamonacc_exit(), signal 11

[goshansp]

after switching to a custom fork provided by @m-sola and centos/fdpass/unixsocket i cannot get clamonacc to crash anymore. unless i am missing something we should implement #199 which would allow us to move from stream/tcp to fdpass/unixsocket which might allow us to close this issue. unfortunately our POC has a gate on 27.07. which we aim to pass.

[goshansp]

#199 has been poc'd and its statical linkage seems to provide a robust workaround for the moment. a sustainable fix on the --stream fd leak would be much appreciated.

[m-sola]

FYI, this is fixed with #227

You can test by following instructions listed above while monitoring the proc fd dir for clamonacc while it's running.

e.g.

ps -C clamonacc -o pid=

ls -l /proc/insert_result_of_ps_command_here/fd

Run the ls command after the clamonacc log shows failed connections to clamd. Previously, clamonacc would leak fds on failed connection and you would be able to see the sockets by monitoring proc. With the fix, you can see via proc that these fds are correctly cleaned up.

[micahsnyder]

Here's the (now merged) PR for 0.103.4: #244