1.0.0-rc: Freshclam fails download database files

[yasuhirokimura]

Describe the bug

I'm maintainer of FreeBSD ClamAV port. Since release candidate of ClamAV 1.0.0 is released, I tried updating ClamAV port from 0.105.1 to 1.0.0-rc on my private copy of FreeBSD ports repository.

https://github.com/yasuhirokimura/freebsd-ports/tree/clamav/security/clamav

I only change the source archive file to use from clamav-0.105.1.tar.gz to clamav-1.0.0-rc.tar.gz, and don't change anything else (build options, build and/or runtime dependency, etc.). And with this simple change 1.0.0-rc is built successfuly on FreeBSD 13.1-RELEASE amd64 environment.

Next I installed 1.0.0-rc on clean installed FreeBSD 13.1-RELEASE amd64 system and tested if it works fine. But unfortunately I faced problem. If I run freshclam, it fails to download database files as following.

--------------------------------------
freshclam daemon 1.0.0-rc (OS: FreeBSD, ARCH: amd64, CPU: amd64)
ClamAV update process started at Wed Oct 26 13:27:19 2022
daily database available for download (remote version: 26699)
ERROR: Verification: Can't verify database integrity
Trying again in 5 secs...
daily database available for download (remote version: 26699)
ERROR: Verification: Can't verify database integrity
Trying again in 5 secs...
daily database available for download (remote version: 26699)
ERROR: Verification: Can't verify database integrity
Giving up on https://database.clamav.net...
ERROR: Update failed for database: daily
ERROR: Database update process failed: Invalid or corrupted CVD/CLD database
ERROR: Update failed.
--------------------------------------
Update process terminated

How to reproduce the problem

1. Make clean install of FreeBSD 13.1-RELEASE amd64 system.
2. Login as root
3. Install Git with pkg install git
4. Clone my private copy of ports repository with cd /usr/ports; git clone -b clamav https://github.com/yasuhirokimura/freebsd-ports.git .
5. Uninstall all packages with pkg-static delete -af.
6. Install ClamAV 1.0.0-rc with cd /usr/ports/security/clamav; make BATCH=yes install.
7. Copy /usr/local/etc/clamd.conf.sample and /usr/local/etc/freshclam.conf.sample to /usr/local/etc/clamd.conf and /usr/local/etc/freshclam.conf respectively.
8. Add clamav_freshclam_enable="YES" to /etc/rc.conf.
9. Start freshclam with service clamav-freshclam start.
10. Check /var/log/clamav/freshclam.log.

[yasuhirokimura]

I remembered there are unit tests and tried them.

0.105.1 passes all tests.

[100% 1/1] cd /usr0/freebsd/ports/work/usr/ports/security/clamav/work/.build && /usr/local/bin/ctest
Test project /usr0/freebsd/ports/work/usr/ports/security/clamav/work/.build
    Start 1: libclamav
1/6 Test #1: libclamav ........................   Passed    6.56 sec
    Start 2: libclamav_rust
2/6 Test #2: libclamav_rust ...................   Passed   28.16 sec
    Start 3: clamscan
3/6 Test #3: clamscan .........................   Passed    1.75 sec
    Start 4: clamd
4/6 Test #4: clamd ............................   Passed   15.80 sec
    Start 5: freshclam
5/6 Test #5: freshclam ........................   Passed    5.37 sec
    Start 6: sigtool
6/6 Test #6: sigtool ..........................   Passed    0.10 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) =  57.73 sec

But 1.0.0-rc fails 2 tests.

[100% 1/1] cd /usr0/freebsd/ports/work/usr/ports/security/clamav/work/.build && /usr/local/bin/ctest
FAILED: CMakeFiles/check /usr0/freebsd/ports/work/usr/ports/security/clamav/work/.build/CMakeFiles/check 
cd /usr0/freebsd/ports/work/usr/ports/security/clamav/work/.build && /usr/local/bin/ctest
Test project /usr0/freebsd/ports/work/usr/ports/security/clamav/work/.build
    Start 1: libclamav
1/6 Test #1: libclamav ........................***Failed    6.70 sec
    Start 2: libclamav_rust
2/6 Test #2: libclamav_rust ...................   Passed    4.09 sec
    Start 3: clamscan
3/6 Test #3: clamscan .........................   Passed    0.06 sec
    Start 4: clamd
4/6 Test #4: clamd ............................   Passed   17.13 sec
    Start 5: freshclam
5/6 Test #5: freshclam ........................***Failed   41.26 sec
    Start 6: sigtool
6/6 Test #6: sigtool ..........................   Passed    0.10 sec

67% tests passed, 2 tests failed out of 6

Total Test time (real) =  69.34 sec

The following tests FAILED:
          1 - libclamav (Failed)
          5 - freshclam (Failed)
Errors while running CTest

Full test log: LastTest.log

[micahsnyder]

Thanks @yasuhirokimura for checking out the release candidate so promptly. The issue you're seeing is very unexpected. I had observed the message "Can't verify database integrity" once before in example program tests on 32bit linux, but never with freshclam or libclamav. I will investigate this as soon as possible and get back to you with whatever I find.

[yasuhirokimura]

@micahsnyder If you need more information, feel free to request me.

[yasuhirokimura]

I think I understand what is happening now. FreeBSD ClamAV ports uses TomsFastMath from FreeBSD package instead of bundled one. If I switch to use the latter, 1.0.0-rc passes all unit tests.

I checked repository of ClamAV and found following commit.

commit 17e52a09a
Author:     Micah Snyder <micasnyd@cisco.com>
AuthorDate: Tue Oct 18 16:29:51 2022 -0700
Commit:     Micah Snyder <micah.d.snyder@gmail.com>
CommitDate: Fri Oct 21 17:21:19 2022 -0700

    Fix EXE/DLL cert verification
    
    RSA certificate verification is failing because we accidentally lowered
    the max size of numbers for floating point multiplication with
    TomsFastMath when upgrading the vendored library code.
    
    This commit restores the default from 4096 to 8192.

Original TomsFastMath doesn't include this change. So it is natual the problem happens if external one is used.

Since 1.0.0-rc requires above change to TomsFastMath, it may be better to remove ENABLE_EXTERNAL_TOMSFASTMATH option from CMakeOptions.cmake.

[micahsnyder]

Per conversations in the mailing list, and from my investigation -- I don't believe the TomsFastMath change is related.

The issue I think was coincidentally not present when you tested after the change. It appears the failure occurs when updating daily.cld (not daily.cvd) with an incremental (.cdiff) update. After the failure, incremental update failure, Freshclam will download the whole daily.cvd, after which point it will successfully update once before failing again.

However, your idea to disable the external TomsFastMath library option may be unrelated, but is still valid. The Authenticode certificate verification feature will fail with an external TomsFastMath library that hasn't been modified to support larger floating point numbers, as it has been failing in 0.105.0/.1. I will split this off into a separate issue and and then create a fix for that.