Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

possible core issue in service start order: c-icap + clamav #276

Closed
kekek2 opened this issue Sep 19, 2017 · 28 comments
Closed

possible core issue in service start order: c-icap + clamav #276

kekek2 opened this issue Sep 19, 2017 · 28 comments
Assignees
Labels
bug Production bug

Comments

@kekek2
Copy link

kekek2 commented Sep 19, 2017

When system boots, c-icap starts before clamd, and could not open tcp socket:

Mon Sep 18 15:38:12 2017, main proc, clamd_connect: Can not connect to clamd server on 127.0.0.1:3310!
Mon Sep 18 15:38:12 2017, main proc, clamd_init: Error while sending command to clamd server
Mon Sep 18 15:38:12 2017, main proc, Registry 'virus_scan::engines' does not exist!

@fichtner
Copy link
Member

This may be a glitch in the proverbial matrix... https://github.com/opnsense/core/blob/master/src/etc/rc.freebsd starts services in a more or less fixed random order

@mimugmail
Copy link
Member

@kekek2 Where do you get this error? I can't reproduce it. I'm seeing c_icap starting before clamd but I dont see such errors.

@fichtner
Copy link
Member

I think it can depend on how the file system returns the file names from the disk ;/

https://github.com/opnsense/core/blob/master/src/etc/rc.freebsd#L68

@kekek2
Copy link
Author

kekek2 commented Sep 19, 2017

/var/log/system.log:

Sep 15 12:19:12 vibro kernel: <118>Starting c_icap.
Sep 15 12:19:12 vibro kernel: <118>Starting clamav_clamd.

/var/log/c-icap/server.log:

Fri Sep 15 12:20:26 2017, main proc, clamd_connect: Can not connect to clamd server on 127.0.0.1:3310!
Fri Sep 15 12:20:26 2017, main proc, clamd_init: Error while sending command to clamd server

time configctl clamav start

OK
configctl clamav start 0,07s user 0,06s system 0% cpu 1:27,73 total

@mimugmail
Copy link
Member

But it retries and finally works without manual interaction, correct?

@kekek2
Copy link
Author

kekek2 commented Sep 19, 2017

Only manual restart c_icap

Tue Sep 19 07:33:52 2017, main proc, clamd_connect: Can not connect to clamd server on 127.0.0.1:3310!
Tue Sep 19 07:33:52 2017, main proc, clamd_init: Error while sending command to clamd server
Tue Sep 19 07:33:52 2017, main proc, Registry 'virus_scan::engines' does not exist!
Tue Sep 19 12:05:49 2017, 39401/1742841344, Registry 'virus_scan::engines' does not exist!
Tue Sep 19 12:05:49 2017, 39401/1742841344, Registry 'virus_scan::engines' does not exist!
Tue Sep 19 12:05:49 2017, 39401/1742840064, Registry 'virus_scan::engines' does not exist!
Tue Sep 19 12:05:49 2017, 39401/1742840064, Registry 'virus_scan::engines' does not exist!
Tue Sep 19 12:05:49 2017, 39401/1742840064, Antivirus engine is not available, allow 204

@fichtner fichtner self-assigned this Sep 21, 2017
@fichtner fichtner changed the title c-icap + clamav possible core issue in service start order: c-icap + clamav Sep 21, 2017
@fichtner
Copy link
Member

I dunno, maybe we need to delay a bit after c-icap start?

@mimugmail
Copy link
Member

Why isn't there a numbering scheme like on linux with S03clamd where you can easily set a boot order? :/

@fichtner
Copy link
Member

well there is but it requires the full rc system of freebsd or better emulation to allow ordering

@fabianfrz
Copy link
Member

@mimugmail on current GNU/Linux this does not exist as well (1). What you can do is writing a dependency into the unit file which should also work with with rc.

(1) look for "After" in systemd service files

@ffries
Copy link

ffries commented Sep 23, 2017

Same issue here.

After restarting c-icap, works like a charm:

VIRUS FOUND
You tried to upload/download a file that contains the virus: Eicar-Test-Signature
The Http location is: http://www.eicar.org/download/eicar_com.zip
For more information contact your system administrator
This message generated by C-ICAP service: avscan?allow204=on&sizelimit=off&mode=simple
Antivirus engine: clamd-0992/23864

Thanks!

@Stephanowicz
Copy link

Stephanowicz commented Sep 29, 2017

Is there a way to 're-register' the clamav module without restarting c-icap?
(Even better would be that c-icap would do it automatically...)

I ask cause clamd crashes sometimes unexpectedly - using monit it can be restarted, but then c-icap has still the error

@fichtner
Copy link
Member

you can also cycle c-icap the same way?

@kekek2
Copy link
Author

kekek2 commented Sep 29, 2017

My workaround

diff --git a/www/c-icap/src/opnsense/scripts/OPNsense/CICAP/setup.sh b/www/c-icap/src/opnsense/scripts/OPNsense/CICAP/setup.sh
index 9e1e741..e84a7d5 100755
--- a/www/c-icap/src/opnsense/scripts/OPNsense/CICAP/setup.sh
+++ b/www/c-icap/src/opnsense/scripts/OPNsense/CICAP/setup.sh
@@ -7,3 +7,52 @@ chmod 750 /var/run/c-icap
 mkdir -p /var/log/c-icap
 chown -R c_icap:c_icap /var/log/c-icap
 chmod 750 /var/log/c-icap
+
+# check which services to enable
+if [ -f /etc/rc.conf ]; then
+	. /etc/rc.conf
+fi
+if [ -f /etc/rc.conf.local ]; then
+	. /etc/rc.conf.local
+fi
+for RC_CONF in $(find /etc/rc.conf.d -type f); do
+	. ${RC_CONF}
+done
+
+rc_enabled()
+{
+	rc_filename=${1}
+	name=${2}
+
+	# check if service has a name
+	if [ -z "${name}" ]; then
+		echo "Error: no name set in ${rc_filename}"
+		return 1
+	fi
+
+	# check if service has a variable
+	eval "$(grep "^rcvar[[:blank:]]*=" ${rc_filename})"
+	if [ -z "${rcvar}" ]; then
+		# FreeBSD does this, leave here for debugging
+		#echo "Error: no rcvar set in $rc_filename"
+		return 1
+	fi
+
+	# check if service is enabled
+	eval "enabled=\$${rcvar}"
+	if [ "${enabled}" != "YES" ]; then
+		return 1
+	fi
+
+	return 0
+}
+
+rc_filename="/usr/local/etc/rc.d/clamav-clamd"
+eval "$(grep "^name[[:blank:]]*=" ${rc_filename})"
+
+if ! rc_enabled ${rc_filename} ${name}; then
+	return 0
+fi
+
+/usr/local/opnsense/scripts/OPNsense/ClamAV/setup.sh
+${rc_filename} start

Edit by Fabian: Correct Syntax highlighting

@kekek2
Copy link
Author

kekek2 commented Sep 29, 2017

The formatting was off...

@Stephanowicz
Copy link

@fichtner

you can also cycle c-icap the same way?

...You mean me?^^
I now created a restart script for clamd which is called by monit when clamd crashes

#!/bin/sh

/usr/local/etc/rc.d/clamav-clamd start
sleep 5
/usr/local/etc/rc.d/c-icap restart

@fichtner
Copy link
Member

@Stephanowicz yes, does this work as expected?

@Stephanowicz
Copy link

Stephanowicz commented Sep 30, 2017

yep, I tried it manually by stopping clamd....

Nevertheless I consider this jut as a workaround - maybe the c-icap guys could implement something that tries to start the av-scanner when detecting that it can't be accessed and then reinits the Connection...
This would also fix the main topic here

@fichtner
Copy link
Member

Ok so we know what to do, now we need to figure out a robust way to integrate it.

Alex’s suggestion isn’t bad but it should ideally be more portable / avoid code duplication.

@Stephanowicz
Copy link

Ok, meanwhile I stumbled over the same problem as in OT...

Regarding monit and its possibility to also test file contents I made another workaround:

In monit add a new test with condition
content = "Registry 'virus_scan::engines' does not exist" for 3 cycles
(not tested enough, yet - could be that one needs to lower this to 2 cycles or to remove it completely)

then add a new service test with file as type and /var/log/c-icap/server.log as path
As start option I used the script mentioned above - it does not check if clamd is already running - but this does not matter as the clamd startscript itself takes care of this. (maybe it would be even better to use the restart option...?)

One 'annoying' thing is, that You have to add a start/stop/restart parameter, though it's not needed...

One also needs to enter a stop script, otherwise You'll get a syntax error - I used /usr/local/etc/rc.d/c-icap stop - probably a dummy script would be the better solution...

As test enter the test You created before.

I copied the error text manually to the log file for testing and received a

Sep 30 16:45:05 lifesense monit[18659]: 'c-icap' content match: Sat Sep 30 16:44:46 2017, 36646/3592524288, Registry 'virus_scan::engines' does not exist!

hope this helps!

Cheers,

Stephan

@Stephanowicz
Copy link

After having tested it now thoroughly I changed the test condition to
content = "(Registry 'virus_scan::engines' does not exist)|(clamd_connect: Can not connect to clamd server)" for 2 cycles
when stopping clamd the first condition doesn't show up, but the second one.

It also has been shown that c-icap doesn't restart properly in this condition - but monit will try it again the next cycle and then succeeds...

So, this is working, but it feels somehow like a dirty hack... :D

@Stephanowicz
Copy link

Update:
After a scheduled reboot this morning we had the same issue that c-icap was started before clamd.
Monit resolved it properly:

Oct 1 04:05:22 monit[11037]: 'lifesense.life' Monit 5.24.0 started 

Oct 1 04:05:23 monit[11037]: 'c-icap' content match: Sun Oct 1 04:03:04 2017, main proc, clamd_connect: Can not connect to clamd server on 127.0.0.1:3310! Sun Oct 1 04:03:04 2017, main proc, Registry 'virus_scan::engines' does not exist! Sun Oct 1 04:03:42 2017, 97721/253863424, Registry 'virus_scan::engines' does not exist! Sun Oct 1 04:03:42 2017, 97721/253863424, Registry 'virus_scan::engines' does not exist! Sun Oct 1 04:03:54 2017, 97721/253854464, Registry 'virus_scan::engines' does not exist! Sun Oct 1 04:03:54 2017, 97721/253854464, Registry 'virus_scan::engines' does not exist! ... 

Oct 1 04:07:23 monit[11037]: 'c-icap' content match: Sun Oct 1 04:07:02 2017, 97721/253863424, Registry 'virus_scan::engines' does not exist! Sun Oct 1 04:07:02 2017, 97721/253854464, Registry 'virus_scan::engines' does not exist! Sun Oct 1 04:07:02 2017, 97721/253863424, Registry 'virus_scan::engines' does not exist! 

Oct 1 04:07:23 monit[11037]: 'c-icap' start: '/usr/local/etc/c-icap/restart start' 

Oct 1 04:09:39 monit[11037]: 'c-icap' content doesn't match 

Cheers, Stephan

@fabianfrz
Copy link
Member

@mimugmail maybe patching this section may help:

spectacle p15107

The file is /usr/local/etc/rc.d/c-icap

@fichtner
Copy link
Member

fichtner commented Oct 1, 2017 via email

@fabianfrz
Copy link
Member

fabianfrz commented Oct 1, 2017

@fichtner This is not even the full issue for your information. If the ICAP server is down, squid will make troubles too (it will say that it cannot reach the ICAP server and blocks the connection (it will mark the server as down)).

The advantage here is that squid connects to the ICAP server as soon as it gets an HTTP request so this is a bit dangerous as a race condition too.

@fichtner
Copy link
Member

@Stephanowicz
Copy link

Hi, thank You - well, anything else that needs to be done - or just copy the file to rc.syshook.d?

Cheers, Stephan

@fichtner
Copy link
Member

Hi @Stephanowicz,

Yes, copy file, make sure to chmod to 755.

Cheers,
Franco

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Production bug
Development

No branches or pull requests

6 participants