SpamPD doesn't restart with correct configuration flags on HUP Signal

[sumdog]

Version: spampd-2.30p4
Operating System: OpenBSD 6.3 Generic amd64

If SpamPD received an HUP signal and was started with specific arguments (e.g. --relayhost or --port), it will respawn itself but not with those configuration flags.

This is a problem on OpenBSD because for some reason when the system boots, the rc system sends SpamPD and HUP, causing it to respawn with a different configuration than the one in /etc/rc.conf.local.

A workaround is to start SpamPD using @reboot via cron. I've documented this in more detail here:

https://penguindreams.org/blog/openbsd-spampd-and-the-startup-bug/

[mpaperno]

Hello, thanks for the report and workaround.

Could you verify this is still an issue with the latest release? And does adding --homedir parameter help?

Asking because v2.4+ introduced some possible fixes.

[mpaperno]

Hello, thanks for the report and workaround.

Could you verify this is still an issue with the latest release? And does adding --homedir parameter help?

Asking because v2.4+ introduced some possible fixes.

[trondd555]

I tried 2.51, and it still seems to have the issue.

Command line parameters:

"--port=10025 --relayhost=127.0.0.1:10027 --tagall --pid=/var/spampd/spampd.pid --homedir=/var/spampd

Sent HUP to one of the processes, and because it didn't get the pid parameter on restart, it failed to start trying to use the default pid file location.

Aug 16 11:24:13 ti24453-obsd spampd[10205]: 2018/08/16-11:24:13 Server closing! 
Aug 16 11:24:13 ti24453-obsd spampd[10205]: 2018/08/16-11:24:13 Re-exec server during HUP 
Aug 16 11:24:18 ti24453-obsd spampd[10205]: 2018/08/16-11:24:18 Couldn't open pid file "/var/run/spampd.pid" [Permission denied].    at line 179 in file /usr/local/libdata/perl5/site_perl/Net/Server.pm 
Aug 16 11:24:18 ti24453-obsd spampd[10205]: 2018/08/16-11:24:18 Server closing! 

[trondd555]

I tried 2.51, and it still seems to have the issue.

Command line parameters:

"--port=10025 --relayhost=127.0.0.1:10027 --tagall --pid=/var/spampd/spampd.pid --homedir=/var/spampd

Sent HUP to one of the processes, and because it didn't get the pid parameter on restart, it failed to start trying to use the default pid file location.

Aug 16 11:24:13 ti24453-obsd spampd[10205]: 2018/08/16-11:24:13 Server closing! 
Aug 16 11:24:13 ti24453-obsd spampd[10205]: 2018/08/16-11:24:13 Re-exec server during HUP 
Aug 16 11:24:18 ti24453-obsd spampd[10205]: 2018/08/16-11:24:18 Couldn't open pid file "/var/run/spampd.pid" [Permission denied].    at line 179 in file /usr/local/libdata/perl5/site_perl/Net/Server.pm 
Aug 16 11:24:18 ti24453-obsd spampd[10205]: 2018/08/16-11:24:18 Server closing! 

[mpaperno]

Thanks for testing.

I guess I'm somewhat vague on how exactly spampd script would get all those startup params back after a HUP? It does not store them anywhere but memory. Would it be solved by using a config file? From what I understand that is typically what a SIGHUP is used for these days, to get a daemon to reload its config.

The bug reports and discussions you linked to seemed to indicate an issue with the OS (Debian) which was fixed/changed in later releases. I'm not sure what we could change in spampd to fix this.

I noticed a recent new fork with a commit which essentially adds a config file... would this help? nicolasb827@1083bc4

[mpaperno]

Thanks for testing.

I guess I'm somewhat vague on how exactly spampd script would get all those startup params back after a HUP? It does not store them anywhere but memory. Would it be solved by using a config file? From what I understand that is typically what a SIGHUP is used for these days, to get a daemon to reload its config.

The bug reports and discussions you linked to seemed to indicate an issue with the OS (Debian) which was fixed/changed in later releases. I'm not sure what we could change in spampd to fix this.

I noticed a recent new fork with a commit which essentially adds a config file... would this help? nicolasb827@1083bc4

[trondd555]

I'm guessing spampd doesn't even know HUP is sent and Net::Server::PreForkSimple (or somewhere in Net::Server) is catching it and restarting itself. But since spampd set the parameters, spampd isn't part of the process to reset them on restart.

Maybe spampd needs to catch HUP and restart Net::Server properly itself. Or spampd needs to catch HUP and ignore it without letting it pass through to Net::Server. Or the fix may need to happen in Net::Server.

Or maybe a config file might solve the issue. I can try the fork.

This is actually happening on OpenBSD.

[trondd555]

I'm guessing spampd doesn't even know HUP is sent and Net::Server::PreForkSimple (or somewhere in Net::Server) is catching it and restarting itself. But since spampd set the parameters, spampd isn't part of the process to reset them on restart.

Maybe spampd needs to catch HUP and restart Net::Server properly itself. Or spampd needs to catch HUP and ignore it without letting it pass through to Net::Server. Or the fix may need to happen in Net::Server.

Or maybe a config file might solve the issue. I can try the fork.

This is actually happening on OpenBSD.

[mpaperno]

I think your guess is correct, and spampd does not in fact really handle any signals internally.

Besides the OS deciding (for some reason) to send a HUP to spampd, is there any other reason one would want to do that, instead of restarting the service? It would still be expected to kill any children/sockets immediately upon a HUP anyway (vs. finishing up any outstanding requests), right?

Yes, we're talking OpenBSD now, but sounded like a similar issue was fixed (?) in Debian. Why is the OS sending a HUP in the first place? (That's not an excuse for spampd to not "do the right thing," I'm just wondering how much time/effort should be spent on this.)

[mpaperno]

I think your guess is correct, and spampd does not in fact really handle any signals internally.

Besides the OS deciding (for some reason) to send a HUP to spampd, is there any other reason one would want to do that, instead of restarting the service? It would still be expected to kill any children/sockets immediately upon a HUP anyway (vs. finishing up any outstanding requests), right?

Yes, we're talking OpenBSD now, but sounded like a similar issue was fixed (?) in Debian. Why is the OS sending a HUP in the first place? (That's not an excuse for spampd to not "do the right thing," I'm just wondering how much time/effort should be spent on this.)

[sumdog]

In the specific case of OpenBSD, a configuration file would have mitigated the startup issue. All the other mail services I have running (opensmtpd, dovecot, clamav, etc.) all use configuration files:

https://github.com/sumdog/bee2/tree/master/ansible/roles/openbsd-email/templates

spampd is the only one that uses daemon flags. Of course the daemon flags shouldn't be an issue, except that as @trondd555 mentioned, signal handling happens somewhere in Net::Server and might be outside of a place where you can cleanly deal with it.

With most services, I think HUP is used to force a config reload. I know that's true of HAProxy, and I think it's true with nginx and others.

[sumdog]

In the specific case of OpenBSD, a configuration file would have mitigated the startup issue. All the other mail services I have running (opensmtpd, dovecot, clamav, etc.) all use configuration files:

https://github.com/sumdog/bee2/tree/master/ansible/roles/openbsd-email/templates

spampd is the only one that uses daemon flags. Of course the daemon flags shouldn't be an issue, except that as @trondd555 mentioned, signal handling happens somewhere in Net::Server and might be outside of a place where you can cleanly deal with it.

With most services, I think HUP is used to force a config reload. I know that's true of HAProxy, and I think it's true with nginx and others.

[trondd555]

I'm not sure why HUP is sent. Something in the init process, I think detaching from the rc scripts that would be starting spampd at boot.

Without a config file to reload, I am not sure why HUP would be needed. Other than an expectation by a user for it to cleanly restart itself like other servers that do use a config file.

[trondd555]

I'm not sure why HUP is sent. Something in the init process, I think detaching from the rc scripts that would be starting spampd at boot.

Without a config file to reload, I am not sure why HUP would be needed. Other than an expectation by a user for it to cleanly restart itself like other servers that do use a config file.

[mpaperno]

Other than an expectation by a user for it to cleanly restart itself like other servers that do use a config file.

Right, in some ways that is reason enough for sure. OTOH this is the first I've been made aware of it in 15 years... :)

I'm guessing then that just adding a config script like in that fork isn't going to be enough... if I understand it correctly, spampd itself would need to be aware of the config file and re-read it upon a HUP-induced reload.

[mpaperno]

Other than an expectation by a user for it to cleanly restart itself like other servers that do use a config file.

Right, in some ways that is reason enough for sure. OTOH this is the first I've been made aware of it in 15 years... :)

I'm guessing then that just adding a config script like in that fork isn't going to be enough... if I understand it correctly, spampd itself would need to be aware of the config file and re-read it upon a HUP-induced reload.

[trondd555]

OTOH this is the first I've been made aware of it in 15 years... :)

Laziness? I know it's been a problem in OpenBSD for years. Seems everyone just kind of accepted it. The port maintainer hasn't updated it since 2.30. They might have lost track of releases when they moved to GitHub? I don't know. I'll at least help get it updated to the latest release from all this.

Yeah, config file or not, I suppose spampd would have to capture the HUP signal and respond accordingly.

[trondd555]

OTOH this is the first I've been made aware of it in 15 years... :)

Laziness? I know it's been a problem in OpenBSD for years. Seems everyone just kind of accepted it. The port maintainer hasn't updated it since 2.30. They might have lost track of releases when they moved to GitHub? I don't know. I'll at least help get it updated to the latest release from all this.

Yeah, config file or not, I suppose spampd would have to capture the HUP signal and respond accordingly.

[trondd555]

So, poking around in Net::Server docs, there are a few possibly relevant things.

There are a couple of hooks that fire when HUP is called. restart_open_hook and restart_close_hook.
https://metacpan.org/pod/distribution/Net-Server/lib/Net/Server.pod#HOOKS

And there is a commandline method that maybe needs to be used.
https://metacpan.org/pod/distribution/Net-Server/lib/Net/Server.pod#RESTARTING

This is a bit out of my development ability. I've been hacking at it trying to make things happen but haven't gotten very far, yet. This may make more sense to you.

[trondd555]

So, poking around in Net::Server docs, there are a few possibly relevant things.

There are a couple of hooks that fire when HUP is called. restart_open_hook and restart_close_hook.
https://metacpan.org/pod/distribution/Net-Server/lib/Net/Server.pod#HOOKS

And there is a commandline method that maybe needs to be used.
https://metacpan.org/pod/distribution/Net-Server/lib/Net/Server.pod#RESTARTING

This is a bit out of my development ability. I've been hacking at it trying to make things happen but haven't gotten very far, yet. This may make more sense to you.

[mpaperno]

Thanks! I'm out for the weekend, but hopefully can have a look afterwards. If you make any progress, let me know! :)

[mpaperno]

Thanks! I'm out for the weekend, but hopefully can have a look afterwards. If you make any progress, let me know! :)

[mpaperno]

Please give #20 a test and see if it solves the issue.

[mpaperno]

Please give #20 a test and see if it solves the issue.