New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SpamPD doesn't restart with correct configuration flags on HUP Signal #19

Closed
sumdog opened this Issue Aug 15, 2018 · 12 comments

Comments

Projects
None yet
3 participants
@sumdog

sumdog commented Aug 15, 2018

Version: spampd-2.30p4
Operating System: OpenBSD 6.3 Generic amd64

If SpamPD received an HUP signal and was started with specific arguments (e.g. --relayhost or --port), it will respawn itself but not with those configuration flags.

This is a problem on OpenBSD because for some reason when the system boots, the rc system sends SpamPD and HUP, causing it to respawn with a different configuration than the one in /etc/rc.conf.local.

A workaround is to start SpamPD using @reboot via cron. I've documented this in more detail here:

https://penguindreams.org/blog/openbsd-spampd-and-the-startup-bug/

@mpaperno

This comment has been minimized.

Show comment
Hide comment
@mpaperno

mpaperno Aug 15, 2018

Owner

Hello, thanks for the report and workaround.

Could you verify this is still an issue with the latest release? And does adding --homedir parameter help?

Asking because v2.4+ introduced some possible fixes.

Owner

mpaperno commented Aug 15, 2018

Hello, thanks for the report and workaround.

Could you verify this is still an issue with the latest release? And does adding --homedir parameter help?

Asking because v2.4+ introduced some possible fixes.

@trondd555

This comment has been minimized.

Show comment
Hide comment
@trondd555

trondd555 Aug 16, 2018

I tried 2.51, and it still seems to have the issue.

Command line parameters:

"--port=10025 --relayhost=127.0.0.1:10027 --tagall --pid=/var/spampd/spampd.pid --homedir=/var/spampd

Sent HUP to one of the processes, and because it didn't get the pid parameter on restart, it failed to start trying to use the default pid file location.

Aug 16 11:24:13 ti24453-obsd spampd[10205]: 2018/08/16-11:24:13 Server closing! 
Aug 16 11:24:13 ti24453-obsd spampd[10205]: 2018/08/16-11:24:13 Re-exec server during HUP 
Aug 16 11:24:18 ti24453-obsd spampd[10205]: 2018/08/16-11:24:18 Couldn't open pid file "/var/run/spampd.pid" [Permission denied].    at line 179 in file /usr/local/libdata/perl5/site_perl/Net/Server.pm 
Aug 16 11:24:18 ti24453-obsd spampd[10205]: 2018/08/16-11:24:18 Server closing! 

trondd555 commented Aug 16, 2018

I tried 2.51, and it still seems to have the issue.

Command line parameters:

"--port=10025 --relayhost=127.0.0.1:10027 --tagall --pid=/var/spampd/spampd.pid --homedir=/var/spampd

Sent HUP to one of the processes, and because it didn't get the pid parameter on restart, it failed to start trying to use the default pid file location.

Aug 16 11:24:13 ti24453-obsd spampd[10205]: 2018/08/16-11:24:13 Server closing! 
Aug 16 11:24:13 ti24453-obsd spampd[10205]: 2018/08/16-11:24:13 Re-exec server during HUP 
Aug 16 11:24:18 ti24453-obsd spampd[10205]: 2018/08/16-11:24:18 Couldn't open pid file "/var/run/spampd.pid" [Permission denied].    at line 179 in file /usr/local/libdata/perl5/site_perl/Net/Server.pm 
Aug 16 11:24:18 ti24453-obsd spampd[10205]: 2018/08/16-11:24:18 Server closing! 
@mpaperno

This comment has been minimized.

Show comment
Hide comment
@mpaperno

mpaperno Aug 16, 2018

Owner

Thanks for testing.

I guess I'm somewhat vague on how exactly spampd script would get all those startup params back after a HUP? It does not store them anywhere but memory. Would it be solved by using a config file? From what I understand that is typically what a SIGHUP is used for these days, to get a daemon to reload its config.

The bug reports and discussions you linked to seemed to indicate an issue with the OS (Debian) which was fixed/changed in later releases. I'm not sure what we could change in spampd to fix this.

I noticed a recent new fork with a commit which essentially adds a config file... would this help? nicolasb827@1083bc4

Owner

mpaperno commented Aug 16, 2018

Thanks for testing.

I guess I'm somewhat vague on how exactly spampd script would get all those startup params back after a HUP? It does not store them anywhere but memory. Would it be solved by using a config file? From what I understand that is typically what a SIGHUP is used for these days, to get a daemon to reload its config.

The bug reports and discussions you linked to seemed to indicate an issue with the OS (Debian) which was fixed/changed in later releases. I'm not sure what we could change in spampd to fix this.

I noticed a recent new fork with a commit which essentially adds a config file... would this help? nicolasb827@1083bc4

@trondd555

This comment has been minimized.

Show comment
Hide comment
@trondd555

trondd555 Aug 16, 2018

I'm guessing spampd doesn't even know HUP is sent and Net::Server::PreForkSimple (or somewhere in Net::Server) is catching it and restarting itself. But since spampd set the parameters, spampd isn't part of the process to reset them on restart.

Maybe spampd needs to catch HUP and restart Net::Server properly itself. Or spampd needs to catch HUP and ignore it without letting it pass through to Net::Server. Or the fix may need to happen in Net::Server.

Or maybe a config file might solve the issue. I can try the fork.

This is actually happening on OpenBSD.

trondd555 commented Aug 16, 2018

I'm guessing spampd doesn't even know HUP is sent and Net::Server::PreForkSimple (or somewhere in Net::Server) is catching it and restarting itself. But since spampd set the parameters, spampd isn't part of the process to reset them on restart.

Maybe spampd needs to catch HUP and restart Net::Server properly itself. Or spampd needs to catch HUP and ignore it without letting it pass through to Net::Server. Or the fix may need to happen in Net::Server.

Or maybe a config file might solve the issue. I can try the fork.

This is actually happening on OpenBSD.

@mpaperno

This comment has been minimized.

Show comment
Hide comment
@mpaperno

mpaperno Aug 16, 2018

Owner

I think your guess is correct, and spampd does not in fact really handle any signals internally.

Besides the OS deciding (for some reason) to send a HUP to spampd, is there any other reason one would want to do that, instead of restarting the service? It would still be expected to kill any children/sockets immediately upon a HUP anyway (vs. finishing up any outstanding requests), right?

Yes, we're talking OpenBSD now, but sounded like a similar issue was fixed (?) in Debian. Why is the OS sending a HUP in the first place? (That's not an excuse for spampd to not "do the right thing," I'm just wondering how much time/effort should be spent on this.)

Owner

mpaperno commented Aug 16, 2018

I think your guess is correct, and spampd does not in fact really handle any signals internally.

Besides the OS deciding (for some reason) to send a HUP to spampd, is there any other reason one would want to do that, instead of restarting the service? It would still be expected to kill any children/sockets immediately upon a HUP anyway (vs. finishing up any outstanding requests), right?

Yes, we're talking OpenBSD now, but sounded like a similar issue was fixed (?) in Debian. Why is the OS sending a HUP in the first place? (That's not an excuse for spampd to not "do the right thing," I'm just wondering how much time/effort should be spent on this.)

@sumdog

This comment has been minimized.

Show comment
Hide comment
@sumdog

sumdog Aug 16, 2018

In the specific case of OpenBSD, a configuration file would have mitigated the startup issue. All the other mail services I have running (opensmtpd, dovecot, clamav, etc.) all use configuration files:

https://github.com/sumdog/bee2/tree/master/ansible/roles/openbsd-email/templates

spampd is the only one that uses daemon flags. Of course the daemon flags shouldn't be an issue, except that as @trondd555 mentioned, signal handling happens somewhere in Net::Server and might be outside of a place where you can cleanly deal with it.

With most services, I think HUP is used to force a config reload. I know that's true of HAProxy, and I think it's true with nginx and others.

sumdog commented Aug 16, 2018

In the specific case of OpenBSD, a configuration file would have mitigated the startup issue. All the other mail services I have running (opensmtpd, dovecot, clamav, etc.) all use configuration files:

https://github.com/sumdog/bee2/tree/master/ansible/roles/openbsd-email/templates

spampd is the only one that uses daemon flags. Of course the daemon flags shouldn't be an issue, except that as @trondd555 mentioned, signal handling happens somewhere in Net::Server and might be outside of a place where you can cleanly deal with it.

With most services, I think HUP is used to force a config reload. I know that's true of HAProxy, and I think it's true with nginx and others.

@trondd555

This comment has been minimized.

Show comment
Hide comment
@trondd555

trondd555 Aug 16, 2018

I'm not sure why HUP is sent. Something in the init process, I think detaching from the rc scripts that would be starting spampd at boot.

Without a config file to reload, I am not sure why HUP would be needed. Other than an expectation by a user for it to cleanly restart itself like other servers that do use a config file.

trondd555 commented Aug 16, 2018

I'm not sure why HUP is sent. Something in the init process, I think detaching from the rc scripts that would be starting spampd at boot.

Without a config file to reload, I am not sure why HUP would be needed. Other than an expectation by a user for it to cleanly restart itself like other servers that do use a config file.

@mpaperno

This comment has been minimized.

Show comment
Hide comment
@mpaperno

mpaperno Aug 16, 2018

Owner

Other than an expectation by a user for it to cleanly restart itself like other servers that do use a config file.

Right, in some ways that is reason enough for sure. OTOH this is the first I've been made aware of it in 15 years... :)

I'm guessing then that just adding a config script like in that fork isn't going to be enough... if I understand it correctly, spampd itself would need to be aware of the config file and re-read it upon a HUP-induced reload.

Owner

mpaperno commented Aug 16, 2018

Other than an expectation by a user for it to cleanly restart itself like other servers that do use a config file.

Right, in some ways that is reason enough for sure. OTOH this is the first I've been made aware of it in 15 years... :)

I'm guessing then that just adding a config script like in that fork isn't going to be enough... if I understand it correctly, spampd itself would need to be aware of the config file and re-read it upon a HUP-induced reload.

@trondd555

This comment has been minimized.

Show comment
Hide comment
@trondd555

trondd555 Aug 16, 2018

OTOH this is the first I've been made aware of it in 15 years... :)

Laziness? I know it's been a problem in OpenBSD for years. Seems everyone just kind of accepted it. The port maintainer hasn't updated it since 2.30. They might have lost track of releases when they moved to GitHub? I don't know. I'll at least help get it updated to the latest release from all this.

Yeah, config file or not, I suppose spampd would have to capture the HUP signal and respond accordingly.

trondd555 commented Aug 16, 2018

OTOH this is the first I've been made aware of it in 15 years... :)

Laziness? I know it's been a problem in OpenBSD for years. Seems everyone just kind of accepted it. The port maintainer hasn't updated it since 2.30. They might have lost track of releases when they moved to GitHub? I don't know. I'll at least help get it updated to the latest release from all this.

Yeah, config file or not, I suppose spampd would have to capture the HUP signal and respond accordingly.

@trondd555

This comment has been minimized.

Show comment
Hide comment
@trondd555

trondd555 Aug 17, 2018

So, poking around in Net::Server docs, there are a few possibly relevant things.

There are a couple of hooks that fire when HUP is called. restart_open_hook and restart_close_hook.
https://metacpan.org/pod/distribution/Net-Server/lib/Net/Server.pod#HOOKS

And there is a commandline method that maybe needs to be used.
https://metacpan.org/pod/distribution/Net-Server/lib/Net/Server.pod#RESTARTING

This is a bit out of my development ability. I've been hacking at it trying to make things happen but haven't gotten very far, yet. This may make more sense to you.

trondd555 commented Aug 17, 2018

So, poking around in Net::Server docs, there are a few possibly relevant things.

There are a couple of hooks that fire when HUP is called. restart_open_hook and restart_close_hook.
https://metacpan.org/pod/distribution/Net-Server/lib/Net/Server.pod#HOOKS

And there is a commandline method that maybe needs to be used.
https://metacpan.org/pod/distribution/Net-Server/lib/Net/Server.pod#RESTARTING

This is a bit out of my development ability. I've been hacking at it trying to make things happen but haven't gotten very far, yet. This may make more sense to you.

@mpaperno

This comment has been minimized.

Show comment
Hide comment
@mpaperno

mpaperno Aug 17, 2018

Owner

Thanks! I'm out for the weekend, but hopefully can have a look afterwards. If you make any progress, let me know! :)

Owner

mpaperno commented Aug 17, 2018

Thanks! I'm out for the weekend, but hopefully can have a look afterwards. If you make any progress, let me know! :)

@mpaperno

This comment has been minimized.

Show comment
Hide comment
@mpaperno

mpaperno Aug 21, 2018

Owner

Please give #20 a test and see if it solves the issue.

Owner

mpaperno commented Aug 21, 2018

Please give #20 a test and see if it solves the issue.

@mpaperno mpaperno closed this in 1e98ca9 Aug 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment