Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fabio is not handling SIGHUP (HUP) signal properly - it dies #400

Closed
sielaq opened this issue Dec 4, 2017 · 10 comments
Closed

Fabio is not handling SIGHUP (HUP) signal properly - it dies #400

sielaq opened this issue Dec 4, 2017 · 10 comments
Milestone

Comments

@sielaq
Copy link
Contributor

sielaq commented Dec 4, 2017

Usually HUP signal is being used to reload configuration (like in consul)
(for example can be used in systemd when reloading/upgrading package) .

Kindly please make fabio survive at least,
or in best scenario reload config when HUP signal is detected.

@magiconair
Copy link
Contributor

That is somewhat intentional. The architecture isn’t designed for hot reload since that’s something that shouldn’t happen often enough to need this. Which problem are you trying to solve?

@tino
Copy link
Contributor

tino commented Dec 4, 2017

"shouldn't happen often enough" still means => I need to find time when I can incur downtime (albeit short). I agree with sielaq that this is the "default" behaviour of most daemon processes.

I would for example use it when adding new listening ports for an TCP proxy implementation, FTP for example.

Of course this can be done staggered when multiple fabio instances are used, but being able to combine with nomad + templating and SIGHUP would make it also possible on a single instance.

@magiconair
Copy link
Contributor

I’m not saying it can’t happen but this is not a trivial change and if you need to do this only once every three months then this would have to wait. Especially, dynamic listeners is something I’d like to have but they should show up when a service registers the port and disappear with the last instance.

@magiconair
Copy link
Contributor

Also downtime is a couple of seconds

@aaronhurt
Copy link
Member

aaronhurt commented Dec 4, 2017

This is indeed not trivial and even load balancers that implement reload still have downtime. HAProxy is a perfect example where you can reload an entire configuration but there is much more to consider. What happens to existing connections if the route to them is deleted? What about websockets and other long-running connections that prevent a graceful shutdown and reload? That's just a few of the issues but those were enough to make us restart HAProxy vs reload anytime the configuration changed and live with a few possible dropped connections between the anycast instances.

True zero downtime reload isn't in the scope of the load balancer process itself. There is a good blog post from Yelp about how they got a real zero downtime reload working with os level connection queuing. The same should in theory work with fabio or any other network service.

@tino
Copy link
Contributor

tino commented Dec 4, 2017

Agreed

@sielaq
Copy link
Contributor Author

sielaq commented Dec 4, 2017

true, but then it shouldn't just die.
Rather informing [INFO] restart/reload has to be done in a controlled way. Application will not be reloaded.
and let fabio still running.Otherwise we have to overwrite the daemon's behavior in systemd for something hack'y like:

ExecReload=/bin/true

@magiconair
Copy link
Contributor

How often do you have to apply changes to the config that require a restart? If you do a restart with a graceful period of 1 sec fabio will have restarted within 2 sec. How much of an impact is that?

@sielaq
Copy link
Contributor Author

sielaq commented Dec 4, 2017

As I said, it is not always about changing config - it about upgrading fabio in a Debian/Ubuntu way.
Since we do Debian wrapper package for Fabio, each installation triggers (by default, not by us)
service reload - which makes Fabio die (since it doesn't handle with HUP signal).

As I said - we can overwrite systemd behavior - but better is to align all "perfect trinity" 🙏 behavior (consul, nomad, fabio) - Fabio is behaving differently here.

@magiconair
Copy link
Contributor

It could ignore the HUP signal. For now I suggest to change the systemd config

magiconair added a commit that referenced this issue Dec 10, 2017
SIGHUP is usually used to trigger a reload of the configuration which
is not supported by fabio. However, the process should not exit either.

Fixes #400
magiconair added a commit that referenced this issue Dec 10, 2017
SIGHUP is usually used to trigger a reload of the configuration which
is not supported. However, the process should not exit either.

Fixes #400
magiconair added a commit that referenced this issue Dec 10, 2017
SIGHUP is usually used to trigger a reload of the configuration which
is not supported. However, the process should not exit either.

Fixes #400
@magiconair magiconair added this to the 1.5.4 milestone Dec 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants