Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev.icinga.com #13567] SIGPIPE shutdown on config reload #4867

Closed
icinga-migration opened this issue Dec 14, 2016 · 47 comments
Closed

[dev.icinga.com #13567] SIGPIPE shutdown on config reload #4867

icinga-migration opened this issue Dec 14, 2016 · 47 comments
Milestone

Comments

@icinga-migration
Copy link
Member

@icinga-migration icinga-migration commented Dec 14, 2016

This issue has been migrated from Redmine: https://dev.icinga.com/issues/13567

Created by Markus on 2016-12-14 14:05:27 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2017-01-12 09:55:02 +00:00)
Target Version: 2.6.1
Last Update: 2017-01-12 09:55:38 +00:00 (in Redmine)

Icinga Version: 2.6.0-1
Backport?: Not yet backported
Include in Changelog: 1

Today we updated our icinga2 from v2.5.4 to the latest 2.6.0-1.

We have master <-> satellite config with one master, one satellite in a remote site and multiple clients connected.

On every config reload icinga2 does a shutdown but does not start up again. When we start icinga2 again everything works as it did with version 2.5.4. We have not made any config changes during the update.

The same happens when we do a restart (not reload!) after config changes on the satellite. Satellite becomes reload signal via API call and does a shutdown.

Any help is welcome.
Markus


#/etc/init.d/icinga2 checkconfig
[ ok ] checking Icinga2 configuration.
#
# /etc/init.d/icinga2 reload
[ ok ] checking Icinga2 configuration.
[ ok ] icinga2 is running.
[ ok ] Reloading icinga2 monitoring daemon: icinga2.

icinga.log on master

[2016-12-14 14:34:47 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 14:34:47 +0100] information/Application: Received request to shut down.
[2016-12-14 14:34:48 +0100] information/Application: Shutting down...
[2016-12-14 14:34:48 +0100] information/CheckerComponent: Checker stopped.

# /etc/init.d/icinga2 status
[FAIL] icinga2 is not running ... failed!

icinga.log on satellite

[2016-12-14 14:34:48 +0100] warning/TlsStream: TLS stream was disconnected.
[2016-12-14 14:34:48 +0100] warning/JsonRpcConnection: API client disconnected for identity 'zipsmon2001.siepha.net'
[2016-12-14 14:34:48 +0100] warning/ApiListener: Removing API client for endpoint 'zipsmon2001.siepha.net'. 0 API clients left.
[2016-12-14 14:34:58 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'zipsmon2001.siepha.net' via host 'zipsmon2001.siepha.net' and port '5665'
[2016-12-14 14:34:58 +0100] critical/TcpSocket: Invalid socket: Connection refused

icinga.log on satellite after config change

[2016-12-14 13:51:47 +0100] information/ApiListener: Restarting after configuration change.
[2016-12-14 13:51:47 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 13:51:48 +0100] information/Application: Received request to shut down.
[2016-12-14 13:51:48 +0100] information/Application: Shutting down...
[2016-12-14 13:51:48 +0100] information/CheckerComponent: Checker stopped.

Version informations of master system:

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.0-1)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 3.16.0-4-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: smithers

Version informations of satellite system:

icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.0-1)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 3.16.0-4-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: smithers

Attachments

Changesets

2016-12-15 10:47:07 +00:00 by mfriedrich fb8f410

Block SIGPIPE signal for check processes

refs #13567

2017-01-12 09:50:43 +00:00 by gbeutner 751ca67

Ignore SIGPIPE earlier in the start-up process

fixes #13567

Relations:

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by seferovic on 2016-12-14 15:02:28 +00:00

Hi... I can confirm and reproduce this behavior... Icinga Director "kills" my master instance on deployment (I deployed an UserGroup object :) )

[2016-12-14 15:50:34 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 15:50:34 +0100] information/Application: Received request to shut down.
[2016-12-14 15:50:35 +0100] information/Application: Shutting down...
[2016-12-14 15:50:35 +0100] information/CheckerComponent: Checker stopped.
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by mfrosch on 2016-12-14 15:42:22 +00:00

Whats the systemd output / status after that exit?

Icinga 2 is doing an restart internally, meaning a new process comes up and replaces the old process in sync.

The full log of one of mine servers is:

[2016-12-14 16:37:26 +0100] information/ApiListener: Restarting after configuration change.
[2016-12-14 16:37:28 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 16:37:28 +0100] information/Application: Received request to shut down.
[2016-12-14 16:37:29 +0100] information/Application: Shutting down...
[2016-12-14 16:37:29 +0100] information/CheckerComponent: Checker stopped.
[2016-12-14 16:37:29 +0100] information/LivestatusListener: Created UNIX socket in '/run/icinga2/cmd/livestatus'.
[2016-12-14 16:37:29 +0100] information/ConfigItem: Activated all objects.
[2016-12-14 16:38:26 +0100] information/ApiListener: New client connection for identity 'XXXXX' from XXXX

The re-connection could be faster possibly, but the service restarts properly...

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by mfrosch on 2016-12-14 15:42:30 +00:00

  • Status changed from New to Feedback
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by mfriedrich on 2016-12-14 15:48:35 +00:00

  • Category set to libbase
  • Assigned to set to Markus

Works on CentOS 7 with Systemd, as well as Debian Wheezy in a Docker container with SystemVinit too.

root@066ecc51d6aa:/# service icinga2 restart
[ ok ] checking Icinga2 configuration.
[....] Restarting icinga2 monitoring daemon: icinga2[2016-12-14 15:45:22 +0000] information/cli: Icinga application loader (version: v2.6.0)
[2016-12-14 15:45:22 +0000] information/cli: Loading configuration file(s).
[2016-12-14 15:45:22 +0000] information/ConfigItem: Committing config item(s).
[2016-12-14 15:45:22 +0000] warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 Zone.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 13 Notifications.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 201 CheckCommands.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 Host.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 User.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 12 Services.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2016-12-14 15:45:22 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2016-12-14 15:45:22 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2016-12-14 15:45:22 +0000] information/ConfigItem: Triggering Start signal for config items
[2016-12-14 15:45:22 +0000] information/ConfigItem: Committing config item(s).
[2016-12-14 15:45:22 +0000] information/ConfigItem: Activated all objects.
. ok
root@066ecc51d6aa:/# service icinga2 status
[ ok ] icinga2 is running.
root@066ecc51d6aa:/# service icinga2 reload
[ ok ] checking Icinga2 configuration.
[ ok ] icinga2 is running.
[ ok ] Reloading icinga2 monitoring daemon: icinga2.
root@066ecc51d6aa:/# service icinga2 status
[ ok ] icinga2 is running.

Any specific OS version/release you're using this on?

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by Markus on 2016-12-14 15:50:36 +00:00

I posted the last line of the log. There is no more icinga process running. It is like you would stop icinga manually.

Status of icinga is

# /etc/init.d/icinga2 status
[FAIL] icinga2 is not running ... failed!
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by mfriedrich on 2016-12-14 15:51:18 +00:00

Please add the entire output of "icinga2 --version" (both of you).

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by Markus on 2016-12-14 15:55:29 +00:00

Version infos are already in my first post.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by mfriedrich on 2016-12-14 15:58:49 +00:00

Ah, yes. So you are using SystemVinit instead of the Systemd default on Debian Jessie, right?

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by Markus on 2016-12-14 16:02:22 +00:00

Yes, sysvinit is used on both systems.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by mfrosch on 2016-12-14 16:09:19 +00:00

Tried it with the bare initscript on Jessie:

_SYSTEMCTL_SKIP_REDIRECT=true /etc/init.d/icinga2 start
_SYSTEMCTL_SKIP_REDIRECT=true /etc/init.d/icinga2 reload

The satellite's log ends with:

[2016-12-14 17:03:38 +0100] information/ApiListener: Restarting after configuration change.
[2016-12-14 17:03:39 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 17:03:39 +0100] information/Application: Received request to shut down.
[2016-12-14 17:03:40 +0100] information/Application: Shutting down...
[2016-12-14 17:03:40 +0100] information/CheckerComponent: Checker stopped.
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by mfrosch on 2016-12-14 16:10:58 +00:00

  • Status changed from Feedback to Assigned
  • Assigned to changed from Markus to mfrosch
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by rwaffen on 2016-12-14 16:17:32 +00:00

can confirm this on CentOS 6.8 ... with initV ... service icinga2 reload stops the icinga2 deamon

 rpm -qa| grep icinga
icinga2-2.6.0-1.el6.x86_64
icingaweb2-vendor-dompdf-0.6.2-1.el6.noarch
icingaweb2-common-2.3.4-1.el6.noarch
icinga2-common-2.6.0-1.el6.x86_64
icinga2-bin-2.6.0-1.el6.x86_64
icinga2-ido-mysql-2.6.0-1.el6.x86_64
icingaweb2-2.3.4-1.el6.noarch
icingaweb2-vendor-HTMLPurifier-4.7.0-1.el6.noarch
icingaweb2-vendor-Parsedown-1.0.0-1.el6.noarch
icingaweb2-vendor-JShrink-1.0.1-1.el6.noarch
icinga2-libs-2.6.0-1.el6.x86_64
icingaweb2-vendor-lessphp-0.4.0-1.el6.noarch

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.6.0)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /var/run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /var/run/icinga2/icinga2.pid

System information:
  Platform: CentOS
  Platform version: 6.8 (Final)
  Kernel: Linux
  Kernel version: 2.6.32-642.3.1.el6.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.2
  Build host: 4d3d57293387
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by mfrosch on 2016-12-14 16:55:14 +00:00

  • File added strace.log
  • Assigned to deleted mfrosch
  • Priority changed from Normal to High

To sum up the problem.

When the icinga2 daemon is started in foreground, or by sysvinit, and a reload is issued, icinga shuts down. And the new process does not take over.

Tested on Debian Jessie:

cat >> /etc/icinga2/icinga2.conf <&1 | tee strace.log &
sleep 5
kill -HUP $(cat /run/icinga2/icinga2.pid)

Interesting parts of the strace:

execve("/usr/sbin/icinga2", ["icinga2", "daemon"], [/* 25 vars */]) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/sbin/icinga2", O_RDONLY)     = 3
Process 17276 attached

# child
[pid 17275] execve("/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2", ["/usr/lib/x86_64-linux-gnu/icinga"..., "--no-stack-rlimit", "daemon"], [/* 25 vars */] 

# SIGHUP
[pid 17275] --- SIGHUP {si_signo=SIGHUP, si_code=SI_USER, si_pid=6359, si_uid=0} ---

Process 17336 attached
[pid 17336] execve("/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2", ["/usr/lib/x86_64-linux-gnu/icinga"..., "--no-stack-rlimit", "daemon", "--reload-internal", "17275"], [/* 26 vars */]) = 0

# new child killing old
[pid 17275] --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=17336, si_uid=109} ---

[pid 17275] open("/run/icinga2/icinga2.pid", O_RDWR) = 11
[pid 17275] open("/var/lib/icinga2/icinga2.state.jE9TzE", O_RDWR|O_CREAT|O_EXCL, 0600) = 11
[pid 17275] open("/var/lib/icinga2/icinga2.state.jE9TzE", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 20
[pid 17275] open("/var/lib/icinga2/modified-attributes.conf.CFMKZF", O_RDWR|O_CREAT|O_EXCL, 0600) = 11
[pid 17275] open("/var/lib/icinga2/modified-attributes.conf.CFMKZF", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 20

## NOT in sync with strace output
[2016-12-14 17:38:40 +0100] information/ConfigItem: Activated all objects.
[2016-12-14 17:38:46 +0100] information/Application: Got reload command: Starting new instance.
[2016-12-14 17:38:46 +0100] information/Application: Received request to shut down.
[2016-12-14 17:38:47 +0100] information/Application: Shutting down...
[2016-12-14 17:38:47 +0100] information/CheckerComponent: Checker stopped.
[2016-12-14 17:38:47 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2016-12-14 17:38:47 +0100] information/IcingaApplication: Icinga has shut down.

# all get killed by SIGPIPE
[pid 17373] open("/var/lib/icinga2/api/log/current", O_WRONLY|O_CREAT|O_APPEND, 0666) = 11
[pid 17373] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=17336, si_uid=109} ---
[pid 17373] +++ killed by SIGPIPE +++
[pid 17372] +++ killed by SIGPIPE +++
[pid 17371] +++ killed by SIGPIPE +++
[pid 17370] +++ killed by SIGPIPE +++
[pid 17369] +++ killed by SIGPIPE +++
[pid 17368] +++ killed by SIGPIPE +++
[pid 17367] +++ killed by SIGPIPE +++
[pid 17342] +++ killed by SIGPIPE +++
[pid 17341] +++ killed by SIGPIPE +++
[pid 17340] +++ killed by SIGPIPE +++
[pid 17339] +++ killed by SIGPIPE +++
[pid 17338] +++ killed by SIGPIPE +++
[pid 17337] +++ killed by SIGPIPE +++
[pid 17366] +++ killed by SIGPIPE +++
[pid 17336] +++ killed by SIGPIPE +++
+++ exited with 0 +++

This does not happen when:

  • on my master (also run in foreground)
  • started by systemd
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by seferovic on 2016-12-14 18:48:52 +00:00

Sorry for not posting the info before...

[root@master ~]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.6.0)

Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /var/run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /var/run/icinga2/icinga2.pid

System information:
  Platform: CentOS
  Platform version: 6.8 (Final)
  Kernel: Linux
  Kernel version: 2.6.32-642.11.1.el6.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.2
  Build host: 4d3d57293387

If you need anything else (mfrosch has a pretty good analysis) I am willing to provide feedback...

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 14, 2016

Updated by seferovic on 2016-12-14 23:34:08 +00:00

Interesting behavior.. if I clean out the conf.d directory and only leave api-users.conf and app.conf .. the reload is working as expected.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2016

Updated by mfriedrich on 2016-12-15 10:52:15 +00:00

  • Relates set to 8900
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2016

Updated by gbeutner on 2016-12-15 10:58:02 +00:00

The problem is that the writing side of the pipes that are passed to the new Icinga child process for stdout/stderr are closed once the old Icinga process dies. This problem has been around for a while, however we didn't notice this because we were ignoring SIGPIPE by default.

This is no longer the case - at least until after the config validation is finished. Only then do we reset SIGPIPE to SIG_IGN.

The result of this is that depending on the length of the log output for the config validation we might end up write()-ing to a pipe whose writing side has been closed. The SIGPIPE signal we get from that causes the new Icinga process to die.

There are two possible fixes:

  1. Ignore SIGPIPE in the child processes (quick & dirty).
  2. Don't use new pipe FDs for the Icinga child process and use the "real" stdout/stderr from the parent process instead. This would also solve the issue of missing journald output after a reload.
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2016

Updated by mfriedrich on 2016-12-15 11:07:07 +00:00

Systemd ignores SIGPIPE by default, that's probably the reason why you only experience that with SystemVinit or running in foreground.

https://lists.freedesktop.org/archives/systemd-devel/2014-August/021982.html

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2016

Updated by mfrosch on 2016-12-15 11:10:29 +00:00

Quick Fix for sysVinit

Please insert into initscript or default/sysconfig:

trap "" SIGPIPE
# or when not using bash for initscript (check shebang):
trap "" 13

This ignores SIGPIPE for all child processes spawned. And mimics the behavior of systemd.exec.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2016

Updated by mfriedrich on 2016-12-15 11:51:18 +00:00

The quick&dirty fix is available in git master & inside the snapshot packages (specifically Centos6 and Debian8 for you, tested CentOS7 over here).

You can also reproduce the issue using the "icinga2x-cluster" Vagrant box using an old snapshot. The current revision will not have this behaviour as it already pulls in the latest snapshot packages.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2016

Updated by Markus on 2016-12-15 20:27:34 +00:00

mfrosch wrote:

h1. Quick Fix for sysVinit

Please insert into initscript or default/sysconfig:

[...]

This ignores SIGPIPE for all child processes spawned. And mimics the behavior of systemd.exec.

I've tested the quick fix in initscript. Reload is working now.
Many thanks.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 16, 2016

Updated by mfriedrich on 2016-12-16 08:41:35 +00:00

  • Duplicated set to 13631
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 16, 2016

Updated by denny on 2016-12-16 10:20:08 +00:00

hi,

I have the same problem:

  • Debian Jessie with Sysvinit
  • Icinga2 2.6.0-1~debmon8+1

===
[...]
[2016-12-16 11:10:56 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2016-12-16 11:10:56 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2016-12-16 11:10:56 +0100] information/Application: Shutting down...
[2016-12-16 11:10:56 +0100] information/CheckerComponent: Checker stopped.

/var/log/icinga2/icinga2.log <
[...]
[2016-12-16 11:10:56 +0100] information/Application: Shutting down...
[2016-12-16 11:10:56 +0100] information/CheckerComponent: Checker stopped.
==

The problem starts with the latest 2.5.x version. I disabled feature by feature to see, when reload works again. Reload works, if I disable the api feature.

=
# icinga2 daemon -C
information/cli: Icinga application loader (version: r2.6.0-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: mon3.f00.net
information/ConfigItem: Instantiated 4 ApiUsers.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 12 Zones.
information/ConfigItem: Instantiated 2 FileLoggers.
information/ConfigItem: Instantiated 8 Endpoints.
information/ConfigItem: Instantiated 4 NotificationCommands.
information/ConfigItem: Instantiated 122 Notifications.
information/ConfigItem: Instantiated 209 CheckCommands.
information/ConfigItem: Instantiated 1 Downtime.
information/ConfigItem: Instantiated 9 Hosts.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 3 HostGroups.
information/ConfigItem: Instantiated 1 EventCommand.
information/ConfigItem: Instantiated 2 UserGroups.
information/ConfigItem: Instantiated 2 Users.
information/ConfigItem: Instantiated 130 Services.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 3 ServiceGroups.
information/ConfigItem: Instantiated 1 ScheduledDowntime.
information/ConfigItem: Instantiated 1 CompatLogger.
information/ConfigItem: Instantiated 1 StatusDataWriter.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 GraphiteWriter.
information/ConfigItem: Instantiated 1 InfluxdbWriter.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

cu denny

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 16, 2016

Updated by mfriedrich on 2016-12-16 10:21:45 +00:00

  • Status changed from Assigned to Feedback

Does the workaround work for you? I'd suggest switching to Systemd on Debian Jessie as well, but that probably results in a flame war.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 16, 2016

Updated by denny on 2016-12-16 10:37:14 +00:00

hi,

I changed the line "158"

trap 'status=2;' INT # handle intr here

with

trap "" SIGPIPE

It doesn't change the behavior. If i put it anywhere else:

===
trap: SIGPIPE: bad trap

===

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 17, 2016

Updated by MarcoKl on 2016-12-17 08:31:02 +00:00

Hello,
In my case, the following workaround works with Ubuntu 16.04.
Insert

trap "" 13

at the beginning of the icinga2 init script.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 18, 2016

Updated by denny on 2016-12-18 16:22:15 +00:00

hi,

thanks @marcokl works for me too :-)

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 19, 2016

Updated by rnekkanti on 2016-12-19 20:38:54 +00:00

Hi..,

I am running Ubuntu 14.04 with icinga2 2.6.0-1. I have tried updating the init script by adding trap "" 13 at the start of the init script and in line 158 but icinga2 does not seem to restart after deploying configs through director. It is just shutting down.

[2016-12-19 20:35:21 +0000] information/ApiListener: New client connection from [127.0.0.1]:60518 (no client certificate)
[2016-12-19 20:35:21 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:60518, user: root)
[2016-12-19 20:35:21 +0000] information/HttpServerConnection: Request: GET /v1/config/stages/director/ops-monitor-test-1-1482179721-0 (from [127.0.0.1]:60518, user: root)
[2016-12-19 20:35:21 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:60518, user: root)
[2016-12-19 20:35:22 +0000] information/Application: Got reload command: Starting new instance.
[2016-12-19 20:35:22 +0000] information/Application: Received request to shut down.
[2016-12-19 20:35:23 +0000] information/Application: Shutting down...
[2016-12-19 20:35:23 +0000] information/CheckerComponent: Checker stopped.
^C

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 21, 2016

Updated by tgelf on 2016-12-21 10:38:34 +00:00

  • Relates set to 13713
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 21, 2016

Updated by tgelf on 2016-12-21 10:38:41 +00:00

  • Relates deleted 13713
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 21, 2016

Updated by tgelf on 2016-12-21 10:38:47 +00:00

  • Duplicated set to 13713
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 21, 2016

Updated by spillerm on 2016-12-21 10:56:11 +00:00

Hi,

adding trap "" SIGPIPE at the beginning of init script solves it for me using Ubuntu 14.04.5 LTS.
Sorry for the duplicated bug report.

Cheers, Marianne

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 21, 2016

Updated by tgelf on 2016-12-21 10:57:37 +00:00

spillerm wrote:

adding trap "" SIGPIPE at the beginning of init script solves it for me using Ubuntu 14.04.5 LTS.

Good to hear that!

Sorry for the duplicated bug report.

Well... sorry for the bug.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 21, 2016

Updated by rnekkanti on 2016-12-21 17:23:56 +00:00

Adding trap "" SIGPIPE toinit script does not seem to work when deploying configs through director. Iicinga2 is still failing to restart.

[2016-12-21 17:19:26 +0000] information/ApiListener: New client connection from [127.0.0.1]:35286 (no client certificate)
[2016-12-21 17:19:26 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:35286, user: root)
[2016-12-21 17:19:26 +0000] information/HttpServerConnection: Request: DELETE /v1/config/stages/director/ops-monitor-test-1-1482184649-0 (from [127.0.0.1]:35286, user: root)
[2016-12-21 17:19:26 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:35286, user: root)
[2016-12-21 17:19:27 +0000] information/HttpServerConnection: Request: POST /v1/config/stages/director (from [127.0.0.1]:35286, user: root)
[2016-12-21 17:19:27 +0000] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ops-monitor-test-1-1482340767-0/zones.d/director-global/001-director-basics.conf
[2016-12-21 17:19:27 +0000] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ops-monitor-test-1-1482340767-0/zones.d/director-global/commands.conf
[2016-12-21 17:19:27 +0000] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ops-monitor-test-1-1482340767-0/zones.d/slave-emv/host_templates.conf
[2016-12-21 17:19:27 +0000] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ops-monitor-test-1-1482340767-0/zones.d/slave-emv/hosts.conf
[2016-12-21 17:19:27 +0000] information/ApiListener: New client connection from [127.0.0.1]:35287 (no client certificate)
[2016-12-21 17:19:27 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:35287, user: root)
[2016-12-21 17:19:27 +0000] information/HttpServerConnection: Request: GET /v1/config/stages/director/ops-monitor-test-1-1482340767-0 (from [127.0.0.1]:35287, user: root)
[2016-12-21 17:19:27 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [127.0.0.1]:35287, user: root)
[2016-12-21 17:19:28 +0000] information/Application: Got reload command: Starting new instance.
[2016-12-21 17:19:28 +0000] information/Application: Received request to shut down.
[2016-12-21 17:19:28 +0000] information/Application: Shutting down...
[2016-12-21 17:19:28 +0000] information/CheckerComponent: Checker stopped.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 22, 2016

Updated by Nagodar on 2016-12-22 08:18:30 +00:00

Same bug I reported here I think: https://dev.icinga.com/issues/13317

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 22, 2016

Updated by mfrosch on 2016-12-22 10:49:54 +00:00

  • Duplicated set to 13317
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 22, 2016

Updated by plarivee on 2016-12-22 16:38:27 +00:00

Had same issue, testing adding
sig "" 13
ubuntu 14.04

seems already more stable. Will report back later

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 22, 2016

Updated by rnekkanti on 2016-12-22 18:21:32 +00:00

sig "" 13 on top of the icinga2 init script seems to fix the issue.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 22, 2016

Updated by plarivee on 2016-12-22 19:21:53 +00:00

have been hamering for the last hour with multiple deploy and it did not stopped.

sig "" 13

seems to be the way in Ubuntu 14.04 for now.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 23, 2016

Updated by marcof on 2016-12-23 08:39:10 +00:00

We installed the 2.6.0-2 version on Ubuntu Trusty and Xenial and RHEL7, this fixed the issue. one strange thing though: "icinga2 --version" still shows 2.6.0-1 but the bug is gone.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 25, 2016

Updated by mfrosch on 2016-12-25 10:20:02 +00:00

marcof wrote:

We installed the 2.6.0-2 version on Ubuntu Trusty and Xenial and RHEL7, this fixed the issue. one strange thing though: "icinga2 --version" still shows 2.6.0-1 but the bug is gone.

icinga2 -version does not reflect the Debian version :)

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Jan 2, 2017

Updated by fugstrolch on 2017-01-02 11:44:48 +00:00

fyi: we have some centos5 Clients with Icinga2 V2.0.6.

Icinga2 -Process on these servers stopped when we've changed our Master's "global-templates"-config.

Adding

trap "" 13

seems to solve this problem.
We had to do this on all Icinga2-Centos-5-Clients.

#!/bin/sh
trap "" 13
#
# chkconfig: 35 90 12
# description: Icinga 2
...
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Jan 9, 2017

Updated by mfriedrich on 2017-01-09 15:20:48 +00:00

  • Target Version set to 2.6.1
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Jan 11, 2017

Updated by mfriedrich on 2017-01-11 16:43:19 +00:00

  • Subject changed from icinga2 does shutdown on config reload to SIGPIPE shutdown on config reload
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Jan 12, 2017

Updated by gbeutner on 2017-01-12 09:55:02 +00:00

  • Status changed from Feedback to Resolved
  • Done % changed from 0 to 100

Applied in changeset 751ca67.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Jan 12, 2017

Updated by mfriedrich on 2017-01-12 09:55:38 +00:00

  • Assigned to set to gbeutner
@Dark-o
Copy link

@Dark-o Dark-o commented May 5, 2017

Hi,
icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.3-1)

on Ubuntu 14.04.5 LTS

Since about a week I'm experiencing the same problem that is driving me crazy.

In the init script I've found:

# Block/ignore SIGPIPE inside Icinga2
# Workaround for a known bug in 2.6.0
# see https://dev.icinga.com/issues/13567#note-19
trap '' 13 # SIGPIPE

But the problem still there.
Any Suggestion?
thx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.