Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev.icinga.com #10226] Icinga2 reload timeout results in killing old and new process because of systemd #3440

Closed
icinga-migration opened this issue Sep 28, 2015 · 13 comments
Milestone

Comments

@icinga-migration
Copy link
Member

@icinga-migration icinga-migration commented Sep 28, 2015

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10226

Created by dgoetz on 2015-09-28 08:15:55 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2015-12-16 15:05:03 +00:00)
Target Version: 2.4.2
Last Update: 2016-02-23 09:58:15 +00:00 (in Redmine)

Icinga Version: 2.3.10
Backport?: Already backported
Include in Changelog: 1

The reload mechanismen send the old process a SIGKILL signal if it fails to stop before timeout.
The old process normally changes the pidfile to contain the new processes pid, this does not happen if killed.
So systemd kills all processes in the cgroup matching the old pid including the new process which results in no active icinga 2 process.

So effectively a reload can result in a stopped monitoring

Attachments

Changesets

2015-12-16 14:57:59 +00:00 by mfriedrich 8b6d335

Fix failed reload via systemd

fixes #10226

2016-02-23 08:16:01 +00:00 by mfriedrich 99efc7b

Fix failed reload via systemd

fixes #10226

Relations:

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Sep 28, 2015

Updated by mfriedrich on 2015-09-28 08:20:50 +00:00

  • Relates set to 9312
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Oct 26, 2015

Updated by mfriedrich on 2015-10-26 16:39:27 +00:00

  • Category set to libbase

Please try the suggested workaround in https://dev.icinga.org/issues/9312#note-8

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Oct 27, 2015

Updated by dgoetz on 2015-10-27 07:18:37 +00:00

Already tried setting the systemd timeouts for start and stop, but it still times out after 60 seconds which is the timeout of icinga we discussed.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Oct 28, 2015

Updated by mfriedrich on 2015-10-28 08:46:13 +00:00

Notes:

  • TerminateAndWaitForEnd()
  • Application::GetInstance()->Run();
  • UpdatePidFile
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Oct 28, 2015

Updated by mfriedrich on 2015-10-28 08:46:28 +00:00

  • Target Version set to Backlog
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Oct 29, 2015

Updated by mfriedrich on 2015-10-29 08:55:19 +00:00

  • Relates set to 7752
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Nov 3, 2015

Updated by mfriedrich on 2015-11-03 09:41:22 +00:00

  • Estimated Hours set to 32
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Nov 25, 2015

Updated by mfriedrich on 2015-11-25 16:17:53 +00:00

  • Priority changed from Normal to High
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 15, 2015

Updated by mfriedrich on 2015-12-15 16:57:30 +00:00

  • Target Version changed from Backlog to 2.4.2
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 16, 2015

Updated by mfriedrich on 2015-12-16 15:00:00 +00:00

  • Status changed from New to Assigned
  • Assigned to set to gbeutner
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 16, 2015

Updated by mfriedrich on 2015-12-16 15:05:03 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset 8b6d335.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 16, 2015

Updated by mfriedrich on 2015-12-16 15:47:32 +00:00

  • File added i2_10226_timeline.JPG

i2_10226_timeline.JPG

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 23, 2016

Updated by gbeutner on 2016-02-23 09:58:15 +00:00

  • Backport? changed from Not yet backported to Already backported
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.