Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev.icinga.com #10819] Cluster config sync ignores zones.d from API packages #3766

Closed
icinga-migration opened this issue Dec 10, 2015 · 42 comments
Closed

Comments

@icinga-migration
Copy link
Member

@icinga-migration icinga-migration commented Dec 10, 2015

This issue has been migrated from Redmine: https://dev.icinga.com/issues/10819

Created by tgelf on 2015-12-10 15:16:27 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2015-12-12 13:58:20 +00:00)
Target Version: 2.4.2
Last Update: 2016-02-23 09:58:24 +00:00 (in Redmine)

Icinga Version: 2.4.1
Backport?: Already backported
Include in Changelog: 1

I'm using the API to deploy config packages. They ship a few files in conf.d and also a few zone directories in zones.d: a global one, the master zone and a few endpoints. When deploying the configuration to a single master my config replication seems to completely ignore those zones.

Might be a config error, but I doubt so: we tried this in multiple environments, v2.4.0 and v2.4.1. dnsmichi was also able to reproduces this problem. Would be great if this could be fixed as it makes the whole feature hard to use for clustered environments.

Thanks,
Thomas

Attachments

Changesets

2015-12-10 21:21:29 +00:00 by mfriedrich c5b13ff

Fix that cluster config sync ignores zones.d directory from API config packages

fixes #10819

2015-12-11 16:03:07 +00:00 by mfriedrich d781c39

Add more debug logging for api package config sync

refs #10819

2015-12-12 11:16:00 +00:00 by mfriedrich 8055f05

Fix cluster config sync for (non-)authoritative configs

Details: https://dev.icinga.org/issues/10819#note-39

refs #10819

2015-12-18 10:53:56 +00:00 by mfriedrich 79899d7

Change log level for authoritative zone config includes

LogWarning will make users wonder about a problem each time
Icinga 2 starts.

refs #10819

2016-02-23 08:09:55 +00:00 by mfriedrich 1f5a216

Fix that cluster config sync ignores zones.d directory from API config packages

fixes #10819

2016-02-23 08:09:55 +00:00 by mfriedrich f0a1872

Add more debug logging for api package config sync

refs #10819

2016-02-23 08:09:55 +00:00 by mfriedrich 48fe703

Fix cluster config sync for (non-)authoritative configs

Details: https://dev.icinga.org/issues/10819#note-39

refs #10819

2016-02-23 08:09:55 +00:00 by mfriedrich 2f8d416

Change log level for authoritative zone config includes

LogWarning will make users wonder about a problem each time
Icinga 2 starts.

refs #10819

Relations:

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by mfriedrich on 2015-12-10 15:20:34 +00:00

  • Subject changed from Syncronization ignores API packages to Cluster config sync ignores zones.d from API packages

    curl -k -s -u root:icinga -H 'Accept: application/json' -X POST
    'https://localhost:5665/v1/config/packages/example-cmdb'

    $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST -d '{ "files": { "zones.d/mbmif.int.netways.de/test.conf": "object Host "cmdb-host" { check_command = "dummy" }" } }' 'https://localhost:5665/v1/config/stages/example-cmdb'
    {"results":[{"code":200.0,"package":"example-cmdb","stage":"mbmif.int.netways.de-1449759585-0","status":"Created stage."}]}

    mbmif /usr/local/icinga2/var/lib/icinga2/api/zones (master) # vim /usr/local/icinga2/var/lib/icinga2/api/packages/example-cmdb/mbmif.int.netways.de-1449759585-0/zones.d/mbmif.int.netways.de/test.conf

    object Host "cmdb-host" { check_command = "dummy" }

    mbmif /usr/local/icinga2/var/lib/icinga2/api/zones (master) # ls
    mbmif /usr/local/icinga2/var/lib/icinga2/api/zones (master) # cd ..
    mbmif /usr/local/icinga2/var/lib/icinga2/api (master) # cd packages/
    mbmif /usr/local/icinga2/var/lib/icinga2/api/packages (master) # ls
    _api example-cmdb
    mbmif /usr/local/icinga2/var/lib/icinga2/api/packages (master) # cd example-cmdb/
    mbmif /usr/local/icinga2/var/lib/icinga2/api/packages/example-cmdb (master) # ls
    active-stage include.conf mbmif.int.netways.de-1449759585-0
    active.conf mbmif.int.netways.de-1449759511-0
    mbmif /usr/local/icinga2/var/lib/icinga2/api/packages/example-cmdb (master) # grep -r include_zones .
    ./mbmif.int.netways.de-1449759511-0/include.conf: include_zones "example-cmdb", "zones.d"
    ./mbmif.int.netways.de-1449759585-0/include.conf: include_zones "example-cmdb", "zones.d"
    mbmif /usr/local/icinga2/var/lib/icinga2/api/packages/example-cmdb (master) # vim mbmif.int.netways.de-1449759511-0/include.conf

    mbmif /usr/local/icinga2 (master) # cd var/lib/
    mbmif /usr/local/icinga2/var/lib (master) # ls
    icinga2
    include "../active.conf"
    if (ActiveStages["example-cmdb"] == "mbmif.int.netways.de-1449759511-0") {
    include_recursive "conf.d"
    include_zones "example-cmdb", "zones.d"
    }

    mbmif /usr/local/icinga2/var/lib/icinga2/api/packages/example-cmdb (master) # ls -la /usr/local/icinga2/var/lib/icinga2/api/zones/
    total 0
    drwxr-xr-x 2 icinga staff 68 Sep 9 14:35 .
    drwxr-xr-x 6 icinga staff 204 Sep 30 11:29 ..

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by mfriedrich on 2015-12-10 15:24:37 +00:00

The problem originates from IsConfigMaster().

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by mfriedrich on 2015-12-10 15:25:09 +00:00

* thread #8: tid = 0x6dc1d, 0x00000001006df943 libremote.dylib`icinga::ApiListener::SyncZoneDirs(this=0x0000000102017600) const + 19 at apilistener-filesync.cpp:149, stop reason = breakpoint 1.1
    frame #0: 0x00000001006df943 libremote.dylib`icinga::ApiListener::SyncZoneDirs(this=0x0000000102017600) const + 19 at apilistener-filesync.cpp:149
   146
   147  void ApiListener::SyncZoneDirs(void) const
   148  {
-> 149      BOOST_FOREACH(const Zone::Ptr& zone, ConfigType::GetObjectsByType()) {
   150          if (!IsConfigMaster(zone))
   151              continue;
   152
(lldb) n
Process 24476 stopped
* thread #8: tid = 0x6dc1d, 0x00000001006dfb65 libremote.dylib`icinga::ApiListener::SyncZoneDirs(this=0x0000000102017600) const + 565 at apilistener-filesync.cpp:150, stop reason = step over
    frame #0: 0x00000001006dfb65 libremote.dylib`icinga::ApiListener::SyncZoneDirs(this=0x0000000102017600) const + 565 at apilistener-filesync.cpp:150
   147  void ApiListener::SyncZoneDirs(void) const
   148  {
   149      BOOST_FOREACH(const Zone::Ptr& zone, ConfigType::GetObjectsByType()) {
-> 150          if (!IsConfigMaster(zone))
   151              continue;
   152
   153          try {
(lldb) p zone
(icinga::Zone::Ptr) $9 = {
  px = 0x0000000101e7d250
}
(lldb) p *zone.px
(icinga::Zone) $10 = {
  m_Parent = {
    px = 0x0000000000000000
  }
}
(lldb) p *zone
error: call to a function 'boost::intrusive_ptr::operator*() const' ('_ZNK5boost13intrusive_ptrIN6icinga4ZoneEEdeEv') that is not present in the target
error: 0 errors parsing expression
error: The expression could not be prepared to run in the target
(lldb) l
   157          }
   158      }
   159  }
   160
   161  void ApiListener::SendConfigUpdate(const JsonRpcConnection::Ptr& aclient)
   162  {
   163      Endpoint::Ptr endpoint = aclient->GetEndpoint();
(lldb) n
Process 24476 stopped
* thread #8: tid = 0x6dc1d, 0x00000001006dfb8f libremote.dylib`icinga::ApiListener::SyncZoneDirs(this=0x0000000102017600) const + 607 at apilistener-filesync.cpp:151, stop reason = step over
    frame #0: 0x00000001006dfb8f libremote.dylib`icinga::ApiListener::SyncZoneDirs(this=0x0000000102017600) const + 607 at apilistener-filesync.cpp:151
   148  {
   149      BOOST_FOREACH(const Zone::Ptr& zone, ConfigType::GetObjectsByType()) {
   150          if (!IsConfigMaster(zone))
-> 151              continue;
   152
   153          try {
   154              SyncZoneDir(zone);
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by mfriedrich on 2015-12-10 21:23:59 +00:00

  • Status changed from New to Assigned
  • Assigned to set to mfriedrich
  • Target Version set to 2.4.2

Fixed

imagine /etc/icinga2 # curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/config/packages/example-cmdb'
{"results":[{"code":200.0,"status":"Created package."}]}i

imagine /etc/icinga2 # cat zones.conf 
object Endpoint NodeName {
}

object Zone ZoneName {
    endpoints = [ NodeName ]
}

imagine /etc/icinga2 # icinga2 variable get ZoneName
imagine

imagine /etc/icinga2 # curl -k -s -u root:icinga -H 'Accept: application/json' -X POST -d '{ "files": { "zones.d/imagine/test.conf": "object Host \"cmdb-host\" { check_command = \"dummy\" }" } }' 'https://localhost:5665/v1/config/stages/example-cmdb'
{"results":[{"code":200.0,"package":"example-cmdb","stage":"imagine-1449782278-1","status":"Created stage."}]}i

imagine /etc/icinga2 # ls -la /var/lib/icinga2/api/packages/example-cmdb/imagine-1449782278-1/
insgesamt 36K
drwx------. 4 icinga icinga 4,0K 10. Dez 22:17 ./
drwx------. 4 icinga icinga 4,0K 10. Dez 22:17 ../
drwx------. 2 icinga icinga 4,0K 10. Dez 22:17 conf.d/
-rw-r--r--. 1 icinga icinga  160 10. Dez 22:17 include.conf
-rw-r--r--. 1 icinga icinga  11K 10. Dez 22:17 startup.log
-rw-r--r--. 1 icinga icinga    1 10. Dez 22:17 status
drwx------. 3 icinga icinga 4,0K 10. Dez 22:17 zones.d/
imagine /etc/icinga2 # ls -la /var/lib/icinga2/api/packages/example-cmdb/imagine-1449782278-1/zones.d/
insgesamt 12K
drwx------. 3 icinga icinga 4,0K 10. Dez 22:17 ./
drwx------. 4 icinga icinga 4,0K 10. Dez 22:17 ../
drwxr-x---. 2 icinga icinga 4,0K 10. Dez 22:17 imagine/
imagine /etc/icinga2 # ls -la /var/lib/icinga2/api/packages/example-cmdb/imagine-1449782278-1/zones.d/imagine/
insgesamt 12K
drwxr-x---. 2 icinga icinga 4,0K 10. Dez 22:17 ./
drwx------. 3 icinga icinga 4,0K 10. Dez 22:17 ../
-rw-r--r--. 1 icinga icinga   51 10. Dez 22:17 test.conf

imagine /etc/icinga2 # ls -la /var/lib/icinga2/api/zones/
insgesamt 12K
drwxr-xr-x. 3 icinga icinga 4,0K 10. Dez 22:18 ./
drwxr-xr-x. 6 icinga icinga 4,0K  1. Sep 20:41 ../
drwx------. 3 icinga icinga 4,0K 10. Dez 22:18 imagine/
imagine /etc/icinga2 # ls -la /var/lib/icinga2/api/zones/imagine/
insgesamt 16K
drwx------. 3 icinga icinga 4,0K 10. Dez 22:18 ./
drwxr-xr-x. 3 icinga icinga 4,0K 10. Dez 22:18 ../
-rw-r--r--. 1 icinga icinga    0 10. Dez 22:18 .authoritative
drwxr-xr-x. 2 icinga icinga 4,0K 10. Dez 22:18 example-cmdb/
-rw-r--r--. 1 icinga icinga   17 10. Dez 22:18 .timestamp

imagine /etc/icinga2 # cat /var/lib/icinga2/api/zones/imagine/example-cmdb/test.conf 
object Host "cmdb-host" { check_command = "dummy" }i
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by mfriedrich on 2015-12-10 21:25:00 +00:00

  • Status changed from Assigned to Resolved
  • Done % changed from 0 to 100

Applied in changeset c5b13ff.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by tgelf on 2015-12-10 21:34:18 +00:00

Cool, thank's a lot! The patch mentions zone dirs, but I didn't completely understand the meaning of IsConfigMaster in this context. Is there anything one has to pay attention when using this, like avoiding to create specific directories in /etc? How behaves ApiListener::ConfigUpdateHandler in case former stages of the same package have been deployed to another master?

Simple use case: "config tool deploys config to the first available master". This must work regardless of whether it was the one that got the last deployment or not.

Cheers,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by tgelf on 2015-12-10 21:34:35 +00:00

  • Status changed from Resolved to Feedback
  • Priority changed from High to Normal
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by mfriedrich on 2015-12-10 21:43:23 +00:00

  • Status changed from Feedback to Resolved

Please test that fix and do not re-open the issue. I've tested it working with the recent changes and you should do the very same.

IsConfigMaster selects the authoritative node holding this zone configuration, previously it was only /etc/icinga2/zones.d being populated (cluster v3 2 years ago), nowadays we have a zone dirs registry where multiple sources may register their zone directories. That happens automatically, nothing the user would see nor have to care about. The fix takes care of checking that accordingly.
ConfigUpdateHandler() just checks whether a local version of the zone exists (etc-zones.d or api-zones.d) - e.g. a satellite holding its own zone configuration, disallowing updates from the master. That scenario is pretty rare, but was taken into account when designing an implementing the cluster a while ago.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by mfriedrich on 2015-12-10 21:43:38 +00:00

  • Priority changed from Normal to High
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by tgelf on 2015-12-10 22:24:48 +00:00

dnsmichi wrote:

Please test that fix and do not re-open the issue. I've tested it working with the recent changes and you should do the very same.

I didn't "re-open" this, just politely asked for feedback. Be sure that we will intensively test this, but first I need to have just a very few more details about WHAT behaviour I'm going to test.

IsConfigMaster selects the authoritative node holding this zone configuration, previously it was only /etc/icinga2/zones.d being populated (cluster v3 2 years ago), nowadays we have a zone dirs registry where multiple sources may register their zone directories. That happens automatically, nothing the user would see nor have to care about. The fix takes care of checking that accordingly.

Ok, so please allow me to dig deeper. Our history of handling polluted zone dirs (especially in /var/lib/icinga2/api) hasn't been free of issues, so one of the intentions of my question above was to find out how a simple two-node cluster behaves when an API user continues to deploy the very same zone to the very same package to both nodes, one at a time. Be assured that we'll test it as it makes part of normal operation. Deployments will usually point to the very same node, but may be directed to the second one in case the other one is down or not reachable. Before testing this I'd like to figure out whether the current design foresees this.

To me it seems that ApiListener::SyncZoneDirs is fine, but ApiListener::ConfigUpdateHandler looks suspicious. However, I'm neither a C** nor an Icinga2 core developer, so I'm just guessing - and that's why I'm asking for feedback. Please understand that to me this is an essential part of the whole replication system. Imagine a scenarios where node A got the latest deployment, cluster is in sync. For whatever reason A is not reachable by the config tool, so the next deployment goes to node B. B reloads, A reconnects and... what happens next? I would expect A to receive the newer zone from B. But is this really what has been implemented? A short yes (or no) regarding this question would be great and allow me to test whether everything works as designed.

ConfigUpdateHandler() just checks whether a local version of the zone exists (etc-zones.d or api-zones.d) - e.g. a satellite holding its own zone configuration, disallowing updates from the master.

That's where my questions start, as the expected behaviour isn't documented at all and didn't work until now I guess. But I mentioned that before. The only additional question that arises is how Icinga2 figures out which node owns a "newer" zone in case it consists of data from multiple sources, packages whatever. That's not so important for me, but it would help to better understand what's going on in case I encounter farther problems in my tests. And what means "holding" a zone configuration? Does this only refer to etc-zones.d (fine) or also api-zones.d as your comment suggests (error-prone, IMO).

That scenario is pretty rare, but was taken into account when designing an implementing the cluster a while ago.

If "that scenario" refers to having a local version of a zone in api-zones.d, the fact that ways too many people learned how to clean up that directory tells me that such scenarios haven't been as rare as you might think ;)

Please forgive me if I set this issue to "feedback" once again. A (very short) explanation of the expected behaviour in addition to that single-line fix should IMO make part of fixing this issue. In my believes this component should be well understood, as it broke our legs too many times in the past.

I'm not asking for extensive details. The most essential question I'd like to see answered is "Should a real-world work-flow as mentioned above work as expected?". If the answer is YES, I have no farther questions and will happily test your patch. Otherwise (or in any case, just to satisfy my curiosity) a few more details about where I should be prepared to expect eventual issues would be very helpful.

Thanks a lot for your time,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 10, 2015

Updated by tgelf on 2015-12-10 22:25:08 +00:00

  • Status changed from Resolved to Feedback
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 08:59:35 +00:00

  • Status changed from Feedback to Resolved

Please join us on Monday and we'll talk about that in real life. Thank you.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 09:14:27 +00:00

  • Status changed from Resolved to New

dnsmichi wrote:

Please join us on Monday and we'll talk about that in real life. Thank you.

What's so hard about writing down just a very few words explaining how this feature was designed to work? I have to reopen this completely, not just for feedback. We tried it out, it doesn't work. Nothing happens at all if a zone is deployed through the API:

[2015-12-11 09:38:53 +0100] notice/ApiListener: Ignoring sync for 'master'. Zone directory '/var/lib/icinga2/api/zones/master' does not exist.

If we manually create /etc/icinga2/zones.d/master the first part of the game works. But that makes this feature pretty useless, so I consider this issue not fixed. In the meantime we'll use this ugly workaround and try to figure out more details related the scenario mentioned before.

I have still no clue why you refuse to explain what the designed/expected behaviour should look like? IMO there is no need for a personal meeting for such a simple question.

Regards,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfrosch on 2015-12-11 09:27:25 +00:00

Usually config packages should only be deployed to one master, so that the internal replication can take care of it.

It might cause weird issues when you deploy to both.

Is the sync working when you deploy a config package with "zones.d/master" to the first master?

Maybe we should validate and talk about cleaning up stuff in another issue, currently /var/lib/icinga2/api/zones is mainly used for stuff from /etc/icinga2/zones.d - and syncs from this.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfrosch on 2015-12-11 09:41:06 +00:00

@tgelf: Did you tried the latest snapshot?

To correct my last comment: /var/lib/icinga2/api/zones is also used by packages, so that zone related stuff in packages is synced there.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 10:02:11 +00:00

  • Status changed from New to Feedback
  • Assigned to changed from mfriedrich to tgelf
  • Target Version deleted 2.4.2

Please post the output of

icinga2 --version
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 10:03:11 +00:00

lazyfrosch wrote:

@tgelf: Did you tried the latest snapshot?

Sure, wouldn't have made much sense otherwise. 2.4.1 - 1.snapshot201512102013.el7.centos

To correct my last comment: /var/lib/icinga2/api/zones is also used by packages, so that zone related stuff in packages is synced there.

Thank you, we know. The current snapshot fills /var/lib/icinga2/api/zones, but as mentioned before only if the corresponding zones exist in /etc/icinga2/zones.d. Zones deployed to packages but not locally configured to the master are ignored. Icinga2 forces me to create 1000 endpoint zones for 1000 agents, so manually creating their directories makes this... well, a little bit "less exciting".

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 10:03:46 +00:00

dnsmichi wrote:

Please post the output of

[...]

v2.4.1-63-g15ca998

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 10:09:41 +00:00

tgelf wrote:

dnsmichi wrote:
> Please post the output of
>
> [...]

v2.4.1-63-g15ca998

That's the version without the proposed fix.

commit 15ca9987faf5cc3fcb418a684d52909c1c128322
Author: Gunnar Beutner 
Date:   Thu Dec 10 16:54:43 2015 +0100

    Implement support for priorities in the WorkQueue class

    fixes #8714

The fixed version looks like this:

michi@mbmif ~/coding/icinga/icinga2 (master) $ icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.4.1-64-gc5b13ff; debug)

Copyright (c) 2012-2015 Icinga Development Team (https://www.icinga.org)
License GPLv2+: GNU GPL version 2 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr/local/icinga2
  Sysconf directory: /usr/local/icinga2/etc
  Run directory: /usr/local/icinga2/var/run
  Local state directory: /usr/local/icinga2/var
  Package data directory: /usr/local/icinga2/share/icinga2
  State path: /usr/local/icinga2/var/lib/icinga2/icinga2.state
  Modified attributes path: /usr/local/icinga2/var/lib/icinga2/modified-attributes.conf
  Objects path: /usr/local/icinga2/var/cache/icinga2/icinga2.debug
  Vars path: /usr/local/icinga2/var/cache/icinga2/icinga2.vars
  PID path: /usr/local/icinga2/var/run/icinga2/icinga2.pid

System information:
  Platform: Mac OS X
  Platform version: 10.11.2
  Kernel: Darwin
  Kernel version: 15.2.0
  Architecture: x86_64
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 10:14:03 +00:00

dnsmichi wrote:

tgelf wrote:
> dnsmichi wrote:
> > Please post the output of
> >
> > [...]
>
> v2.4.1-63-g15ca998

That's the version without the proposed fix.

Installed RPM:

  • icinga2-2.4.1-1.snapshot201512102013.el7.centos.x86_64

Latest package on packages.icinga.org:

  • icinga2-2.4.1-1.snapshot201512102013.el7.centos.x86_64.rpm 10-Dec-2015 21:18 9.2K

I'll happily test a newer one.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 10:24:12 +00:00

lazyfrosch wrote:

Usually config packages should only be deployed to one master, so that the internal replication can take care of it.

That's what we do, we always deploy to the very same master. But of course an outage of that master must not stop me from continuing my deployments to the next master.

It might cause weird issues when you deploy to both.

That's what I expected and that's why I ask for clarification. It's hard to accept that it seems to be impossible to get an explanation of how such a vital component was designed to behave.

Is the sync working when you deploy a config package with "zones.d/master" to the first master?

Only under special preconditions. I collected all variants with matching log lines, but will wait for the next snapshot build right now.

Maybe we should validate and talk about cleaning up stuff in another issue, currently /var/lib/icinga2/api/zones is mainly used for stuff from /etc/icinga2/zones.d - and syncs from this.

That's not what I've been told yesterday, but honestly - as a user I should not care. If someone manages it to come up with an explanation of how it was designed to work we could figure out which parts work, which don't and where tweaking the architecture might make sense.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 10:46:29 +00:00

tgelf wrote:

dnsmichi wrote:
> tgelf wrote:
> > dnsmichi wrote:
> > > Please post the output of
> > >
> > > [...]
> >
> > v2.4.1-63-g15ca998
>
> That's the version without the proposed fix.

Installed RPM:
* icinga2-2.4.1-1.snapshot201512102013.el7.centos.x86_64

Latest package on packages.icinga.org:
* icinga2-2.4.1-1.snapshot201512102013.el7.centos.x86_64.rpm 10-Dec-2015 21:18 9.2K

I'll happily test a newer one.

There's an issue on the build server with cmake & ccache (similar to #10823). I've triggered a manual snapshot build which is available at http://packages.icinga.org/epel/7/snapshot/x86\_64/

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 11:04:05 +00:00

  • Relates set to 10825
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 11:17:04 +00:00

dnsmichi wrote:

There's an issue on the build server with cmake & ccache (similar to #10823). I've triggered a manual snapshot build which is available at http://packages.icinga.org/epel/7/snapshot/x86\_64/

Thank you, I'm running v2.4.1-64-gc5b13ff right now. Master 1 log after config dump related reload:

information/ApiListener: New client connection for identity 'master2'
information/ApiListener: Sending updates for endpoint 'master2'.
information/ApiListener: Syncing zone 'agent1' to endpoint 'master2'.
information/ApiListener: Syncing zone 'master' to endpoint 'master2'.
information/ApiListener: Syncing global zone 'director-global' to endpoint 'master2'.
information/ApiListener: Syncing runtime objects to endpoint 'master2'.
information/ApiListener: Finished sending updates for endpoint 'master2'.
warning/ApiListener: Ignoring config update for zone 'master' because we have an authoritative version of the zone's config.

Same for the second master (log order seems a little bit strange, so try to not confuse agent/master information):

information/ApiListener: New client connection for identity 'master1'
information/ApiListener: Sending updates for endpoint 'master1'.
information/ApiListener: Syncing zone 'master' to endpoint 'master1'.
information/ApiListener: Syncing runtime objects to endpoint 'master1'.
information/ApiListener: Finished sending updates for endpoint 'master1'.
information/ApiListener: New client connection for identity 'agent1'
warning/ApiListener: Ignoring config update for unknown zone 'director-global'.
warning/ApiListener: Ignoring config update for zone 'master' because we have an authoritative version of the zone's config.
warning/ApiListener: Ignoring config update for unknown zone 'agent1'.

/var/lib/icinga2/api/packages/director/master1-1449831415-0/zones.d/ on master1:

├── director-global
│   ├── users.conf
│   └── zones.conf
├── master
│   ├── hosts.conf
│   └── host_templates.conf
└── agent1
    └── endpoints.conf

Please do not try to understand what I have in agent1/endpoints.conf - this is one of the many attempts we made to get this working. Prio 1 for me right now is getting files shipped in a reasonable way.

/var/lib/icinga2/zones on master1:

├── director-global
│   └── director
│       ├── users.conf
│       └── zones.conf
├── master
│   └── director
│       ├── hosts.conf
│       └── host_templates.conf
└── agent1
    └── director
        └── endpoints.conf

/var/lib/icinga2/zones on master2:

└── master
    ├── director
    └── _etc
        └── director
            ├── hosts.conf
            └── host_templates.conf

Both masters are member of the master zone, they didn't know anything about director-global (a global zone) and agent1 (a non-global zone). They work on master1, but are not synchronized. As logs show they are sent by master1 but not accepted on master2 - exactly as expected yesterday evening when trying to understand ApiListener::ConfigUpdateHandler. I guess it fails here:

    BOOST_FOREACH(const Dictionary::Pair& kv, update) {
        Zone::Ptr zone = Zone::GetByName(kv.first);

        if (!zone) {
            Log(LogWarning, "ApiListener")
                << "Ignoring config update for unknown zone '" << kv.first << "'.";
            continue;

Cheers,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 11:36:45 +00:00

There's only one configuration master allowed inside the zone. Your log indicates that the second master has its own local configuration for the "master" zone:

Same for the second master (log order seems a little bit strange, so try to not confuse agent/master information):

information/ApiListener: New client connection for identity 'master1'
information/ApiListener: Sending updates for endpoint 'master1'.
information/ApiListener: Syncing zone 'master' to endpoint 'master1'.
information/ApiListener: Syncing runtime objects to endpoint 'master1'.
information/ApiListener: Finished sending updates for endpoint 'master1'.
information/ApiListener: New client connection for identity 'agent1'
warning/ApiListener: Ignoring config update for unknown zone 'director-global'.
warning/ApiListener: Ignoring config update for zone 'master' because we have an authoritative version of the zone's config.
warning/ApiListener: Ignoring config update for unknown zone 'agent1'.

It ignores the update sent from master1. Remove the zone configuration from inside zones.d (be it etc or api config packages) from this node and it will start to work. Having multiple configuration masters inside a zone is not supported.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 12:01:33 +00:00

dnsmichi wrote:

There's only one configuration master allowed inside the zone. Your log indicates that the second master has its own local configuration for the "master" zone:

Well, it has it's own Zone-Definition itself, sure - otherwise it would hardly work. But none of the nodes has /etc/icinga2/zones.d/master.

It ignores the update sent from master1. Remove the zone configuration from inside zones.d (be it etc or api config packages) from this node and it will start to work. Having multiple configuration masters inside a zone is not supported.

My major problem is that it completely ignores all the zones it DOESN'T know about. The "master" zone seems to have worked at least once, but you're right, logs are telling something different. In my believes also that part (deploying elements to master zone) somehow MUST work in such a scenario. How should one otherwise deploy hosts that the master zone should check?

Cheers,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 12:22:15 +00:00

tgelf wrote:

dnsmichi wrote:
> There's only one configuration master allowed inside the zone. Your log indicates that the second master has its own local configuration for the "master" zone:

Well, it has it's own Zone-Definition itself, sure - otherwise it would hardly work. But none of the nodes has /etc/icinga2/zones.d/master.

Both have /var/lib/icinga2/api/packages/****/zones.d/master somewhere. Otherwise the config update wouldn't be ignored.

> It ignores the update sent from master1. Remove the zone configuration from inside zones.d (be it etc or api config packages) from this node and it will start to work. Having multiple configuration masters inside a zone is not supported.

My major problem is that it completely ignores all the zones it DOESN'T know about. The "master" zone seems to have worked at least once, but you're right, logs are telling something different. In my believes also that part (deploying elements to master zone) somehow MUST work in such a scenario. How should one otherwise deploy hosts that the master zone should check?

The Endpoint and Zone object configuration must be in place before pushing configuration packages for zones. That's a matter of trust relationships and knowing which zone is capable of the configuration sync and which is not allowed (e.g. "master" is configured inside zones.conf, but "foo" is not - deploy a config package for the zone "foo", it does not get synced for obvious reasons).

Though I think this problem is far away from the original issue which obviously has been fixed. Please go for a new issue, or join a real life conversation about the problems you'll see with the current cluster trust model vs api packages.

Kind regards,
Michael

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 13:04:51 +00:00

dnsmichi wrote:

Both have /var/lib/icinga2/api/packages/****/zones.d/master somewhere. Otherwise the config update wouldn't be ignored.

No, sorry. Master2 didn't have and still hasn't. Any other ideas, something else I could check?

The Endpoint and Zone object configuration must be in place before pushing configuration packages for zones. That's a matter of trust relationships and knowing which zone is capable of the configuration sync and which is not allowed (e.g. "master" is configured inside zones.conf, but "foo" is not - deploy a config package for the zone "foo", it does not get synced for obvious reasons).

It's vital for the whole construct that the API allows me to deploy additional zones and endpoints. However, thank you for your comment - it leads me to one more possible way of how to solve this part of the issue. I'll give it a try and report.

Though I think this problem is far away from the original issue which obviously has been fixed.

Not really, please see above.

Thanks,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 13:58:32 +00:00

Ok, I guess I reproducible figured out one part of the problem. Starting with an empty master2:

  • master zone is synched from master1
  • other zones are refused
  • master2 reloads
  • director-global zone is synched from master1 as it has been declared in the master zone
  • agent1 zone is synched from master1 as it has been declared in the master zone
  • master zone is refused (!!) as master2 believes to be authoritative for it

Some related logs:

information/ApiListener: New client connection for identity 'master1'
information/ApiListener: Sending updates for endpoint 'master1'.
information/ApiListener: Syncing runtime objects to endpoint 'master1'.
information/ApiListener: Finished sending updates for endpoint 'master1'.

Now it ignores the director-global zone:

warning/ApiListener: Ignoring config update for unknown zone 'director-global'.

Unfortunately we do not see that it accepted the master zone - but we see that it updated some files:

information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/001-director-zone.conf.conf
information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/endpoints.conf
information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/host_templates.conf
information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/hosts.conf
information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/services.conf
information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/zones.conf

It ignores the agent1 zone and reloads:

warning/ApiListener: Ignoring config update for unknown zone 'agent1'.
information/ApiListener: Restarting after configuration change.

Right now it accepted the master zone, but refused the other zones defined in master/director/zones.conf. After the restart, it goes on:

information/ApiListener: New client connection for identity 'master1'
information/ApiListener: Sending updates for endpoint 'master1'.
information/ApiListener: Syncing zone 'master' to endpoint 'master1'.
information/ApiListener: Syncing runtime objects to endpoint 'master1'.
information/ApiListener: Finished sending updates for endpoint 'master1'.
information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/users.conf

This is my major problem, it now believes to be authoritative for master and will refuse all future updates:

warning/ApiListener: Ignoring config update for zone 'master' because we have an authoritative version of the zone's config.

However, it now accepted our agent1 zone:

information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/agent1//director/endpoints.conf
information/ApiListener: Restarting after configuration change.

And it happily connects to agent1 and ships it all the global zones:

information/ApiListener: New client connection for identity 'agent1'
information/ApiListener: Sending updates for endpoint 'agent1'.
information/ApiListener: Syncing global zone 'director-global' to endpoint 'agent1'.
information/ApiListener: Syncing zone 'agent1' to endpoint 'agent1'.
information/ApiListener: Syncing runtime objects to endpoint 'agent1'.
information/ApiListener: Finished sending updates for endpoint 'agent1'.

Things go worse after that - but it doesn't look that bad. What we need to fix first of all is the fact that it accepts only the very first sync for a specific zone.

Cheers,
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 14:04:22 +00:00

One more hint, please have another look at how /var/lib/icinga2/api/zones looked on both nodes in comment #24 - is it correct that master2 places the files it got via config sync from master1 in _etc/director? It also created the .authoritative file - it didn't do so for the other zones. They are written to api/zones/director and not to api/zones/_etc/director.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 14:17:47 +00:00

Please show the tree on both master1 and master2:

tree /var/lib/icinga2/api/packages/director/
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 14:34:56 +00:00

I did already mention that it was emtpy (and never existed during this tests) on master2:

# master2:~  # tree /var/lib/icinga2/api/packages/director/
/var/lib/icinga2/api/packages/director/ [error opening dir]
0 directories, 0 files

And it is filled on master1:

/var/lib/icinga2/api/packages/director/
├── active.conf
├── active-stage
├── include.conf
├── master1-1449831415-0
│   ├── conf.d
│   │   └── 001-director-basics.conf
│   ├── include.conf
│   ├── startup.log
│   ├── status
│   └── zones.d
│       ├── director-global
│       │   ├── users.conf
│       │   └── zones.conf
│       ├── master
│       │   ├── hosts.conf
│       │   └── host_templates.conf
│       └── agent1
│           └── endpoints.conf
├── master1-1449839303-0
 ... (it has three inactive stages right now)
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 14:37:36 +00:00

Btw, that was one of the inactive stages of course, we are testing various config variants. The active one looks slightly different:

└── master1-1449841262-0
    ├── conf.d
    │   └── 001-director-basics.conf
    ├── include.conf
    ├── startup.log
    ├── status
    └── zones.d
        ├── director-global
        │   └── users.conf
        └── master
            ├── 001-director-zone.conf
            ├── endpoints.conf
            ├── hosts.conf
            ├── host_templates.conf
            ├── services.conf
            └── zones.conf

But I guess your question targeted only master2, you didn't believe it was empty ;)

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 16:01:05 +00:00

I've tested this for a while now in my 2 instance cluster, and it works like expected.

Tests

Fresh start, clearing all data:

mbmif /usr/local/tests/icinga2/master-slave (master) # rm -rf icinga2{a,b}/lib/icinga2/api/zones/master/ icinga2a/lib/icinga2/api/packages/example-cmdb/

Start node1, icinga2a

Start node2, icinga2b

Create a package and push config

michi@mbmif ~ $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST 'https://localhost:7000/v1/config/packages/example-cmdb'{"results":[{"code":200.0,"status":"Created package."}]}


michi@mbmif ~ $ curl -k -s -u root:icinga -H 'Accept: application/json' -X POST -d '{ "files": { "zones.d/master/test.conf": "object Host \"cmdb-host\" { check_command = \"dummy\" }" } }' 'https://localhost:7000/v1/config/stages/example-cmdb'
{"results":[{"code":200.0,"package":"example-cmdb","stage":"mbmif.int.netways.de-1449846966-0","status":"Created stage."}]}

icinga2a

mbmif /usr/local/tests/icinga2/master-slave (master) # ./run icinga2a
[2015-12-11 16:15:34 +0100] information/cli: Icinga application loader (version: v2.4.1-66-g138d416; debug)
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a.conf
[2015-12-11 16:15:34 +0100] information/Utility: Loading library 'libchecker.dylib'
[2015-12-11 16:15:34 +0100] information/Utility: Loading library 'libcompat.dylib'
[2015-12-11 16:15:34 +0100] information/Utility: Loading library 'libdb_ido_mysql.dylib'
[2015-12-11 16:15:34 +0100] information/Utility: Loading library 'liblivestatus.dylib'
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/itl
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command.conf
[2015-12-11 16:15:34 +0100] information/Utility: Loading library 'libmethods.dylib'
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command-icinga.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/timeperiod.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command-plugins.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/include.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/include.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/../active.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google.com10.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google.com11.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google.com2.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google.com3.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google.com6.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google.com7.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google.dns.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google.zone.conf
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442332428-0/conf.d/hosts/google1.conf
[2015-12-11 16:15:34 +0100] information/ConfigItem: Committing config items
[2015-12-11 16:15:34 +0100] information/ApiListener: My API identity: icinga2a
[2015-12-11 16:15:34 +0100] warning/ApplyRule: Apply rule 'icinga' (in icinga2a.conf: 114:1-114:22) for type 'Service' does not match anywhere!
[2015-12-11 16:15:34 +0100] warning/ApplyRule: Apply rule 'remote-client' (in icinga2a.conf: 124:1-124:29) for type 'Service' does not match anywhere!
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 3 Endpoints.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 3 Zones.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 4 Services.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 11 Hosts.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 54 CheckCommands.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Instantiated 1 LivestatusListener.
[2015-12-11 16:15:34 +0100] information/ScriptGlobal: Dumping variables to file 'icinga2a/cache/icinga2/icinga2.vars'
[2015-12-11 16:15:34 +0100] information/ConfigObject: Restoring program state from file 'icinga2a/lib/icinga2/icinga2.state'
[2015-12-11 16:15:34 +0100] information/ConfigObject: Restored 84 objects. Loaded 0 new objects without state.
[2015-12-11 16:15:34 +0100] information/ConfigItem: Triggering Start signal for config items
[2015-12-11 16:15:34 +0100] information/ApiListener: Checking zone 'satellite': 1 registered config dirs.
[2015-12-11 16:15:34 +0100] information/ApiListener: Config path: 'icinga2a/etc/icinga2/zones.d/satellite'.
[2015-12-11 16:15:34 +0100] information/ApiListener: Copying zone configuration files for zone 'satellite' to  'icinga2a/lib/icinga2/api/zones/satellite'.
[2015-12-11 16:15:34 +0100] information/ApiListener: Checking zone 'master': 1 registered config dirs.
[2015-12-11 16:15:34 +0100] information/ApiListener: Config path: 'icinga2a/etc/icinga2/zones.d/master'.
[2015-12-11 16:15:34 +0100] information/ApiListener: Copying zone configuration files for zone 'master' to  'icinga2a/lib/icinga2/api/zones/master'.
[2015-12-11 16:15:34 +0100] information/ApiListener: Checking zone 'global-templates': 1 registered config dirs.
[2015-12-11 16:15:34 +0100] information/ApiListener: Config path: 'icinga2a/etc/icinga2/zones.d/global-templates'.
[2015-12-11 16:15:34 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2015-12-11 16:15:34 +0100] information/LivestatusListener: Created UNIX socket in 'icinga2a/run/icinga2/cmd/livestatus'.
[2015-12-11 16:15:34 +0100] information/ApiListener: Copying zone configuration files for zone 'global-templates' to  'icinga2a/lib/icinga2/api/zones/global-templates'.
[2015-12-11 16:15:34 +0100] information/ApiListener: Adding new listener on port '7000'
[2015-12-11 16:15:34 +0100] information/ConfigItem: Activated all objects.
[2015-12-11 16:15:34 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:34 +0100] information/ConfigCompiler: Compiling config file: icinga2a/lib/icinga2/modified-attributes.conf
[2015-12-11 16:15:34 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2b' via host 'localhost' and port '8000'
[2015-12-11 16:15:34 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:34 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:15:34 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:34 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '8000'
[2015-12-11 16:15:39 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:39 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2b' via host 'localhost' and port '8000'
[2015-12-11 16:15:39 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:39 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:39 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '8000'
[2015-12-11 16:15:39 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:15:44 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:44 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2b' via host 'localhost' and port '8000'
[2015-12-11 16:15:44 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:44 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:44 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:15:44 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '8000'
[2015-12-11 16:15:45 +0100] information/ApiListener: New client connection for identity 'icinga2b'
[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'global-templates': 1 registered config dirs.
[2015-12-11 16:15:45 +0100] information/ApiListener: Config path: 'icinga2a/etc/icinga2/zones.d/global-templates'.
[2015-12-11 16:15:45 +0100] warning/ApiListener: Ignoring config update for zone 'global-templates' because we have an authoritative version of the zone's config.
[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'satellite': 1 registered config dirs.
[2015-12-11 16:15:45 +0100] information/ApiListener: Sending updates for endpoint 'icinga2b'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Config path: 'icinga2a/etc/icinga2/zones.d/satellite'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Syncing zone 'satellite' to endpoint 'icinga2b'.
[2015-12-11 16:15:45 +0100] warning/ApiListener: Ignoring config update for zone 'satellite' because we have an authoritative version of the zone's config.
[2015-12-11 16:15:45 +0100] information/ApiListener: Syncing zone 'master' to endpoint 'icinga2b'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Syncing global zone 'global-templates' to endpoint 'icinga2b'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga2b'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Finished sending updates for endpoint 'icinga2b'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Replayed 58 messages.
[2015-12-11 16:15:49 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:49 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:49 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:15:54 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:54 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:54 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:15:59 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:59 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:59 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:16:03 +0100] information/ApiListener: New client connection (no client certificate)
[2015-12-11 16:16:03 +0100] information/HttpServerConnection: Request: POST /v1/config/packages/example-cmdb (root)
[2015-12-11 16:16:04 +0100] information/DbConnection: Pausing IDO connection: ido-mysql
[2015-12-11 16:16:04 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:16:04 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:16:04 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:16:06 +0100] information/ApiListener: New client connection (no client certificate)
[2015-12-11 16:16:06 +0100] information/HttpServerConnection: Request: POST /v1/config/stages/example-cmdb (root)
[2015-12-11 16:16:06 +0100] information/ConfigPackageUtility: Updating configuration file: icinga2a/lib/icinga2/api/packages/example-cmdb/mbmif.int.netways.de-1449846966-0/zones.d/master/test.conf
[2015-12-11 16:16:06 +0100] information/Application: Got reload command: Starting new instance.
[2015-12-11 16:16:07 +0100] information/Application: Received request to shut down.
[2015-12-11 16:16:07 +0100] information/Application: Shutting down...
[2015-12-11 16:16:07 +0100] information/CheckerComponent: Checker stopped.
[2015-12-11 16:16:07 +0100] information/ConfigObject: Dumping program state to file 'icinga2a/lib/icinga2/icinga2.state'
[2015-12-11 16:16:07 +0100] information/IcingaApplication: Icinga has shut down.

icinga2b

mbmif /usr/local/tests/icinga2/master-slave (master) # ./run icinga2b
[2015-12-11 16:15:44 +0100] information/cli: Icinga application loader (version: v2.4.1-66-g138d416; debug)
[2015-12-11 16:15:44 +0100] information/ConfigCompiler: Compiling config file: icinga2b.conf
[2015-12-11 16:15:44 +0100] information/Utility: Loading library 'libcompat.dylib'
[2015-12-11 16:15:44 +0100] information/Utility: Loading library 'libdb_ido_mysql.dylib'
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/itl
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command.conf
[2015-12-11 16:15:45 +0100] information/Utility: Loading library 'libmethods.dylib'
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command-icinga.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/timeperiod.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command-plugins.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/include.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/include.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/../active.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.com10.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.com11.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.com2.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.com3.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.com5.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.com6.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.com7.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.dns.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google.zone.conf
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442490392-0/conf.d/hosts/google1.conf
[2015-12-11 16:15:45 +0100] information/ConfigItem: Committing config items
[2015-12-11 16:15:45 +0100] information/ApiListener: My API identity: icinga2b
[2015-12-11 16:15:45 +0100] warning/ApplyRule: Apply rule 'icinga' (in icinga2b.conf: 121:1-121:22) for type 'Service' does not match anywhere!
[2015-12-11 16:15:45 +0100] warning/ApplyRule: Apply rule 'remote-client' (in icinga2b.conf: 131:1-131:29) for type 'Service' does not match anywhere!
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 3 Endpoints.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 3 Zones.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 4 Services.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 12 Hosts.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Instantiated 54 CheckCommands.
[2015-12-11 16:15:45 +0100] information/ScriptGlobal: Dumping variables to file 'icinga2b/cache/icinga2/icinga2.vars'
[2015-12-11 16:15:45 +0100] information/ConfigObject: Restoring program state from file 'icinga2b/lib/icinga2/icinga2.state'
[2015-12-11 16:15:45 +0100] information/ConfigObject: Restored 82 objects. Loaded 0 new objects without state.
[2015-12-11 16:15:45 +0100] information/ConfigItem: Triggering Start signal for config items
[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'satellite': 0 registered config dirs.
[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'global-templates': 0 registered config dirs.
[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'master': 0 registered config dirs.
[2015-12-11 16:15:45 +0100] information/ApiListener: Adding new listener on port '8000'
[2015-12-11 16:15:45 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2015-12-11 16:15:45 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:45 +0100] information/ConfigItem: Activated all objects.
[2015-12-11 16:15:45 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2a' via host 'localhost' and port '7000'
[2015-12-11 16:15:45 +0100] information/ConfigCompiler: Compiling config file: icinga2b/lib/icinga2/modified-attributes.conf
[2015-12-11 16:15:45 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:45 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:15:45 +0100] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.0')
[2015-12-11 16:15:45 +0100] information/ApiListener: New client connection for identity 'icinga2a'
[2015-12-11 16:15:45 +0100] information/ApiListener: Sending updates for endpoint 'icinga2a'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Syncing zone 'satellite' to endpoint 'icinga2a'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Syncing global zone 'global-templates' to endpoint 'icinga2a'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga2a'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Finished sending updates for endpoint 'icinga2a'.
[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'global-templates': 0 registered config dirs.
[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'master': 0 registered config dirs.
[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'satellite': 0 registered config dirs.
[2015-12-11 16:15:50 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:50 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:50 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:15:55 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:15:55 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:15:55 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:16:00 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:16:00 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:16:00 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:16:05 +0100] information/JsonRpcConnection: Reconnecting to API endpoint 'icinga2c' via host 'localhost' and port '9000'
[2015-12-11 16:16:05 +0100] critical/TcpSocket: Invalid socket: Connection refused
[2015-12-11 16:16:05 +0100] critical/ApiListener: Cannot connect to host 'localhost' on port '9000'
[2015-12-11 16:16:07 +0100] warning/TlsStream: TLS stream was disconnected.
[2015-12-11 16:16:07 +0100] warning/JsonRpcConnection: Error while reading JSON-RPC message for identity 'icinga2a': Error: std::exception


    (0) 2   libbase.dylib                       0x00000001037b2386 __cxa_throw + 214
    (1) 3   libbase.dylib                       0x00000001038c16b8 _ZN5boost15throw_exceptionIN6icinga13openssl_errorEEEvRKT_ + 120
    (2) 4   libbase.dylib                       0x00000001038becc3 _ZN5boost16exception_detail16throw_exception_IN6icinga13openssl_errorEEEvRKT_PKcS8_i + 179
    (3) 5   libbase.dylib                       0x00000001038be0e4 _ZNK6icinga9TlsStream11HandleErrorEv + 212
    (4) 6   libbase.dylib                       0x00000001038be4ca _ZN6icinga9TlsStream4ReadEPvmb + 266
    (5) 7   libbase.dylib                       0x000000010387e289 _ZN6icinga17StreamReadContext14FillFromStreamERKN5boost13intrusive_ptrINS_6StreamEEEb + 281
    (6) 8   libbase.dylib                       0x000000010383b04a _ZN6icinga9NetString20ReadStringFromStreamERKN5boost13intrusive_ptrINS_6StreamEEEPNS_6StringERNS_17StreamReadContextEb + 122
    (7) 9   libremote.dylib                     0x0000000102f416cd _ZN6icinga7JsonRpc11ReadMessageERKN5boost13intrusive_ptrINS_6StreamEEEPNS2_INS_10DictionaryEEERNS_17StreamReadContextEb + 93
    (8) 10  libremote.dylib                     0x0000000102f439d4 _ZN6icinga17JsonRpcConnection14ProcessMessageEv + 132
    (9) 11  libremote.dylib                     0x0000000102f42b2f _ZN6icinga17JsonRpcConnection20DataAvailableHandlerEv + 63
    (10) 12  libremote.dylib                     0x0000000102f47314 _ZNK5boost4_mfi3mf0IvN6icinga17JsonRpcConnectionEE4callINS_13intrusive_ptrIS3_EEEEvRT_PKv + 132
    (11) 13  libremote.dylib                     0x0000000102f4725c _ZNK5boost4_mfi3mf0IvN6icinga17JsonRpcConnectionEEclINS_13intrusive_ptrIS3_EEEEvRT_ + 44
    (12) 14  libremote.dylib                     0x0000000102f47200 _ZN5boost3_bi5list1INS0_5valueINS_13intrusive_ptrIN6icinga17JsonRpcConnectionEEEEEEclINS_4_mfi3mf0IvS5_EENS1_IRKNS3_INS4_6StreamEEEEEEEvNS0_4typeIvEERT_RT0_i + 80
    (13) 15  libremote.dylib                     0x0000000102f4719f _ZN5boost3_bi6bind_tIvNS_4_mfi3mf0IvN6icinga17JsonRpcConnectionEEENS0_5list1INS0_5valueINS_13intrusive_ptrIS5_EEEEEEEclIRKNS9_INS4_6StreamEEEEEvOT_ + 79
    (14) 16  libremote.dylib                     0x0000000102f46f38 _ZN5boost6detail8function26void_function_obj_invoker1INS_3_bi6bind_tIvNS_4_mfi3mf0IvN6icinga17JsonRpcConnectionEEENS3_5list1INS3_5valueINS_13intrusive_ptrIS8_EEEEEEEEvRKNSC_INS7_6StreamEEEE6invokeERNS1_15function_bufferESK_ + 56
    (15) 17  libbase.dylib                       0x00000001038867db _ZNK5boost9function1IvRKNS_13intrusive_ptrIN6icinga6StreamEEEEclES6_ + 155
    (16) 18  libbase.dylib                       0x000000010388672b _ZNK5boost8signals26detail20call_with_tuple_argsINS1_9void_typeEE8m_invokeINS_8functionIFvRKNS_13intrusive_ptrIN6icinga6StreamEEEEEEJLj0EEJSC_EEES3_PvRT_NS1_19unsigned_meta_arrayIJXspT0_EEEENSt3__15tupleIJDpT1_EEE + 59
    (17) 19  libbase.dylib                       0x00000001038866b8 _ZNK5boost8signals26detail20call_with_tuple_argsINS1_9void_typeEEclINS_8functionIFvRKNS_13intrusive_ptrIN6icinga6StreamEEEEEEJSC_ELm1EEES3_RT_NSt3__15tupleIJDpT0_EEEN4mpl_6size_tIXT1_EEE + 216
    (18) 20  libbase.dylib                       0x00000001038865ce _ZNK5boost8signals26detail21variadic_slot_invokerINS1_9void_typeEJRKNS_13intrusive_ptrIN6icinga6StreamEEEEE8m_invokeINS_10shared_ptrINS1_15connection_bodyINSt3__14pairINS1_15slot_meta_groupENS_8optionalIiEEEENS0_4slotIFvS9_ENS_8functionISL_EEEENS0_5mutexEEEEEEES3_RKT_PKS3_ + 78
    (19) 21  libbase.dylib                       0x0000000103886579 _ZNK5boost8signals26detail21variadic_slot_invokerINS1_9void_typeEJRKNS_13intrusive_ptrIN6icinga6StreamEEEEEclINS_10shared_ptrINS1_15connection_bodyINSt3__14pairINS1_15slot_meta_groupENS_8optionalIiEEEENS0_4slotIFvS9_ENS_8functionISL_EEEENS0_5mutexEEEEEEES3_RKT_ + 41
    (20) 22  libbase.dylib                       0x000000010388647b _ZNK5boost8signals26detail20slot_call_iterator_tINS1_21variadic_slot_invokerINS1_9void_typeEJRKNS_13intrusive_ptrIN6icinga6StreamEEEEEENSt3__115__list_iteratorINS_10shared_ptrINS1_15connection_bodyINSC_4pairINS1_15slot_meta_groupENS_8optionalIiEEEENS0_4slotIFvSA_ENS_8functionISM_EEEENS0_5mutexEEEEEPvEESR_E11dereferenceEv + 107
    (21) 23  libbase.dylib                       0x00000001038863e5 _ZN5boost9iterators20iterator_core_access11dereferenceINS_8signals26detail20slot_call_iterator_tINS4_21variadic_slot_invokerINS4_9void_typeEJRKNS_13intrusive_ptrIN6icinga6StreamEEEEEENSt3__115__list_iteratorINS_10shared_ptrINS4_15connection_bodyINSF_4pairINS4_15slot_meta_groupENS_8optionalIiEEEENS3_4slotIFvSD_ENS_8functionISP_EEEENS3_5mutexEEEEEPvEESU_EEEENT_9referenceERKSZ_ + 21
    (22) 24  libbase.dylib                       0x000000010388631d _ZNK5boost9iterators6detail20iterator_facade_baseINS_8signals26detail20slot_call_iterator_tINS4_21variadic_slot_invokerINS4_9void_typeEJRKNS_13intrusive_ptrIN6icinga6StreamEEEEEENSt3__115__list_iteratorINS_10shared_ptrINS4_15connection_bodyINSF_4pairINS4_15slot_meta_groupENS_8optionalIiEEEENS3_4slotIFvSD_ENS_8functionISP_EEEENS3_5mutexEEEEEPvEESU_EES7_NS0_25single_pass_traversal_tagERKS7_lLb0ELb0EEdeEv + 29
    (23) 25  libbase.dylib                       0x0000000103886255 _ZNK5boost8signals219optional_last_valueIvEclINS0_6detail20slot_call_iterator_tINS4_21variadic_slot_invokerINS4_9void_typeEJRKNS_13intrusive_ptrIN6icinga6StreamEEEEEENSt3__115__list_iteratorINS_10shared_ptrINS4_15connection_bodyINSF_4pairINS4_15slot_meta_groupENS_8optionalIiEEEENS0_4slotIFvSD_ENS_8functionISP_EEEENS0_5mutexEEEEEPvEESU_EEEEvT_SZ_ + 69
    (24) 26  libbase.dylib                       0x0000000103885f0f _ZNK5boost8signals26detail16combiner_invokerIvEclINS0_19optional_last_valueIvEENS1_20slot_call_iterator_tINS1_21variadic_slot_invokerINS1_9void_typeEJRKNS_13intrusive_ptrIN6icinga6StreamEEEEEENSt3__115__list_iteratorINS_10shared_ptrINS1_15connection_bodyINSH_4pairINS1_15slot_meta_groupENS_8optionalIiEEEENS0_4slotIFvSF_ENS_8functionISR_EEEENS0_5mutexEEEEEPvEESW_EEEEvRT_T0_S13_ + 175
    (25) 27  libbase.dylib                       0x0000000103885c8f _ZN5boost8signals26detail11signal_implIFvRKNS_13intrusive_ptrIN6icinga6StreamEEEENS0_19optional_last_valueIvEEiNSt3__14lessIiEENS_8functionIS9_EENSF_IFvRKNS0_10connectionES8_EEENS0_5mutexEEclES8_ + 975
    (26) 28  libbase.dylib                       0x000000010387e65f _ZN5boost8signals26signalIFvRKNS_13intrusive_ptrIN6icinga6StreamEEEENS0_19optional_last_valueIvEEiNSt3__14lessIiEENS_8functionIS8_EENSE_IFvRKNS0_10connectionES7_EEENS0_5mutexEEclES7_ + 47
    (27) 29  libbase.dylib                       0x000000010387db35 _ZN6icinga6Stream19SignalDataAvailableEv + 85
    (28) 30  libbase.dylib                       0x00000001038bdf66 _ZN6icinga9TlsStream7OnEventEi + 2694
    (29) 31  libbase.dylib                       0x00000001038be002 _ZThn288_N6icinga9TlsStream7OnEventEi + 34
    (30) 32  libbase.dylib                       0x000000010386ffcc _ZN6icinga12SocketEvents10ThreadProcEv + 2684
    (31) 33  libbase.dylib                       0x0000000103878ec6 _ZN5boost6detail11thread_dataIPFvvEE3runEv + 22
    (32) 34  libboost_thread-mt.dylib            0x00000001027fb715 _ZN5boost12_GLOBAL__N_112thread_proxyEPv + 53
    (33) 35  libsystem_pthread.dylib             0x00007fff94d14c13 _pthread_body + 131
    (34) 36  libsystem_pthread.dylib             0x00007fff94d14b90 _pthread_body + 0
    (35) 37  libsystem_pthread.dylib             0x00007fff94d12375 thread_start + 13




[2015-12-11 16:16:07 +0100] warning/JsonRpcConnection: API client disconnected for identity 'icinga2a'
[2015-12-11 16:16:07 +0100] warning/ApiListener: Removing API client for endpoint 'icinga2a'. 0 API clients left.
[2015-12-11 16:16:07 +0100] information/ApiListener: New client connection for identity 'icinga2a'
[2015-12-11 16:16:07 +0100] information/ApiListener: Sending updates for endpoint 'icinga2a'.
[2015-12-11 16:16:07 +0100] information/ApiListener: Syncing zone 'satellite' to endpoint 'icinga2a'.
[2015-12-11 16:16:07 +0100] information/ApiListener: Syncing global zone 'global-templates' to endpoint 'icinga2a'.
[2015-12-11 16:16:07 +0100] information/ApiListener: Syncing zone 'master' to endpoint 'icinga2a'.
[2015-12-11 16:16:07 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga2a'.
[2015-12-11 16:16:07 +0100] information/ApiListener: Finished sending updates for endpoint 'icinga2a'.
[2015-12-11 16:16:07 +0100] information/ApiListener: Checking zone 'global-templates': 0 registered config dirs.
[2015-12-11 16:16:07 +0100] information/ApiListener: Checking zone 'master': 0 registered config dirs.
[2015-12-11 16:16:07 +0100] information/ApiListener: Updating configuration file: icinga2b/lib/icinga2/api/zones/master//example-cmdb/test.conf
[2015-12-11 16:16:07 +0100] information/ApiListener: Checking zone 'satellite': 0 registered config dirs.
[2015-12-11 16:16:07 +0100] information/ApiListener: Restarting after configuration change.
[2015-12-11 16:16:07 +0100] information/Application: Got reload command: Starting new instance.
[2015-12-11 16:16:07 +0100] information/Application: Received request to shut down.
[2015-12-11 16:16:07 +0100] information/Application: Shutting down...
[2015-12-11 16:16:07 +0100] information/DbConnection: Pausing IDO connection: ido-mysql
[2015-12-11 16:16:07 +0100] information/ConfigObject: Dumping program state to file 'icinga2b/lib/icinga2/icinga2.state'
[2015-12-11 16:16:07 +0100] information/IcingaApplication: Icinga has shut down.

Notes

That's debug output I've added for better testing. I'll change that into debug messages later.

[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'master': 0 registered config dirs.

[2015-12-11 16:15:45 +0100] information/ApiListener: Checking zone 'global-templates': 1 registered config dirs.
[2015-12-11 16:15:45 +0100] information/ApiListener: Config path: 'icinga2a/etc/icinga2/zones.d/global-templates'.
[2015-12-11 16:15:45 +0100] warning/ApiListener: Ignoring config update for zone 'global-templates' because we have an authoritative version of the zone's config.

Possible Problems

Either you'll have zone directories (they can be empty too) in

  • /etc/icinga2/zones.d/
  • /var/lib/icinga2/api/packages///zones.d/

If one of these matches on the second node, it will disallow any updates sent by the first master.

Furthermore cluster config updates won't happen if the config files did not change on disk. So it might be the case that you don't see any config sync updates on restart of the nodes, just because there is nothing to sync. Better clear the api/zones directory on both nodes beforehand.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 17:22:40 +00:00

dnsmichi wrote:

I've tested this for a while now in my 2 instance cluster, and it works like expected.

Glad to hear that.

h3. Tests

Does not look like a default config, but still looks like a reasonable test setup to me. Here what I did for cleanup:

systemctl stop icinga2
/bin/rm -r /var/lib/icinga2/api/zones/*
/bin/rm -r /var/lib/icinga2/api/repository/*
/bin/rm -r /var/lib/icinga2/api/packages/*
/bin/rm /var/lib/icinga2/api/log/*
/bin/rm /var/lib/icinga2/icinga2.state
/bin/rm /var/log/icinga2/debug.log

tree /var/lib/icinga2/api/
/var/lib/icinga2/api/
├── log
├── packages
├── repository
└── zones

Startet both nodes after cleaning up both of them.

That's debug output I've added for better testing. I'll change that into debug messages later.

Messages differ in current GIT and snapshots (using v2.4.1-68-gd781c39), here you go:

Master 1:

[17:47:27] notice/ApiListener: Registered config directories for zone 'catalog': /var/lib/icinga2/api/packages/director/master1-1449852446-0/zones.d/catalog
[17:47:27] notice/ApiListener: Registered config directories for zone 'master': /var/lib/icinga2/api/packages/director/master1-1449852446-0/zones.d/master

Master 2:

[17:47:27] notice/ApiListener: Registered config directories for zone 'catalog': 
[17:47:27] notice/ApiListener: Registered config directories for zone 'master': 
[17:47:29] notice/ApiListener: Registered config directories for zone 'director-global': 
[17:47:29] notice/ApiListener: Registered config directories for zone 'catalog': /var/lib/icinga2/api/zones/catalog
[17:47:29] notice/ApiListener: Registered config directories for zone 'director-global': 
[17:47:29] notice/ApiListener: Registered config directories for zone 'master': /var/lib/icinga2/api/zones/master

The 'master' zone got replicated from master1 (who got it through the API) to master2, master2 therefore reloaded between 17:14:27 and 17:47:29. And while the initial replication was fine, it started to refuse that config afterwards:

[17:47:29] warning/ApiListener: Ignoring config update for zone 'master' because we have an authoritative version of the zone's config.

As you can see, Master 2 feels not responsible for 'catalog' and 'master', but two seconds later it is. This perfectly reflects what I continued to explain. More details:

  • /etc/icinga2/zones.d is empty on both nodes, it always was
  • 'master' is the main zone, it's zone definition is in /etc/icinga2
  • 'catalog' is a global zone, it's zone definition is in /etc/icinga2. Has been added to show differences to zones deployed via API
  • 'director-global' is a global zone, has been deployed through the api, it's related Zone-Definition has been deployed to a file in the master zone
  • 'agent1' is a non-global zone with 'master' as it's parent zone. It doesn't even show up in above logs, but it's deployment and initial replication both worked fine

h3. Possible Problems

Either you'll have zone directories (they can be empty too) in

* /etc/icinga2/zones.d/

Not, never had.

* /var/lib/icinga2/api/packages///zones.d/

Always on master1 after config dump, never on master2.

If one of these matches on the second node, it will disallow any updates sent by the first master.

Not the case here. While IMO Icinga2 MUST be able to cope with an outdated zone config in a stage, this was never the issue here. I mentioned that we can exclude those errors a couple of times in this ticket.

Furthermore cluster config updates won't happen if the config files did not change on disk. So it might be the case that you don't see any config sync updates on restart of the nodes, just because there is nothing to sync. Better clear the api/zones directory on both nodes beforehand.

As all my logs showed, the problem is that master1 HAS something to sync, but master2 doesn't accept it as it feels authoritative for the zone. We are talking about the initial config dump here, so there definitively ARE changes. Subsequent changes are always accepted on master1 but no longer on master2.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 18:18:33 +00:00

[17:47:27] notice/ApiListener: Registered config directories for zone 'catalog': 
[17:47:27] notice/ApiListener: Registered config directories for zone 'master': 
[17:47:29] notice/ApiListener: Registered config directories for zone 'director-global': 
[17:47:29] notice/ApiListener: Registered config directories for zone 'catalog': /var/lib/icinga2/api/zones/catalog
[17:47:29] notice/ApiListener: Registered config directories for zone 'director-global': 
[17:47:29] notice/ApiListener: Registered config directories for zone 'master': /var/lib/icinga2/api/zones/master

That's impossible unless you are using "include_zones" manually by yourself inside your configuration (and that's considered dangerous as it will result in duplicated includes). Please show the recursive grep output from inside your configuration.

grep -r include_zone /var/lib/icinga2/api/packages/
grep -r include_zone /etc/icinga2
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by mfriedrich on 2015-12-11 18:25:37 +00:00

Ok, discard my last comment, I think I found something.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 11, 2015

Updated by tgelf on 2015-12-11 21:26:34 +00:00

For the sake of completeness, the only occurrence of include_zones in this setup is the autogenerated

include_zones "director", "zones.d"

in /var/lib/icinga2/api/packages/director/master1-1449852446-0/include.conf on master1. It does not exist on master2. Grepped through /etc/icinga2 and /var/lib/icinga2 on both of them.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 12, 2015

Updated by mfriedrich on 2015-12-12 11:10:45 +00:00

  • File added i2_10819_notes.jpg

Noted the idea, slept well and then wrote it down to finally implement and test a fix (cannot get this out of my head anyways).

Changing the config authority checks unveiled a different, hidden problem caused by a change we did implement the API config packages in early September.

I will not open additional issues as this will harden the backport process of issues and git commits but mark these bugs/problems below.

During startup, additional includes are collected next to the user-defined ones:

  • /etc/icinga2/zones.d (RegisterZoneDir as "_etc" tag)
  • /var/lib/icinga2/api/packages (each package and stage may use include_zones and register the package name as zone tag)
  • /var/lib/icinga2/api/zones which contains synced copies from the cluster config sync

Unfortunately /var/lib/icinga2/api/zones which are included inside IncludeNonLocalZone() call IncludeZoneDirRecursive which registers a zone tag for "_etc". This is bug 1 in addition to this issue. It will make Icinga 2 always copy /var/lib... to /var/lib... but normally you won't recognize that during the initial "copy-all-zones-config to /var/lib" process in ApiListener::SyncZoneDirs() on startup.

Fixing this bug by removing the RegisterZoneDir() call would obviously fix the issue you've found. Note: This issue does not happen on the first config sync, it will happen on restart(s) when /var/lib/icinga2/api/zones is already populated and SyncZoneDir() will think through IsConfigMaster() that this is an authoritative copy (which it is not).

Second Problem: There are 2 config authoritative checks in 2 location. One lives in aplistener-filesync.cpp and ensures that

  • SyncZoneDir only copies zones to /var/lib/icinga2/api/zones it has an authoritative local copy inside "etc" or "api packages"
  • UpdateConfigHandler only accepts configuration when it does not have a local config authority

Both checks use the same functionality, previously it was just a check for "/etc/icinga2/zones.d", now it is a check for all registered local zone config authorities (that's already fixed).

The second config authoritative check lives inside daemonutility.cpp in IncludeNonLocalZone(). It ensures that

  • the config in /var/lib/icinga2/api/zones/ is only included if there are no other authoritative versions ("etc" or "api package") locally on disk
  • the config sync added a marker that this zone is authoritative (prefixed with a dot, as file)

Third problem: The check in IncludeNonLocalZone() for authoritative copies does not take packages with zones.d into account. So even if the problems above are fixed, it will certainly crash on duplicate config include on the master (probably if you accidentally remove .authoritative inside the directory, which is a safety hook, but who knows what users will do). I did not test that in detail, but I rest assured that changing the authoritative check into etc and api packages will certainly help unveil user-defined problems.

So the third problem can be fixed by introducing a general function called HasZoneConfigAuthority() previously called IsConfigMaster(). In order to use them in libcli and libremote, we'll add them to libconfig, where the registered zone dir functionality lives already.

Fourth problem: The order of including etc, packages and cluster synced config was wrong. etc and api packages will call RegisterZoneDir() which gets evaluated inside HasZoneConfigAuthority(). Previously it was: etc, cluster synced config, api packages. This would leave api packages with zones.d not being taken into account for config inclusion and probably also break the initial config compilation. I did not test such failure scenario in detail, just fixing the code how it's supposed to be.

Fifth problem: SyncZoneDir() doesn't need an additional authoritative config check, as it already calls ConfigCompiler::GetZoneDirs() and only syncs registered zone directories. Further is needs to check wether there is new config available, since it now always logs that it will sync something even if there is nothing to sync from within newConfig.

I've pushed a fix to git master, and will wait for your feedback from tests. In order to properly test it with daemons running in foreground in my not-so-regular-but-still-working-for-years cluster test setup, killing icinga2a, editing the api package config so it is modified, and starting icinga2a again will make icinga2b accept a new configuration, even if it has loaded one already from inside /var/lib/icinga2/api/zones.

Off-topic: Obviously no-one including myself ever tested the api packages with zones.d configuration. That's one for QA & feature requestors preventing such problems in the first place.

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 12, 2015

Updated by tgelf on 2015-12-12 13:58:20 +00:00

  • Status changed from Feedback to Resolved

I deployed it to the very same environment, works as it should right now. Thanks a lot for your effort, Michael! I'll leave this issue closed right now ;)

Some related topics we should discuss on Monday:

  • How to handle failover scenarios. I tried it out, and as expected everything stops working. Even on failback: as soon as you deployed to the other node once they will not sync themselves again. It would be great if we could solve this in the Core itself, as it would probably solve other issues involving outdated stuff in /var too. No problem for me if not, I can handle this in the Director project in a safe way, detect and fix such issues. We should nonetheless carefully document eventual pitfalls for others.
  • Things are a little bit weird when it goes to adding/removing zones - something that basically happens for every host when it is running an agent.
    • Adding a new host: currently you experience at least two restarts on master2: master1 got a new config, restarts. master2 reconnects, gets a new config, ignores the new zone master1 wants to sync as it doesn't know about, reloads, learns about the new zone from the now parsed new parent zone, therefore accepts the config sync for the new zone, restarts again.
    • Deleting a host (with it's endpoint and zone): Zone remains on master2 - and that's evil. Especially when you recreate the host later on in another zone. These are basically one-click operations in director. I moved host "test" to a new "test" zone, deployed, moved it back to the "master" zone and removed the "test" zone. Result: config for the "test" zone remains alive on master2, and interestingly host "test" remained a member of the zone "test" on master2 while it was moved back to "master" on master1. I didn't dig deeper, let's discuss this on Monday.

Thanks again, enjoy your weekend!
Thomas

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Dec 12, 2015

Updated by mfriedrich on 2015-12-12 14:52:45 +00:00

  • Assigned to changed from tgelf to mfriedrich
  • Target Version set to 2.4.2

Thanks for testing this that quick :)

@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Feb 23, 2016

Updated by gbeutner on 2016-02-23 09:58:24 +00:00

  • Backport? changed from Not yet backported to Already backported
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.