HA master doesn't seem to activate the api stage

[dasJ]

Describe the bug

Creating a HA master as explained in the manual doesn't seem to work properly. I'll explain the steps I performed below.
When both masters are running (let's say I had A running, and added B as a second master), A is performing the checks while B writes to IDO.
This works great and I can see the messages A sends to B in the debug log: Received 'event::CheckResult' message from 'A'.
However, the event is never written to IDO. Inspecting the output of icinga2 object list shows that the objects for which the checks are executed do not exist.
I therefore think icinga2 just doesn't write any results to the IDO when it doesn't know the object.
However, the file with the objects (/var/lib/icinga2/api/zones/master/_etc/icinga2.conf) exists and is valid.

To Reproduce

I created the HA setup by:

1. Prepare the IDO feature on B
2. Stop A
3. Copy /var/lib/icinga2 from A to B
4. Start A and B

Expected behavior

B should create all objects it needs in order to write the results to the IDO.
Also, a warning would be nice if it was asked to write results into the IDO and it doesn't know the object of which the results are to be written.

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version):
  I tested both icinga 2.10.5 and 2.11
• Operating System and version:
• Enabled features (icinga2 feature list):
  On both A and B:

  Disabled features: debuglog elasticsearch gelf graphite influxdb mainlog opentsdb perfdata
  Enabled features: api checker ido-mysql notification syslog
  

• Icinga Web 2 version and modules (System - About):
• Config validation (icinga2 daemon -C):
  On A:
• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

[dasJ]

Well, both nodes didn't have the same instance ID in the ido object. Docs didn't mention anything about that so I set both to the hostnames of each node 🤷‍♂️

[dnsmichi]

I was wondering why you did copy /var/lib/icinga2 for the HA setup - typically you don't need that during setup where you treat the secondary master as satellite. After the initial connection, runtime created objects are synced and stored in the local _api package again.

[dasJ]

Mainly because the docs mention this. So is this unneeded? Or is only icinga2.state and the ca directory (for HA ca) needed?

[dnsmichi]

Maybe the docs should be clarified on this, I only added this because of customer requests. The initial sync is required when you decide e.g. 1 month after setting up one master, that a secondary would be needed. This is to ensure that the state file is intact, and you may of course overcome the problem with different _api stage names (doesn't matter though). One problem I do see with the description is the content of /var/lib/icinga2/api/zones where the secondary master must not have the .authoritative marker file inside, since it doesn't have an authoritative copy of the configuration.

I'll think about this and fix it, thanks for the discussion.