Sync rules create duplicate objects and behave differently on icingaweb2 and icingacli

[chgarling]

Expected Behavior

The sync rule jobs should create my service-sets and services without any problems.
Multiple runs of the sync rules should not create duplicate objects. Already existing objects should be
recognized and modified if required.

Current Behavior

Currently the first run of my sync rules create the objects (service-sets and services) as expected.
A second run would create the objects again and does not recognize, that they already exist. This results in
a failing deployment because of duplicate entries.

The current behavior is blocking me, because in this state I can not automate it without any problems.

Also I compared the changes a sync rule would apply between icingaweb2 and icingacli:
icingaweb2 would create the objects but icingacli would modify the same number of objects for the same sync rule.

Possible Solution

The debug possibilities for the sync rules could be better. The import runs can be easily debugged with
the fetch command so the output can be diffed and checked for differences. It would be very helpful to have
the possibility to diff two sync rule runs. With the actual behavior I can not figure out why two sync rule runs
try to create the same objects again.

Steps to Reproduce (for bugs)

1. Run and import all import sources
2. Run sync rule for a service-set
3. Run sync rule for the service regarding the service-set
4. Deploy
5. Run sync rule for a service-set again (should not come up with any new objects or modifications)

Your Environment

• Director version: 1.7.2
• incubator: 0.6.0-beta3
• ipl: v0.5.0
• reactbundle: 0.7.0
• Icinga Web 2 version: 2.8.2
• Icinga 2 version: r2.12.2-1
• Operating System and version: Debian 10.6 x86_64
• Webserver: apache2 2.4.38-3+deb10u4
• PHP versions: php (php-fpm) 7.3.19-1~deb10u1
• php-icinga: 2.8.2-1.buster
• php-imagick: 3.4.3-4.1

[chgarling]

Example for a service-set:

Example for a servicet:

[chgarling]

Here you can see the differences between icingaweb2 and icingacli:

[chgarling]

This happens if I try to run the sync rule for a service-set a second time and if I try to apply it:

[chgarling]

Also I am wondering, why there are no counted objects for the service-sets and only 3 services counted. I have definitely more:

[Thomas-Gelf]

Could you please put both the Import Source definition and the Sync Rule in a Basket and provide a related snapshot? Please before sending the JSON file please use a Text editor to mask sensible data. And could you please upgrade to incubator 0.6.0 and the latest Icinga Director master? It is about to be tagged as v1.8.0 and should be rock-solid.

Thanks,
Thomas

NB: The counters might be wrong, that code is older than the ServiceSet implementation. Please open a dedicated issue.

[chgarling]

Hi Thomas.
of course, but I am not exactly sure what to do regarding the basket you asked me for. Could you explain that?
Meanwhile I will do the updates and perform another test. Maybe its already fixed.
Regards, Christian

[chgarling]

Hi Thomas.

I have updated incubator to v0.6.0 and reactbundle to v0.8.0 but I can not find a director release v1.8.0. I only can find v.1.7.2 as lastest release:

https://github.com/Icinga/icingaweb2-module-director/

Regards, Christian

[Thomas-Gelf]

It hasn't been tagged yet, and if there is such a bug I'd prefer fixing it before tagging the release (scheduled for today). So please download the latest master archive

[chgarling]

Hi Thomas,

okay, after reading your previous post a second time I already understood what you meant with "update to master" :-)

I have done the update right now and will perform my tests. I will inform you about the results in a few minutes.

[chgarling]

I did my tests and its still the same behavior. I will provide the snapshot to you, but I need some more information about how to do that.

[Thomas-Gelf]

Director Dashboard -> Icinga Director Configuration -> Configuration Baskets.
Create a new one, Import Sources & Sync Rules -> Custom Selection
Go to your Import & Sync definitions -> Add to Basket
Back to the Basket -> Snapshot -> Download
Text Editor -> check whether there is sensible/personal data in there, if so: mask it with *** or similar. Alternatively I can also provide you a secure upload link.

[chgarling]

Thanks,
I have prepared the JSON file. Please provide the secure upload link for that.

[Thomas-Gelf]

https://nextcloud.icinga.com/index.php/s/XDE3o5Dtrn8Dze8

[chgarling]

Thanks, I have uploaded the file. If you need more information just let me know.

Regards, Christian

[chgarling]

Good morning Thomas,

just an additional information:

Some of the import sources use a self-written API in front of PuppetDB and Foreman which aggregates data in the way we need it to build our checks. If you need the JSON it generates, please let me know.

Regards, Christian

[chgarling]

Hello @Thomas-Gelf,

did you already find some time to investigate in this? I would like to go live with my new monitoring setup until christmas if possible.

Regards, Christian

[chgarling]

Hi @Thomas-Gelf ,

is there anything I can help with?

Regards, Christian

[chgarling]

Hello @Thomas-Gelf,

I just want to ask you if there are any news in this issue?

Regards, Christian

[chgarling]

Hello @Thomas-Gelf,

I did some more investigation. I looked at the code and found out that the application/controllers/SyncruleController.php is involved. It seems like the following condition is not true:
if ($object->hasBeenLoadedFromDb()) {

This is strange because the objects are stored in the icinga_service_set table. I switched on MariaDBs general_log but found only some unspecific queries like:
SELECT icinga_service_set.* FROM icinga_service_set

So I suppose the director has all the magic to check if an object is already in the database inside. Also I suppose the query for the objects do not match the search pattern so they are considered as not existing.

Finally I compared a manually created service-set in the director with a sync-rule generated service-set:

The manual one:
id: 192 object_name: Disk Checks - Altbestand object_type: template host_id: NULL description: NULL assign_filter: "Altbestand"=host.groups
The generated one:
id: 8074 object_name: Bareos_Job_k4729 object_type: template host_id: NULL description: NULL assign_filter: host.name=%22k4729.pixsoftware.de%22
Obviously the assign_filter looks a bit different. Maybe the url encoded double-quotes or the ordering of the parameter and the value could be the problem?

Regards, Christian

[chgarling]

To verify my guess I manually added a servie-set and assigned it to a host directly. It looks like the same:
id: 8075 object_name: Test-CGa object_type: template host_id: NULL description: NULL assign_filter: host.name=%22k4339.pixsoftware.de%22
It has the same notation as my generated one above, so this seems not to be the problem.

Could it be a problem with the object_name? I generate the object_name based on the check name in combination with the hostname to keep them unique.

Regards, Christian

[Thomas-Gelf]

This has been forgotten, sorry for that. Seems that and not many people seem to synchronize Service Templates and Sets. Probably because those who want to pre-seed their installations with such objects settled on using Configuration Baskets, I guess. Nonetheless - this should of course work. Finally got some time to investigate this issue, should have been fixed.