Crash when upgrading from 2.10.0 to 2.10.1 (SELinux related)

[stevie-sy]

I wanted to upgrade vom 2.10.0 to 2.10.1 on our CentOS 7.5 System. But after the start of icinga2 failed. with

I saw in the Audit.log a lot of "denied" for icinga. e.g.

after I imported the selinux rules which I generated with "audit2allow" & "semodule", I could restart the icinga Service with "systemctl reset-failed icinga2.service"

But this doesn't work:

the file "/var/lib/icinga2/api/packages/_api/include.conf" in this message has only this Content:

and this is the Content of the crash-file in the message of systemctl:

I wanted to downgrade to 2.10.0. But with this I had also troubles now.
So I had to fix this with more SELinux rules.

After this import-rules the icinga2 Service run:
first Import file for sexlinux (with this icinga crashed again like I wrote before)

second Import file for selinux (after this one I could start icinga2:

[dnsmichi]

That's an SELinux problem then, it seems the policies for creating runtime data by the daemon in /var/lib/icinga2 are now denied. In therms of the icinga2-selinux package, nothing changed from .0 to .1. Maybe @dgoetz can shed some light when he's back from vacation :)

In terms of the meaning "crash" - the exit is not super fancy, but actually "permission denied" errors should immediately halt the daemon.

[dnsmichi]

PS: I now know where you are working, likely in my hometown. You might want to replace the screenshots ;)

[stevie-sy]

ups - thank you .. I had a Little stress do bring up the Cluster again and at the same time I wrote that issue :-) ... and it's friday .. :-)

[ekeih]

Due to the history of comment edits this information is still available. As far as I know there is no way to remove leaked information from a GitHub issue. :(

[stevie-sy]

@ekeih and is it possible do delete this issue?

[ekeih]

@stevie-sy I think this is also not possible: isaacs/github#253

[dnsmichi]

@ekeih yeah, I forgot that you can now edit issues. @stevie-sy I cannot delete issues, GH doesn't allow that for public repos. You may want to ask GH support, if it really hurts. If not, make it public via https://icinga.com/about/customers/#shareyourstory :)

[stevie-sy]

Thank you. I found an Option: delete old Version of the edit history. in my pull down menue there ist the text "deleted". if you don't see nothing it worked for me and the day is saved :-)

[ekeih]

@stevie-sy I only see that you deleted the content of old versions. But the content is not visible anymore 👍
Good to know for the future :)

[stevie-sy]

thank you for the check. (y)

[dnsmichi]

Poke @dgoetz :)

[dgoetz]

@stevie-sy Can you please run restorecon -Rv /var/lib/icinga2? This should change the label of the files from var_lib_t to icinga2_var_lib_t. If possible test afterwards something that creates a new file in the directory, it should be also icinga2_var_lib_t as policy is telling the domain icinga2_t to create file with icinga2_var_lib_t if created in a directory labeled icinga2_var_lib_t.

[stevie-sy]

@dgoetz thanks for the answer. The nodes which I updated already to 2.10.1 are still fixed . And the nodes got config updates from the config-master and the director.

I controlled the nodes, where I stopped the updates from 2.10.0 to 2.10.1, there the folder has the label _system_u:object_r:var_lib_t:s0 _. Every node was a fresh Installation with 2.10.0.
By the way I cotrolled also an old server with Version 2.9.2 and the Folder has also the correct Label system_u:object_r:icinga2_var_lib_t:s0

After restorecon I tried to update one of the left nodes to 2.10.1. The node crashed also with

In the crash-file you can find the line:

that are the Labels of the files in the Cache Folder:

I deleted the files icinga2.vars.* and icinga2.debug.* But that also didn't work. So there are again SELinux permission missing. So i did again some manully runs with audit2allow & semodule (at least 6 times)
The result is a new te-file with:

#============= icinga2_t ==============

#!!!! WARNING: 'var_t' is a base type.
allow icinga2_t var_t:file unlink;

#!!!! WARNING 'icinga2_t' is not allowed to write or create to tmp_t.  Change the label to icinga2_tmp_t.
allow icinga2_t tmp_t:file write;

#!!!! This avc is allowed in the current policy
allow icinga2_t var_t:file unlink;

allow icinga2_t ldconfig_exec_t:file execute;

#!!!! This avc can be allowed using the boolean 'authlogin_nsswitch_use_ldap'
allow icinga2_t random_device_t:chr_file getattr;
allow icinga2_t self:process execstack;
allow icinga2_t sysfs_t:dir read;
allow icinga2_t tmp_t:file setattr;

#!!!! This avc is allowed in the current policy
allow icinga2_t tmp_t:file write;

#!!!! This avc is allowed in the current policy
allow icinga2_t var_t:file unlink;

#!!!! This avc is allowed in the current policy
allow icinga2_t ldconfig_exec_t:file execute;

#!!!! This avc is allowed in the current policy
allow icinga2_t random_device_t:chr_file getattr;

#!!!! This avc is allowed in the current policy
allow icinga2_t self:process execstack;
allow icinga2_t self:process execmem;

#!!!! This avc is allowed in the current policy
allow icinga2_t sysfs_t:dir read;

#!!!! This avc is allowed in the current policy
allow icinga2_t tmp_t:file { setattr write };

#!!!! This avc is allowed in the current policy
allow icinga2_t self:process { execmem execstack };

With that it worked for now. Two nodes are left for the update. One of them is our config master. But I won't update this one if it isn't clear why this happens.

So is it possible that the rpm-packes for 2.10.0 were not correctly installed?

[dgoetz]

You should not allow these things as they are granting to much access.

I think I found the problem. With 2.10 the package layout changed and the scriptlet during installation/update only runs a relabeling for icinga2-bin and icinga2-common and not icinga2 because in older versions it did not include any file. Can you run a test for me by running the update and /sbin/fixfiles -R icinga2 restore afterwards? If this fixes the problem without any further steps my assumption is correct and I will create a PR for packaging.

[stevie-sy]

Yes I know. But I think the many permissions are because of our own scripts for some checks with MS SQL, DB2, IBM z/OS etc. Because SELinux denied the access to do what they should do :-)
If a have a little free time maybe I can optimize them to reduce the rights. That's only a short note on the side.

A tested it on one left node. Your command writes following output:

But I can (re)start the node and it seems it runs correctly because the icinga2.log says he is checking.

[dgoetz]

Thanks so I will create a PR and depending on release plan I will ask for a package release.

The scripts should run in a separate domain like nagios_services_plugin_t or nagios_unconfined_plugin_exec_t so not require any change to the icinga2_t domain. So if you want to increase security you simply have to label the plugins with the best matching context. I included tips on this in https://icinga.com/docs/icinga2/latest/doc/22-selinux/#general followed by a list of available types.

[stevie-sy]

You're welcome :-)

And thanks for the tipps and the link. I will put it on my To-Do when I have time for it.

[dnsmichi]

@dgoetz We can do that for 2.10.2, depending on users test feedback this is happening soon enough.