Startup of the distributed environment fails

[mcktr]

Expected Behavior

Both nodes in the distributed environment start.

Current Behavior

Provisioning fails with the following error messsage

The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

I saw in the puppet provisioning log, that some steps are skipped because of a failed dependency:

==> icinga2-master1: Warning: /Stage[main]/Profiles::Icinga::Icingaweb2/Icingaweb2::Module[map]/Icingaweb2::Inisection[module-map]/Concat[/etc/icingaweb2/modules/map/config.ini]/File[/etc/icingaweb2/modules/map/config.ini]: Skipping because of failed dependencies
==> icinga2-master1: Notice: /Stage[main]/Profiles::Icinga::Icingaweb2/Concat::Fragment[module_maps_dashboards]/Concat_fragment[module_maps_dashboards]: Dependency Service[icinga2] has failures: true

I sshed into the box and looked why Icinga 2 fails to start, it seems that Icinga 2 can't find/read the certificate because the file is almost empty.

michael@metis ~/Coding/icinga/icinga-vagrant/distributed (master) $ vagrant ssh icinga2-master1
 _____     _                   
|_   _|   (_)                  
  | |  ___ _ _ __   __ _  __ _ 
  | | / __| | '_ \ / _` |/ _` |
 _| || (__| | | | | (_| | (_| |
|_____\___|_|_| |_|\__, |\__,_|
                    __/ |      
                   |___/       
[vagrant@icinga2-master1 ~]$ sudo -s
[root@icinga2-master1 vagrant]# icinga2 daemon -C
[2018-05-24 20:06:58 +0200] information/cli: Icinga application loader (version: v2.8.4-721-g653a2b4)
[2018-05-24 20:06:58 +0200] information/cli: Loading configuration file(s).
[2018-05-24 20:06:58 +0200] information/ConfigItem: Committing config item(s).
[2018-05-24 20:06:58 +0200] warning/globals.getHostGeoLocation: Cannot find 'be' in GeoLocationShort
[2018-05-24 20:06:58 +0200] warning/ApiListener: Attribute 'key_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-05-24 20:06:58 +0200] warning/ApiListener: Attribute 'ca_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-05-24 20:06:58 +0200] warning/ApiListener: Attribute 'cert_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-05-24 20:06:58 +0200] warning/ApiListener: Please read the upgrading documentation for v2.8: https://www.icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/
[2018-05-24 20:06:58 +0200] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/certs//icinga2-master1.vagrant.demo.icinga.com.crt': 0, "error:00000000:lib(0):func(0):reason(0)"
[2018-05-24 20:06:58 +0200] critical/config: Error: Cannot get certificate from cert path: '/var/lib/icinga2/certs//icinga2-master1.vagrant.demo.icinga.com.crt'.
Location: in /etc/icinga2/features-enabled/api.conf: 3:1-3:24
/etc/icinga2/features-enabled/api.conf(1): # This file is managed by Puppet. DO NOT EDIT.
/etc/icinga2/features-enabled/api.conf(2): 
/etc/icinga2/features-enabled/api.conf(3): object ApiListener "api"  {
                                           ^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/features-enabled/api.conf(4):   cert_path = "/etc/icinga2/pki/icinga2-master1.vagrant.demo.icinga.com.crt"
/etc/icinga2/features-enabled/api.conf(5):   key_path = "/etc/icinga2/pki/icinga2-master1.vagrant.demo.icinga.com.key"

[2018-05-24 20:06:58 +0200] critical/config: 1 error
[root@icinga2-master1 vagrant]# ls -la /var/lib/icinga2/certs/icinga2-master1.vagrant.demo.icinga.com.crt 
-rw-r--r--. 1 icinga icinga 54 24. Mai 20:04 /var/lib/icinga2/certs/icinga2-master1.vagrant.demo.icinga.com.crt
[root@icinga2-master1 vagrant]# cat /var/lib/icinga2/certs/icinga2-master1.vagrant.demo.icinga.com.crt 
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
[root@icinga2-master1 vagrant]# 

A full provisioning log:

vagrant-icinga2-distributed-full-log.txt

Steps to Reproduce (for bugs)

1. Run vagrant up in the distributed folder

Context

Ran into this issue while testing.

Your Environment

• Vagrant version (vagrant -v): Vagrant 2.1.1
• Box name: distributed / icinga2-master1
• Release version: ca2cfab
• Operating system: Ubuntu 18.04 LTS (Bionic Beaver)
• Provider (VirtualBox, Parallels, libvirt): Oracle VM VirtualBox Manager 5.2.10_Ubuntu

[itblaked]

+1 Received same for Distributed
Also same end result for Standalone.

Vagrant version: Vagrant 2.0.2
box name: standalone/icinga2, distributed/icinga2 master1
Release version: c885d98
Operating System: Fedora 28
Provider: libvirt 4.1.0

[dnsmichi]

I don't have time atm to refactor this box. Probably I'll delete it, the setup wizards in 2.9 are super easy to build such an environment with just two blank VMs.

[dnsmichi]

==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/File[/var/lib/icinga2/ca]/ensure: created
==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/File[/var/lib/icinga2/ca/ca.crt]/ensure: defined content as '{md5}d339d32d448c50dcfd22a8cbc9a71e5b'
==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/File[/var/lib/icinga2/ca/ca.key]/ensure: defined content as '{md5}e2a7d1f44a793ce1c52334ef05d722f0'
==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/File[/etc/icinga2/pki/ca.crt]/ensure: defined content as '{md5}d339d32d448c50dcfd22a8cbc9a71e5b'
==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/Exec[icinga2 pki create certificate signing request]/returns: executed successfully
==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/File[/etc/icinga2/pki/icinga2-master1.vagrant.demo.icinga.com.key]/seluser: seluser changed 'unconfined_u' to 'system_u'
==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/Exec[icinga2 pki sign certificate]: Triggered 'refresh' from 1 event
==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/File[/etc/icinga2/pki/icinga2-master1.vagrant.demo.icinga.com.crt]/seluser: seluser changed 'unconfined_u' to 'system_u'
==> icinga2-master1: Notice: /Stage[main]/Icinga2::Pki::Ca/File[/etc/icinga2/pki/icinga2-master1.vagrant.demo.icinga.com.csr]/ensure: removed

This creates an empty client certificate.

[root@icinga2-master1 ~]# ls -la /var/lib/icinga2/certs//icinga2-master1.vagrant.demo.icinga.com.crt
-rw-r--r--. 1 icinga icinga 54 Jun 26 11:26 /var/lib/icinga2/certs//icinga2-master1.vagrant.demo.icinga.com.crt
[root@icinga2-master1 ~]# icinga2 daemon -C
[2018-06-26 11:34:26 +0200] information/cli: Icinga application loader (version: v2.8.4-795-ged1e45c)
[2018-06-26 11:34:26 +0200] information/cli: Loading configuration file(s).
[2018-06-26 11:34:26 +0200] information/ConfigItem: Committing config item(s).
[2018-06-26 11:34:26 +0200] warning/globals.getHostGeoLocation: Cannot find 'be' in GeoLocationShort
[2018-06-26 11:34:27 +0200] warning/ApiListener: Attribute 'key_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-06-26 11:34:27 +0200] warning/ApiListener: Attribute 'ca_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-06-26 11:34:27 +0200] warning/ApiListener: Attribute 'cert_path' for object 'api' of type 'ApiListener' is deprecated and should not be used.
[2018-06-26 11:34:27 +0200] warning/ApiListener: Please read the upgrading documentation for v2.8: https://www.icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/
[2018-06-26 11:34:27 +0200] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/certs//icinga2-master1.vagrant.demo.icinga.com.crt': 0, "error:00000000:lib(0):func(0):reason(0)"
[2018-06-26 11:34:27 +0200] critical/config: Error: Cannot get certificate from cert path: '/var/lib/icinga2/certs//icinga2-master1.vagrant.demo.icinga.com.crt'.
Location: in /etc/icinga2/features-enabled/api.conf: 3:1-3:24
/etc/icinga2/features-enabled/api.conf(1): # This file is managed by Puppet. DO NOT EDIT.
/etc/icinga2/features-enabled/api.conf(2):
/etc/icinga2/features-enabled/api.conf(3): object ApiListener "api"  {
                                           ^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/features-enabled/api.conf(4):   cert_path = "/etc/icinga2/pki/icinga2-master1.vagrant.demo.icinga.com.crt"
/etc/icinga2/features-enabled/api.conf(5):   key_path = "/etc/icinga2/pki/icinga2-master1.vagrant.demo.icinga.com.key"

[2018-06-26 11:34:27 +0200] critical/config: 1 error

[root@icinga2-master1 ~]# openssl x509 -in /var/lib/icinga2/certs//icinga2-master1.vagrant.demo.icinga.com.crt -text
unable to load certificate

[root@icinga2-master1 ~]# cat /etc/icinga2/pki/icinga2-master1.vagrant.demo.icinga.com.crt
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

https://github.com/Icinga/puppet-icinga2/blob/master/manifests/pki/ca.pp#L157

[root@icinga2-master1 ~]# icinga2 pki new-cert --cn icinga2-master1.vagrant.demo.icinga.com --key icinga2-master1.vagrant.demo.icinga.com.key --csr icinga2-master1.vagrant.demo.icinga.com.csr
information/base: Writing private key to 'icinga2-master1.vagrant.demo.icinga.com.key'.
critical/SSL: Error while opening private RSA key file 'icinga2-master1.vagrant.demo.icinga.com.key': 33558541, "error:0200100D:system library:fopen:Permission denied"
[root@icinga2-master1 ~]# cd /tmp/
[root@icinga2-master1 tmp]# icinga2 pki new-cert --cn icinga2-master1.vagrant.demo.icinga.com --key icinga2-master1.vagrant.demo.icinga.com.key --csr icinga2-master1.vagrant.demo.icinga.com.csr
information/base: Writing private key to 'icinga2-master1.vagrant.demo.icinga.com.key'.
information/base: Writing certificate signing request to 'icinga2-master1.vagrant.demo.icinga.com.csr'.
[root@icinga2-master1 tmp]# icinga2 pki sign-csr --csr icinga2-master1.vagrant.demo.icinga.com.csr --cert icinga2-master1.vagrant.demo.icinga.com.crt
critical/SSL: Could not read RSA key from CA key file '/var/lib/icinga2/ca/ca.key': 101159039, "error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key"
information/pki: Writing certificate to file 'icinga2-master1.vagrant.demo.icinga.com.crt'.

[dnsmichi]

I've created the Puppet hieradata profiles based on a patch in git master which lately has been reverted. Therefore the used CA does not provide a valid RSA key. See Icinga/icinga2#5555. This isn't visible in the other boxes since they're just generating the certificates at runtime.

[dnsmichi]

Fixed it with creating new static certificates.