icinga2 option generates self signed certificates that are rejected by master

[kaprizki]

Expected Behavior

After running module on the new server the new icinga2 certificates are generated and signed by the master. New client is able to connect to the master.

Current Behavior

Currently certificates are generated as self-signed so master refuses the connections with the error:

[2017-06-29 12:46:15 +0000] information/ApiListener: New client connection for identity 'i-08a8fe47981e3ecbe' from [10.254.251.93]:46288 (certificate validation failed: code 18: self signed certificate)
[2017-06-29 12:46:15 +0000] warning/JsonRpcConnection: API client disconnected for identity 'i-08a8fe47981e3ecbe'

Possible Solution

Additional step needs to be implemented to sign the certificate using token generated on master. The API call /actions/generate-ticket could be used.

curl -k -s -u "root:**"   -H 'Accept: application/json' -X POST 'https://caf-gov-ops-icinga-001-55010.cloudawarefederal.com:5665/v1/actions/generate-ticket' -d '{ "cn": "mbmif.int.netways.de" }'
{"results":[{"code":200.0,"status":"Generated PKI ticket '3fffaf7ba449db4c0d0bda96e5e8df740fe1e9c5' for common name 'mbmif.int.netways.de'.","ticket":"3fffaf7ba449db4c0d0bda96e5e8df740fe1e9c5"}]}

Steps to Reproduce (for bugs)

1. Setup new client with icinga2 options to generate icinga2 certificates.
2. Observer the log file on master

Context

I need to setup thousands of servers automatically in a clustered icinga2 environment. New servers should be able to connect to satellite/master automatically and securely without human involvement.

Your Environment

• Module version (puppet module list):
• Puppet version (puppet -V): 4.10.1
• Operating System and version: Ubuntu, Amazon, many other.

[lbetz]

That you describe is the expected behavior. For the very first time the client requires a self sign certificate to establish the connection a connection to the ca on the master to send a certificate request. The icinga2::feature::api class has to know the ticket salt from the master to authenticate its request against the master.

[lbetz]

But there is a security risk because at the moment the ticket salt is also written to the config api.conf on the client. See #325.

[kaprizki]

So - with more debugging it looks like the ticket that is generated by the module is incorrect.

In the puppet run I get:

Error: icinga2 pki request --host '**redacted**' --port 5665 --ca '/etc/icinga2/pki/ca.crt' --key '/etc/icinga2/pki/i-024db287fada9c25f.key' --cert '/etc/icinga2/pki/i-024db287fada9c25f.crt' --trustedcert '/etc/icinga2/pki/trusted-cert.crt' --ticket 'eb351e6abaa6737f14ea97076efb82b4c2b65f7f' returned 1 instead of one of [0]

However the correct ticket should be:

/etc/icinga2/pki# icinga2 pki ticket --cn i-024db287fada9c25f
3ad344a3235a01b7b343fda935663da0ca126c6b

and when I pass it to the command it works fine:

 icinga2 pki request --host '**dnsnamerdacted**' --port 5665 --ca '/etc/icinga2/pki/ca.crt' --key '/etc/icinga2/pki/i-024db287fada9c25f.key' --cert '/etc/icinga2/pki/i-024db287fada9c25f.crt' --trustedcert '/etc/icinga2/pki/trusted-cert.crt' --ticket 3ad344a3235a01b7b343fda935663da0ca126c6b
information/cli: Writing signed certificate to file '/etc/icinga2/pki/i-024db287fada9c25f.crt'.
information/cli: Writing CA certificate to file '/etc/icinga2/pki/ca.crt'.

Debugging further hoping to finally have a fix ....

I set salt to TicketSalt that is set in /etc/icinga2/constants.conf using the module

  class { '::icinga2::feature::api':
    pki             => 'icinga2',
    ca_host         => $host_id,
    ticket_salt     => 'TicketSalt',
    # when having multiple masters, you should enable:
    # accept_config => true,
    endpoints       => {
      'NodeName'    => {
      },
    },
    zones           => {
      'ZoneName' => {
        'endpoints' => ['NodeName'],
      },
    },
  }

[lbetz]

Ok, looks strange. What happens if you set the ticket_salt of icinga2::feature::api directly on the client?

[kaprizki]

I am not sure I understand the question. I set ticket_salt to the same value on both master and client using icinga2::feature::api.

They are essentially identical.

The ticket generated by the ruby code embedded in the module for some reason returns different value that the direct call to icinga2 does.

[lbetz]

Sorry I can't reproduce this. On my CentOS and Debian worked all fine. Please send your complete code.

[lbetz]

Ok, so I close this issue.

[reduardo7]

I have the same error!

[dnsmichi]

How does your configuration look like?

[dnsmichi]

I mean the Puppet configuration/manifests, in order to reproduce the error.

[deric]

I'm having similar problems. Using

openssl x509 -text -noout -in /var/lib/icinga2/certs/$(hostname -f).crt | grep CN=

turned out to be quite helpful. The command should return:

        Issuer: CN=<your master or puppet server FQDN>
        Subject: CN=<machine's FQDN>

Due to misconfiguration some of the certificates were issued by an incorrect master.

[greenaar]

Also observing the same issue. The certs are coming up self signed instead of signed by the ca. I've confirmed the ca.crt in /etc/icinga2/pki and /var/lib/icinga2/certs are the same (in this case, from the puppet master), and the client certs in both locations are self signed.

[kbabioch]

Has this been addressed at some point? I'm also running into this issue, and so are many other people (at least according to Google search results). Not sure what to do about it, no real solution has been proposed so far.

[SimonHoenscheid]

if you set $ticket_salt to the Ticket salt of the master node, the ticket on the node should be generated correctly.

[lbetz]

The procedure to configure a distributed setup is described in the docs. But the structure of the info could be better.

See:
Create a new CA with the icinga2 CLI command and a certificate signed by this new CA. This is useful especially when setting up a new Icinga 2 master.

class { '::icinga2':
constants => {
'TicketSalt' => '5a3d695b8aef8f18452fc494593056a4',
}
}

class { '::icinga2::pki::ca': }

class { '::icinga2::feature::api':
pki => 'none',
endpoints => {
'localhost' => {
'host' => 'localhost',
}
},
zones => {
'master' => {
'endpoints' => ['localhost']
}
}
}

[kbabioch]

Something about this does not work for me. I've wiped the old PKI (CA and other certificates) and now try to setup everything from scratch. The ticket salt is set, and the CA cert is correctly retrieved and shown to me in the node wizard. Then I'm asked for the salt. I'm executing the command on the master node and copy & paste the ticket salt over the the client, but after a while all I get is:

critical/cli: Could not fetch valid response. Please check the master log.
critical/cli: Failed to fetch signed certificate from master 'xxx, 5665'. Please try again.

It used to work fine in the past, so I'm pretty sure I've changed something and broke it, but there is essentially no debugging output. All I see is:

certificate validation failed: code 18: self signed certificate

The certificate in /var/lib/icinga2/certs/ is self-signed and NOT signed by the CA, e.g.:

$ openssl x509 -noout -text -in *.crt | grep Issuer
        Issuer: CN = xxx.xx

It should be signed / issued by Icinga CA, though, so something about the signing process is not working correctly :-/.

Any idea how I can debug this further? I have already the debuglog and mainlog features enabled, but they are not really helpful as far as I can tell.

[lbetz]

Please post your manifest.

…

On 25. Feb 2019, at 14:10, Karol Babioch ***@***.***> wrote: Something about this does not work for me. I've wiped the old PKI (CA and other certificates) and now try to setup everything from scratch. The ticket salt is set, and the CA cert is correctly retrieved and shown to me in the node wizard. Then I'm asked for the salt. I'm executing the command on the master node and copy & paste the ticket salt over the the client, but after a while all I get is: critical/cli: Could not fetch valid response. Please check the master log. critical/cli: Failed to fetch signed certificate from master 'xxx, 5665'. Please try again. It used to work fine in the past, so I'm pretty sure I've changed something and broke it, but there is essentially no debugging output. All I see is: certificate validation failed: code 18: self signed certificate The certificate in /var/lib/icinga2/certs/ is self-signed and NOT signed by the CA, e.g.: $ openssl x509 -noout -text -in *.crt | grep Issuer Issuer: CN = xxx.xx It should be signed / issued by Icinga CA, though, so something about the signing process is not working correctly :-/. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#332 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEDqLYAglCjUSLfY3gY2cKwLyVemV2J9ks5vQ-C2gaJpZM4OKoiW>.

[kbabioch]

Please post your manifest.

What exactly do you need and/or where do I find this manifest?

What is really strange in my case is, that it works for one node, but not for the other. I've setup the whole environment multiple times now, just to be sure, but I can reproduce this and I'm not exactly sure how to best debug this.

So what I did so far, is the following:

• Install icinga2 on mon1
• Run icinga2 node wizard on mon1 and setup a master node
• Install icinga2 on client1
• Run icinga2 node wizard on client1 and setup a client
  • After I've provided the FQDN of the master node, I'm shown the fingerprint of the root CA
  • After confirming the correctness of this data, I'm asked for a ticket
  • After providing the ticket (generated on mon1) I'm asked a couple of more questions (whether to accept commands and config from master) and for the zone names, etc.
  • Finally the configuration is changed and I can start Icinga
• Everything is working for client1
• Install icinga2 on client2
• Run icinga2 node wizard on client2 and setup a client
  • After I've provided the FQDN of the master node, I'm shown the fingerprint of the root CA
  • After confirming the correctness of this data, I'm asked for a ticket
  • After providing the ticket (generated on mon1) I'm being shown an error message:

critical/cli: Could not fetch valid response. Please check the master log.
critical/cli: Failed to fetch signed certificate from master 'mon1, 5665'. Please try again.

Not sure what is going on here, since it works for the one client, but not for the other one. It still successfully retrieved the root CA, so basic communication with the master's API is working.

[kbabioch]

I've narrowed this bug down somewhat, but only now realized that this repository is dedicated to the Puppet module, but my bug is within Icinga 2 itself: Icinga/icinga2#6981