Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix TLS context not being updated on signed certificate messages on agents #7654

Merged
merged 2 commits into from
Nov 27, 2019

Conversation

mcktr
Copy link
Member

@mcktr mcktr commented Nov 25, 2019

This uses the class member variable for the SSL context inside the lambda expression which is captured by this.

The problem occurs with the following setup:

Master:

  • should connect to the agent

Agent

  • removed host attribute for the master
  1. Setup master
root@deb10i2m1:/# icinga2 node setup --master --disable-confd 

information/cli: Checking in existing certificates for common name 'deb10i2m1'...
information/cli: Certificates not yet generated. Running 'api setup' now.
information/cli: Generating new CA.
information/base: Writing private key to '/var/lib/icinga2/ca//ca.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/ca//ca.crt'.
information/cli: Generating new CSR in '/var/lib/icinga2/certs//deb10i2m1.csr'.
information/base: Writing private key to '/var/lib/icinga2/certs//deb10i2m1.key'.
information/base: Writing certificate signing request to '/var/lib/icinga2/certs//deb10i2m1.csr'.
information/cli: Signing CSR with CA and writing certificate to '/var/lib/icinga2/certs//deb10i2m1.crt'.
information/pki: Writing certificate to file '/var/lib/icinga2/certs//deb10i2m1.crt'.
information/cli: Copying CA certificate to '/var/lib/icinga2/certs//ca.crt'.
information/cli: Generating master configuration for Icinga 2.
information/cli: Adding new ApiUser 'root' in '/etc/icinga2/conf.d/api-users.conf'.
information/cli: Reading '/etc/icinga2/icinga2.conf'.
information/cli: Enabling the 'api' feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Generating zone and object configuration.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
information/cli: Updating the APIListener feature.
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Updating 'NodeName' constant in '/etc/icinga2/constants.conf'.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating 'ZoneName' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Updating 'TicketSalt' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Edit the api feature config file '/etc/icinga2/features-available/api.conf' and set a secure 'ticket_salt' attribute.
information/cli: Updating '"conf.d"' include in '/etc/icinga2/icinga2.conf'.
information/cli: Created backup file '/etc/icinga2/icinga2.conf.orig'.
information/cli: Disabled conf.d inclusion
information/cli: Updating '"conf.d/api-users.conf"' include in '/etc/icinga2/icinga2.conf'.
information/cli: Backup file '/etc/icinga2/icinga2.conf.orig' already exists. Skipping backup.
information/cli: Make sure to restart Icinga 2.
  1. Add agent to masters zone.conf
root@deb10i2m1:/# vim /etc/icinga2/zones.conf

/*
 * Generated by Icinga 2 node setup commands
 * on 2019-11-25 21:29:20 +0000
 */

object Endpoint "deb10i2m1" {
}

object Zone "master" {
	endpoints = [ "deb10i2m1" ]
}

object Zone "global-templates" {
	global = true
}

object Zone "director-global" {
	global = true
}

object Endpoint "deb10i2c1" {
	host = "172.17.0.3"
}

object Zone "deb10i2c1" {
	parent = "master"
	endpoints = [ "deb10i2c1" ]
}

And (re)start the master.

  1. Setup agent using the wizard
root@deb10i2c1:/# icinga2 node wizard   
    
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]:  

Starting the Agent/Satellite setup routine...

Please specify the common name (CN) [deb10i2c1]: 

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): deb10i2m1

Do you want to establish a connection to the parent node from this node? [Y/n]: 
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 172.17.0.2
Master/Satellite endpoint port [5665]: 

Add more master/satellite endpoints? [y/N]: 
Parent certificate information:

 Subject:     CN = deb10i2m1
 Issuer:      CN = Icinga CA
 Valid From:  Nov 25 21:29:20 2019 GMT
 Valid Until: Nov 21 21:29:20 2034 GMT
 Fingerprint: DF 76 B5 03 88 D0 EE 38 D1 4E 55 A2 0F B3 3D E9 97 65 78 BF 

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'deb10i2c1'): 

No ticket was specified. Please approve the certificate signing request manually
on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details).
Please specify the API bind host/port (optional):
Bind Host []: 
Bind Port []: 

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.

Local zone name [deb10i2c1]: 
Parent zone name [master]: 

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: 

Do you want to disable the inclusion of the conf.d directory [Y/n]: 
Disabling the inclusion of the conf.d directory...

Done.

Now restart your Icinga 2 daemon to finish the installation!

  1. Remove host attribute from agents zones.conf file
root@deb10i2c1:/# vim /etc/icinga2/zones.conf

/*
 * Generated by Icinga 2 node setup commands
 * on 2019-11-25 21:34:44 +0000
 */

object Endpoint "deb10i2m1" {
}

object Zone "master" {
	endpoints = [ "deb10i2m1" ]
}

object Endpoint "deb10i2c1" {
}

object Zone "deb10i2c1" {
	endpoints = [ "deb10i2c1" ]
	parent = "master"
}

object Zone "global-templates" {
	global = true
}

object Zone "director-global" {
	global = true
}
  1. Now restart the agent to apply the configuration changes

  2. Sign the CSR on the master

root@deb10i2m1:/# icinga2 ca sign 791301d98c0316a8d99e37c96e1e9d75de554d52a0305ba741bf811ddccae56d
information/cli: Signed certificate for 'CN = deb10i2c1'.
  1. Watch the agents log
[2019-11-25 21:39:14 +0000] information/ApiListener: New client connection for identity 'deb10i2m1' from [172.17.0.2]:36322
[2019-11-25 21:39:14 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'deb10i2m1'.
[2019-11-25 21:39:14 +0000] information/ApiListener: Sending config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:14 +0000] information/ApiListener: Finished sending config file updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:14 +0000] information/ApiListener: Syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:14 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:14 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:14 +0000] information/ApiListener: Sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:14 +0000] information/ApiListener: Finished sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:14 +0000] information/ApiListener: Finished syncing endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:14 +0000] information/JsonRpcConnection: Received certificate update message for CN 'deb10i2c1'
[2019-11-25 21:39:14 +0000] information/JsonRpcConnection: Updating CA certificate in '/var/lib/icinga2/certs//ca.crt'.
[2019-11-25 21:39:14 +0000] information/JsonRpcConnection: Updating client certificate for CN 'deb10i2c1' in '/var/lib/icinga2/certs//deb10i2c1.crt'.
[2019-11-25 21:39:14 +0000] information/JsonRpcConnection: Updating the client certificate for CN 'deb10i2c1' at runtime and reconnecting the endpoints.
[2019-11-25 21:39:14 +0000] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2m1'
[2019-11-25 21:39:14 +0000] warning/ApiListener: Removing API client for endpoint 'deb10i2m1'. 0 API clients left.
[2019-11-25 21:39:24 +0000] information/ApiListener: New client connection for identity 'deb10i2m1' from [172.17.0.2]:36326
[2019-11-25 21:39:24 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'deb10i2m1'.
[2019-11-25 21:39:24 +0000] information/ApiListener: Sending config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:24 +0000] information/ApiListener: Finished sending config file updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:24 +0000] information/ApiListener: Syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:24 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:24 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:24 +0000] information/ApiListener: Sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:24 +0000] information/ApiListener: Finished sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:24 +0000] information/ApiListener: Finished syncing endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:24 +0000] information/JsonRpcConnection: Received certificate update message for CN 'deb10i2c1'
[2019-11-25 21:39:24 +0000] information/JsonRpcConnection: Updating CA certificate in '/var/lib/icinga2/certs//ca.crt'.
[2019-11-25 21:39:24 +0000] information/JsonRpcConnection: Updating client certificate for CN 'deb10i2c1' in '/var/lib/icinga2/certs//deb10i2c1.crt'.
[2019-11-25 21:39:24 +0000] information/JsonRpcConnection: Updating the client certificate for CN 'deb10i2c1' at runtime and reconnecting the endpoints.
[2019-11-25 21:39:24 +0000] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2m1'
[2019-11-25 21:39:24 +0000] warning/ApiListener: Removing API client for endpoint 'deb10i2m1'. 0 API clients left.
[2019-11-25 21:39:34 +0000] information/ApiListener: New client connection for identity 'deb10i2m1' from [172.17.0.2]:36330
[2019-11-25 21:39:34 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'deb10i2m1'.
[2019-11-25 21:39:34 +0000] information/ApiListener: Sending config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:34 +0000] information/ApiListener: Finished sending config file updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:34 +0000] information/ApiListener: Syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:34 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:34 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:34 +0000] information/ApiListener: Sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:34 +0000] information/ApiListener: Finished sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:34 +0000] information/ApiListener: Finished syncing endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:34 +0000] information/JsonRpcConnection: Received certificate update message for CN 'deb10i2c1'
[2019-11-25 21:39:34 +0000] information/JsonRpcConnection: Updating CA certificate in '/var/lib/icinga2/certs//ca.crt'.
[2019-11-25 21:39:34 +0000] information/JsonRpcConnection: Updating client certificate for CN 'deb10i2c1' in '/var/lib/icinga2/certs//deb10i2c1.crt'.
[2019-11-25 21:39:34 +0000] information/JsonRpcConnection: Updating the client certificate for CN 'deb10i2c1' at runtime and reconnecting the endpoints.
[2019-11-25 21:39:34 +0000] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2m1'
[2019-11-25 21:39:34 +0000] warning/ApiListener: Removing API client for endpoint 'deb10i2m1'. 0 API clients left.
[2019-11-25 21:39:44 +0000] information/ApiListener: New client connection for identity 'deb10i2m1' from [172.17.0.2]:36332
[2019-11-25 21:39:44 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'deb10i2m1'.
[2019-11-25 21:39:44 +0000] information/ApiListener: Sending config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:44 +0000] information/ApiListener: Finished sending config file updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:44 +0000] information/ApiListener: Syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:44 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:44 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:44 +0000] information/ApiListener: Sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:44 +0000] information/ApiListener: Finished sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:44 +0000] information/ApiListener: Finished syncing endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:44 +0000] information/JsonRpcConnection: Received certificate update message for CN 'deb10i2c1'
[2019-11-25 21:39:44 +0000] information/JsonRpcConnection: Updating CA certificate in '/var/lib/icinga2/certs//ca.crt'.
[2019-11-25 21:39:44 +0000] information/JsonRpcConnection: Updating client certificate for CN 'deb10i2c1' in '/var/lib/icinga2/certs//deb10i2c1.crt'.
[2019-11-25 21:39:44 +0000] information/JsonRpcConnection: Updating the client certificate for CN 'deb10i2c1' at runtime and reconnecting the endpoints.
[2019-11-25 21:39:44 +0000] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2m1'
[2019-11-25 21:39:44 +0000] warning/ApiListener: Removing API client for endpoint 'deb10i2m1'. 0 API clients left.
[2019-11-25 21:39:54 +0000] information/ApiListener: New client connection for identity 'deb10i2m1' from [172.17.0.2]:36334
[2019-11-25 21:39:54 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'deb10i2m1'.
[2019-11-25 21:39:54 +0000] information/ApiListener: Sending config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:54 +0000] information/ApiListener: Finished sending config file updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:54 +0000] information/ApiListener: Syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:54 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'deb10i2m1'.
[2019-11-25 21:39:54 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:54 +0000] information/ApiListener: Sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:54 +0000] information/ApiListener: Finished sending replay log for endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:54 +0000] information/ApiListener: Finished syncing endpoint 'deb10i2m1' in zone 'master'.
[2019-11-25 21:39:54 +0000] information/JsonRpcConnection: Received certificate update message for CN 'deb10i2c1'
[2019-11-25 21:39:54 +0000] information/JsonRpcConnection: Updating CA certificate in '/var/lib/icinga2/certs//ca.crt'.
[2019-11-25 21:39:54 +0000] information/JsonRpcConnection: Updating client certificate for CN 'deb10i2c1' in '/var/lib/icinga2/certs//deb10i2c1.crt'.
[2019-11-25 21:39:54 +0000] information/JsonRpcConnection: Updating the client certificate for CN 'deb10i2c1' at runtime and reconnecting the endpoints.
[2019-11-25 21:39:54 +0000] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2m1'
[2019-11-25 21:39:54 +0000] warning/ApiListener: Removing API client for endpoint 'deb10i2m1'. 0 API clients left.

Notice that the agents always updates the certificate. The agent is in a loop. A restart of the agent ends the loop and the master can successfully connect and performs no further certificate updates.

While the agent was stuck in the loop I connected to the agent with openssl s_client. The used certificate was not updated, the issuer of the certificate was the agent and not the Icinga CA. After the agent restarts the correct certificate was used (issuer: Icinga CA).

fixes #7650

This makes the SSL context in ApiListener::SpawnCoroutine non-const to
address an issue when an agent recieves an signed certificate from the
master and tries to update the SSL context. This also uses the class
member variable which is captured by `this` in the lamda expression.

fixes #7650
@mcktr mcktr added bug Something isn't working area/api REST API labels Nov 25, 2019
@mcktr mcktr requested a review from dnsmichi November 25, 2019 21:47
Turns out that the switch to the class member variable inside the lambda
expression is sufficient to fix the issue.
@dnsmichi dnsmichi changed the title Fix ApiListener const SSL context Fix TLS context not being updated on signed certificate messages on agents Nov 26, 2019
@dnsmichi dnsmichi added this to the 2.12.0 milestone Nov 26, 2019
@dnsmichi
Copy link
Contributor

@Al2Klimov This also affects AddConnection() doesn't it?

IoEngine::SpawnCoroutine(io, [this, endpoint, &io, sslContext](asio::yield_context yc) {

Copy link
Member

@Al2Klimov Al2Klimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... but this is one more reason for just approving this. If this no-op fixes a problem – even better.

@@ -416,7 +416,7 @@ bool ApiListener::AddListener(const String& node, const String& service)
Log(LogInformation, "ApiListener")
<< "Started new listener on '[" << localEndpoint.address() << "]:" << localEndpoint.port() << "'";

IoEngine::SpawnCoroutine(io, [this, acceptor, sslContext](asio::yield_context yc) { ListenerCoroutineProc(yc, acceptor, sslContext); });
IoEngine::SpawnCoroutine(io, [this, acceptor](asio::yield_context yc) { ListenerCoroutineProc(yc, acceptor, m_SSLContext); });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please have a look where the previously explicitly captured variable comes from and what it's a copy of. IMO this change is a no-op...

@Al2Klimov
Copy link
Member

And yes, AddConnection() is likely to be "affected" as well.

@dnsmichi
Copy link
Contributor

@Al2Klimov Thanks a lot 👍

@mcktr Can you please take care of this in a new PR? I'll approve and merge here.

Copy link
Contributor

@dnsmichi dnsmichi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for debugging and fixing this 👍

@dnsmichi dnsmichi merged commit aee80d7 into master Nov 27, 2019
@Al2Klimov Al2Klimov deleted the bugfix/apilistener-const-sslcontext branch November 27, 2019 09:11
mcktr added a commit that referenced this pull request Nov 28, 2019
This avoids copying the TLS context in the ApiListener class and removes
the obsolete variable.

This is a follow-up from #7654
dnsmichi pushed a commit that referenced this pull request Nov 29, 2019
This avoids copying the TLS context in the ApiListener class and removes
the obsolete variable.

This is a follow-up from #7654
@lippserd lippserd mentioned this pull request Dec 6, 2019
8 tasks
N-o-X pushed a commit that referenced this pull request Sep 11, 2020
This avoids copying the TLS context in the ApiListener class and removes
the obsolete variable.

This is a follow-up from #7654
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api REST API bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Connection-less mode: Restart of Agent is required after certificate is signed
3 participants