Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix open connections when agent waits for CA approval #7686

Merged
merged 1 commit into from
Dec 9, 2019
Merged

Fix open connections when agent waits for CA approval #7686

merged 1 commit into from
Dec 9, 2019

Conversation

mcktr
Copy link
Member

@mcktr mcktr commented Dec 3, 2019

This closes the agent connection when the certificate sign requests
waits for CA approval.

For testing this use the setup from #7680 (comment).

Monitor the open files with:

(for pid in $(pidof icinga2); do lsof -p $pid; done) | wc -l

The amount should be constant.

Before:

[2019-12-03 20:10:13 +0000] information/ApiListener: Reconnecting to endpoint 'deb10i2a1' via host '172.17.0.3' and port '5665'
[2019-12-03 20:10:13 +0000] warning/ApiListener: Certificate validation failed for endpoint 'deb10i2a1': code 18: self signed certificate
[2019-12-03 20:10:13 +0000] information/ApiListener: New client connection for identity 'deb10i2a1' to [172.17.0.3]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:10:13 +0000] information/ApiListener: Finished reconnecting to endpoint 'deb10i2a1' via host '172.17.0.3' and port '5665'
[2019-12-03 20:10:13 +0000] information/JsonRpcConnection: Received certificate request for CN 'deb10i2a1' not signed by our CA.
[2019-12-03 20:10:13 +0000] information/JsonRpcConnection: Certificate request for CN 'deb10i2a1' is pending. Waiting for approval.
[2019-12-03 20:10:23 +0000] information/ApiListener: Reconnecting to endpoint 'deb10i2a1' via host '172.17.0.3' and port '5665'
[2019-12-03 20:10:23 +0000] warning/ApiListener: Certificate validation failed for endpoint 'deb10i2a1': code 18: self signed certificate
[2019-12-03 20:10:23 +0000] information/ApiListener: New client connection for identity 'deb10i2a1' to [172.17.0.3]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:10:23 +0000] information/ApiListener: Finished reconnecting to endpoint 'deb10i2a1' via host '172.17.0.3' and port '5665'
[2019-12-03 20:10:23 +0000] information/JsonRpcConnection: Received certificate request for CN 'deb10i2a1' not signed by our CA.
[2019-12-03 20:10:23 +0000] information/JsonRpcConnection: Certificate request for CN 'deb10i2a1' is pending. Waiting for approval.
[2019-12-03 20:10:33 +0000] information/ApiListener: Reconnecting to endpoint 'deb10i2a1' via host '172.17.0.3' and port '5665'
[2019-12-03 20:10:33 +0000] warning/ApiListener: Certificate validation failed for endpoint 'deb10i2a1': code 18: self signed certificate
[2019-12-03 20:10:33 +0000] information/ApiListener: New client connection for identity 'deb10i2a1' to [172.17.0.3]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:10:33 +0000] information/ApiListener: Finished reconnecting to endpoint 'deb10i2a1' via host '172.17.0.3' and port '5665'
[2019-12-03 20:10:33 +0000] information/JsonRpcConnection: Received certificate request for CN 'deb10i2a1' not signed by our CA.
[2019-12-03 20:10:33 +0000] information/JsonRpcConnection: Certificate request for CN 'deb10i2a1' is pending. Waiting for approval.
[2019-12-03 20:10:43 +0000] information/ApiListener: Reconnecting to endpoint 'deb10i2a1' via host '172.17.0.3' and port '5665'
[2019-12-03 20:10:43 +0000] warning/ApiListener: Certificate validation failed for endpoint 'deb10i2a1': code 18: self signed certificate
[2019-12-03 20:10:43 +0000] information/ApiListener: New client connection for identity 'deb10i2a1' to [172.17.0.3]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:10:43 +0000] information/ApiListener: Finished reconnecting to endpoint 'deb10i2a1' via host '172.17.0.3' and port '5665'
[2019-12-03 20:10:43 +0000] information/JsonRpcConnection: Received certificate request for CN 'deb10i2a1' not signed by our CA.
[2019-12-03 20:10:43 +0000] information/JsonRpcConnection: Certificate request for CN 'deb10i2a1' is pending. Waiting for approval.

After:

[2019-12-03 20:54:46 +0100] information/ApiListener: Reconnecting to endpoint 'deb10i2c1' via host '172.17.0.2' and port '5665'
[2019-12-03 20:54:46 +0100] warning/ApiListener: Certificate validation failed for endpoint 'deb10i2c1': code 18: self signed certificate
[2019-12-03 20:54:46 +0100] information/ApiListener: New client connection for identity 'deb10i2c1' to [172.17.0.2]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:54:46 +0100] information/ApiListener: Finished reconnecting to endpoint 'deb10i2c1' via host '172.17.0.2' and port '5665'
[2019-12-03 20:54:46 +0100] information/JsonRpcConnection: Received certificate request for CN 'deb10i2c1' not signed by our CA.
[2019-12-03 20:54:46 +0100] information/JsonRpcConnection: Certificate request for CN 'deb10i2c1' is pending. Waiting for approval.
[2019-12-03 20:54:46 +0100] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2c1'
[2019-12-03 20:54:46 +0100] information/ConfigObject: Dumping program state to file '/usr/local/icinga2/var/lib/icinga2/icinga2.state'
[2019-12-03 20:54:56 +0100] information/ApiListener: New client connection for identity 'deb10i2c1' from [172.17.0.2]:53020 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:54:56 +0100] information/JsonRpcConnection: Received certificate request for CN 'deb10i2c1' not signed by our CA.
[2019-12-03 20:54:56 +0100] information/JsonRpcConnection: Certificate request for CN 'deb10i2c1' is pending. Waiting for approval.
[2019-12-03 20:54:56 +0100] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2c1'
[2019-12-03 20:54:56 +0100] information/ApiListener: Reconnecting to endpoint 'deb10i2c1' via host '172.17.0.2' and port '5665'
[2019-12-03 20:54:56 +0100] warning/ApiListener: Certificate validation failed for endpoint 'deb10i2c1': code 18: self signed certificate
[2019-12-03 20:54:56 +0100] information/ApiListener: New client connection for identity 'deb10i2c1' to [172.17.0.2]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:54:56 +0100] information/ApiListener: Finished reconnecting to endpoint 'deb10i2c1' via host '172.17.0.2' and port '5665'
[2019-12-03 20:54:56 +0100] information/JsonRpcConnection: Received certificate request for CN 'deb10i2c1' not signed by our CA.
[2019-12-03 20:54:56 +0100] information/JsonRpcConnection: Certificate request for CN 'deb10i2c1' is pending. Waiting for approval.
[2019-12-03 20:54:56 +0100] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2c1'
[2019-12-03 20:55:06 +0100] information/ApiListener: New client connection for identity 'deb10i2c1' from [172.17.0.2]:53026 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:55:06 +0100] information/JsonRpcConnection: Received certificate request for CN 'deb10i2c1' not signed by our CA.
[2019-12-03 20:55:06 +0100] information/JsonRpcConnection: Certificate request for CN 'deb10i2c1' is pending. Waiting for approval.
[2019-12-03 20:55:06 +0100] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2c1'
[2019-12-03 20:55:06 +0100] information/ApiListener: Reconnecting to endpoint 'deb10i2c1' via host '172.17.0.2' and port '5665'
[2019-12-03 20:55:06 +0100] warning/ApiListener: Certificate validation failed for endpoint 'deb10i2c1': code 18: self signed certificate
[2019-12-03 20:55:06 +0100] information/ApiListener: New client connection for identity 'deb10i2c1' to [172.17.0.2]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-12-03 20:55:06 +0100] information/ApiListener: Finished reconnecting to endpoint 'deb10i2c1' via host '172.17.0.2' and port '5665'
[2019-12-03 20:55:06 +0100] information/JsonRpcConnection: Received certificate request for CN 'deb10i2c1' not signed by our CA.
[2019-12-03 20:55:06 +0100] information/JsonRpcConnection: Certificate request for CN 'deb10i2c1' is pending. Waiting for approval.
[2019-12-03 20:55:06 +0100] warning/JsonRpcConnection: API client disconnected for identity 'deb10i2c1'

fixes #7680

@mcktr
Fix open connections when agent waits for CA approval
9d55a82 
This closes the agent connection when the certificate sign requests
waits for CA approval.

refs #7680
@mcktr mcktr added bug Something isn't working area/distributed Distributed monitoring (master, satellites, clients) labels Dec 3, 2019
@mcktr mcktr requested a review from dnsmichi December 3, 2019 20:27
@dnsmichi
Copy link
Contributor

dnsmichi commented Dec 4, 2019

Thanks a lot 👍

@lippserd @Al2Klimov This may have an influence on the JsonRCP problem, e.g. when you have thousands of agents which request signed certificates, and they re-connect all the time thus resulting in many open connections. @Al2Klimov please test in this specific regard, if you can better reproduce the crash.

@dnsmichi dnsmichi added this to the 2.12.0 milestone Dec 5, 2019
@lippserd lippserd mentioned this pull request Dec 6, 2019
Closed
8 tasks
@dnsmichi dnsmichi merged commit f47142c into master Dec 9, 2019
@dnsmichi dnsmichi deleted the bugfix/open-connections branch December 9, 2019 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dnsmichi dnsmichi dnsmichi approved these changes

Assignees
No one assigned
Labels
area/distributed Distributed monitoring (master, satellites, clients) bug Something isn't working
Projects
None yet
Milestone
2.12.0
Development

Successfully merging this pull request may close these issues.

Icinga 2 reconnects in a loop for self-signed certificates
2 participants
@mcktr @dnsmichi