Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hot-reloadable LDAP bind password #104320

Merged

Conversation

slobodanadamovic
Copy link
Contributor

@slobodanadamovic slobodanadamovic commented Jan 12, 2024

This PR enables secure_bind_password secure setting to be updated without the need to restart nodes.

The secure_bind_password must be updated on the AD/LDAP server, then changed in the Elasticsearch keystore and reloaded via reload_secure_settings API.

Note: This PR does not support grace period where both old and new passwords are active.

@slobodanadamovic slobodanadamovic added >enhancement :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) Team:Security Meta label for security team labels Jan 12, 2024
@slobodanadamovic slobodanadamovic self-assigned this Jan 12, 2024
@elasticsearchmachine
Copy link
Collaborator

Hi @slobodanadamovic, I've created a changelog YAML for you.

@@ -1960,16 +1969,6 @@ public void reload(Settings settings) throws Exception {
}
}

private void reloadSharedSecretsForJwtRealms(Settings settingsWithKeystore) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this logic to JwtRealm.

* The implementors of this interface will be called when the values of {@code SecureSetting}s should be reloaded by security plugin.
* For more information about reloading plugin secure settings, see {@link ReloadablePlugin}.
*/
public interface ReloadableSecurityComponent {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change was inspired by Tim's POC. I liked the generic approach and decided to adopt it.

@slobodanadamovic slobodanadamovic marked this pull request as ready for review January 26, 2024 10:54
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-security (Team:Security)

Copy link
Contributor

@jakelandis jakelandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (with a couple non-blocking comments)

* For additional info, see also: {@link ReloadablePlugin#reload(Settings)}.
*
* @param settings
* Settings used at the time of reloading the component.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify this abit ? i.e. Settings used at the time of reloading. If there were updates to the secure settings those updates will be present in this Settings object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will try to clarify it. I was mostly trying to echo the same description that ReloadablePlugin has, but by omitting some details it ended up not making sense.
In the end the fact is that the settings passed to the reloadable component will contain all node settings (loaded during node startup) merged with the decrypted secure ones (loaded from the keystore). The secure settings can only be accessed during the method call (should not be stored or attempted to be accessed async), because once the reload method finishes, the KeyStoreWrapper will be closed, which will wipe all secure settings.

* if the operation fails. The component should continue to operate as
* if the failing call didn't happen.
*/
void reload(Settings settings) throws Exception;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not a fan of interfaces that require explicity handling of overly generic checked exceptions. Is there any way to make this a more specific exception, or just omit the checked requirement entirely ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intention was to be consistent with higher-level ReloadablePlugin interface. But on a second thought it is not necessary to replicate it here. I will remove it.

// the existing pooled connections. LDAP connections are stateful, and once a connection is
// established and bound, it remains open until explicitly closed or until a connection
// timeout occurs. Changing the bind password on the LDAP server does not automatically
// invalidate existing connections. Hence, simply setting the new bind request is sufficient.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the addtional comments.

// established and bound, it remains open until explicitly closed or until a connection
// timeout occurs. Changing the bind password on the LDAP server does not automatically
// invalidate existing connections. Hence, simply setting the new bind request is sufficient.
connectionPool.setBindRequest(bindRequest.get());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my earlier POC, I attempted to reload the connection pool by triggering the health check.
That means that any connections that are using a now-expired password would get dropped and replaced.

Is it intentional that you don't do that?

https://github.com/tvernum/elasticsearch/blob/6b5a97cf08eaed4158f9a7a7a9a7a60cf52b9b6c/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/ldap/PoolingSessionFactory.java#L150

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is intentional. The health check does not help in detecting connections whose bind credentials have changed. Once connection is bound, it remains open even if the bind password changes. Both default health check implementation LDAPConnectionPoolHealthCheck (used when nothing is configured) and GetEntryLDAPConnectionPoolHealthCheck (used when health check is enabled) are doing only simple checks - none of these health checks can detect the password change. The only way to detect that bind password has changed is to attempt to re-bind again.

Based on my testing, after reloading bind password, the already bound connection can still be used for searching and authenticating users. The catch is that it can only be used once. This is because we call maybeForkThenBindAndRevert which under the hood calls LDAPConnectionPool#bindAndRevertAuthentication. The bindAndRevertAuthentication will attempt to re-bind (revert) the connection's authentication using old bind credentials which will fail and connection will be terminated. Polling new connections from the connection pool will use the new bind credentials.

I've also spent some time unit testing this today and running local integration tests against OpenLDAP server. The behaviour seems consistent.

@slobodanadamovic slobodanadamovic merged commit e87f58d into elastic:main Feb 2, 2024
19 checks passed
jedrazb pushed a commit to jedrazb/elasticsearch that referenced this pull request Feb 2, 2024
This PR enables `secure_bind_password` setting to be updated 
via `reload_secure_settings` API, without the need to restart nodes.

The `secure_bind_password` must be updated on the AD/LDAP 
server, changed in the Elasticsearch keystore and reloaded via 
`reload_secure_settings` API. This change does not include a 
support for the grace period, where both old and new passwords 
are active. The new password change is active immediately after 
reload and will be used when establishing new LDAP connections.
LDAP connections are stateful, and once a connection is established 
and bound, it remains open until explicitly closed or until a connection
timeout occurs. Changing the bind password on will not affect and 
invalidate existing connections.
slobodanadamovic added a commit that referenced this pull request Feb 7, 2024
This is a followup to #104320 which 
adds validation during secure setting reload of a `bind_password`. 
The reload of `secure_bind_password` will now fail with an exception instead 
of logging a deprecation warning.
felixbarny pushed a commit to felixbarny/elasticsearch that referenced this pull request Feb 8, 2024
)

This is a followup to elastic#104320 which 
adds validation during secure setting reload of a `bind_password`. 
The reload of `secure_bind_password` will now fail with an exception instead 
of logging a deprecation warning.
slobodanadamovic added a commit that referenced this pull request Feb 13, 2024
This is a followup to #104320, which updates the docs for `secure_bind_password` 
settings and marks them as reloadable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) Team:Security Meta label for security team v8.13.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants