Preserve REST client auth despite 401 response #30558

tvernum · 2018-05-14T07:55:06Z

The default behaviour for Apache HTTP client is to mimic the standard
browser behaviour of clearing the authentication cache (for a given
host) if that host responds with 401.
This behaviour is appropriate in a interactive browser environment
where the user is given the opportunity to provide alternative
credentials, but it is not the preferred behaviour for the ES REST
client.

X-Pack may respond with a 401 status if a request is made before the
node/cluster has recovered sufficient state to know how to handle the
provided authentication credentials - for example the security index
need to be recovered before we can authenticate native users.
In these cases the correct behaviour is to retry with the same
credentials (rather than discarding those credentials).

The default behaviour for Apache HTTP client is to mimic the standard browser behaviour of clearing the authentication cache (for a given host) if that host responds with 401. This behaviour is appropriate in a interactive browser environment where the user is given the opportunity to provide alternative credentials, but it is not the preferred behaviour for the ES REST client. X-Pack may respond with a 401 status if a request is made before the node/cluster has recovered sufficient state to know how to handle the provided authentication credentials - for example the security index need to be recovered before we can authenticate native users. In these cases the correct behaviour is to retry with the same credentials (rather than discarding those credentials).

elasticmachine · 2018-05-14T07:55:08Z

Pinging @elastic/es-core-infra

elasticmachine · 2018-05-14T07:55:09Z

Pinging @elastic/es-security

tvernum · 2018-05-14T07:55:40Z

Relates: #21336

javanna · 2018-05-14T09:08:19Z

client/rest/src/main/java/org/elasticsearch/client/RestClientBuilder.java

@@ -204,7 +204,8 @@ private CloseableHttpAsyncClient createHttpClient() {
            HttpAsyncClientBuilder httpClientBuilder = HttpAsyncClientBuilder.create().setDefaultRequestConfig(requestConfigBuilder.build())
                //default settings for connection pooling may be too constraining
                .setMaxConnPerRoute(DEFAULT_MAX_CONN_PER_ROUTE).setMaxConnTotal(DEFAULT_MAX_CONN_TOTAL)
-                .setSSLContext(SSLContext.getDefault());
+                .setSSLContext(SSLContext.getDefault())
+                .setTargetAuthenticationStrategy(new PersistentCredentialsAuthenticationStrategy());


is this good as a default in every case? Thinking of installation without x-pack installed and a different authentication method (e.g proxy or something along those lines)

I think it is, but I'm keen to hear other opinions.

Given that we default to preemptive authentication it seems right to default to preemptive authentication that actually works.

If the server rejects the supplied credentials, then you're going to need some custom behaviour to provide alternative credentials - which probably means a custom auth strategy, although I think it could be possible to use a custom credentials provider and rely on the clear-cache behaviour.

Note the http client has separate strategies for proxy auth and target auth, though a reverse proxy would look like the target to the client.

jaymode

I left two minor comments, OTT LGTM

jaymode · 2018-05-14T17:56:26Z

...rest/src/main/java/org/elasticsearch/client/PersistentCredentialsAuthenticationStrategy.java

+ * The default handler in Apache HTTP client mimics standard browser behaviour of clearing authentication
+ * credentials if it receives a 401 response from the server. While this can be useful for browser, it is
+ * rarely the desired behaviour with the Elasticsearch REST API.
+ * When an Elasticsearch node starts up with X-Pack enabled, the standard behaviour is to respond with a


I'd personally leave x-pack out of the javadocs, even without x-pack I think this is the right default since there needs to be a way to get the proper credentials for a host and for the rest client this needs to be provided by the user as part of configuration.

jaymode · 2018-05-14T17:56:34Z

...rest/src/main/java/org/elasticsearch/client/PersistentCredentialsAuthenticationStrategy.java

+ * what users exist.
+ * The desired behaviour under these circumstances is for the Rest client to retry with the same credentials.
+ */
+class PersistentCredentialsAuthenticationStrategy extends TargetAuthenticationStrategy {


make it final?

tvernum · 2018-05-16T02:04:01Z

@javanna Do you want to discuss this further? I'm happy to talk it through when our timezones overlap.

javanna · 2018-05-17T07:25:02Z

I am good thanks @tvernum ! Mine was just a question, I have no concerns around your change, thanks for taking care of this.

* es/master: (74 commits) Preserve REST client auth despite 401 response (#30558) [test] packaging: add windows boxes (#30402) Make xpack modules instead of a meta plugin (#30589) Mute ShrinkIndexIT [ML] DeleteExpiredDataAction should use client with origin (#30646) Reindex: Fixed typo in assertion failure message (#30619) [DOCS] Fixes list of unconverted snippets in build.gradle [DOCS] Reorganizes RBAC documentation SQL: Remove dependency for server's version from JDBC driver (#30631) Test: increase search logging for LicensingTests Adjust serialization version in IndicesOptions [TEST] Fix compilation Remove version argument in RangeFieldType (#30411) Remove unused DirectoryUtils class. (#30582) Mitigate date histogram slowdowns with non-fixed timezones. (#30534) Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg (#29594) Removes AwaitsFix on IndicesOptionsTests Template upgrades should happen in a system context (#30621) Fix bug in BucketMetrics path traversal (#30632) Fixes IndiceOptionsTests to serialise correctly (#30644) ...

* es/ccr: (75 commits) Preserve REST client auth despite 401 response (elastic#30558) [test] packaging: add windows boxes (elastic#30402) Make xpack modules instead of a meta plugin (elastic#30589) Mute ShrinkIndexIT [ML] DeleteExpiredDataAction should use client with origin (elastic#30646) Reindex: Fixed typo in assertion failure message (elastic#30619) [DOCS] Fixes list of unconverted snippets in build.gradle [DOCS] Reorganizes RBAC documentation SQL: Remove dependency for server's version from JDBC driver (elastic#30631) Test: increase search logging for LicensingTests Adjust serialization version in IndicesOptions [TEST] Fix compilation Remove version argument in RangeFieldType (elastic#30411) Remove unused DirectoryUtils class. (elastic#30582) Mitigate date histogram slowdowns with non-fixed timezones. (elastic#30534) Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg (elastic#29594) Removes AwaitsFix on IndicesOptionsTests Template upgrades should happen in a system context (elastic#30621) Fix bug in BucketMetrics path traversal (elastic#30632) Fixes IndiceOptionsTests to serialise correctly (elastic#30644) ...

…ngs-to-true * elastic/master: (25 commits) [DOCS] Replace X-Pack terms with attributes [ML] Clean left behind model state docs (elastic#30659) Correct typos filters agg docs duplicated 'bucket' word removal (elastic#30677) top_hits doc example description update (elastic#30676) [Docs] Replace InetSocketTransportAddress with TransportAdress (elastic#30673) [TEST] Account for increase in ML C++ memory usage (elastic#30675) User proper write-once semantics for GCS repository (elastic#30438) Remove bogus file accidentally added Add detailed assert message to IndexAuditUpgradeIT (elastic#30669) Adjust fast forward for token expiration test (elastic#30668) Improve explanation in rescore (elastic#30629) Deprecate `nGram` and `edgeNGram` names for ngram filters (elastic#30209) Watcher: Fix watch history template for dynamic slack attachments (elastic#30172) Fix _cluster/state to always return cluster_uuid (elastic#30656) [Tests] Add debug information to CorruptedFileIT Preserve REST client auth despite 401 response (elastic#30558) [test] packaging: add windows boxes (elastic#30402) Make xpack modules instead of a meta plugin (elastic#30589) Mute ShrinkIndexIT ...

The default behaviour for Apache HTTP client is to mimic the standard browser behaviour of clearing the authentication cache (for a given host) if that host responds with 401. This behaviour is appropriate in a interactive browser environment where the user is given the opportunity to provide alternative credentials, but it is not the preferred behaviour for the ES REST client. X-Pack may respond with a 401 status if a request is made before the node/cluster has recovered sufficient state to know how to handle the provided authentication credentials - for example the security index need to be recovered before we can authenticate native users. In these cases the correct behaviour is to retry with the same credentials (rather than discarding those credentials).

* 6.x: Mute testCorruptFileThenSnapshotAndRestore Plugins: Remove meta plugins (#30670) Upgrade to Lucene-7.4.0-snapshot-59f2b7aec2 (#30726) Docs: Add uptasticsearch to list of clients (#30738) [TEST] Reduce forecast overflow to disk test memory limit (#30727) [DOCS] Removes redundant index.asciidoc files (#30707) [DOCS] Moves X-Pack configurationg pages in table of contents (#30702) [ML][TEST] Fix bucket count assertion in ModelPlotsIT (#30717) [ML][TEST] Make AutodetectMemoryLimitIT less fragile (#30716) [Build] Add test admin when starting gradle run with trial license and [ML] provide tmp storage for forecasting and possibly any ml native jobs #30399 Tests: Fail if test watches could not be triggered (#30392) Watcher: Prevent duplicate watch triggering during upgrade (#30643) [ML] add version information in case of crash of native ML process (#30674) Add detailed assert message to IndexAuditUpgradeIT (#30669) Preserve REST client auth despite 401 response (#30558) Make TransportClusterStateAction abide to our style (#30697) [DOCS] Fixes edit URLs for stack overview (#30583) [DOCS] Add missing callout in IndicesClientDocumentationIT Backport get settings API changes to 6.x (#30494) Silence sleep based watcher test [DOCS] Replace X-Pack terms with attributes Improve explanation in rescore (#30629) [test] packaging: add windows boxes (#30402) [ML] Clean left behind model state docs (#30659) filters agg docs duplicated 'bucket' word removal (#30677) top_hits doc example description update (#30676) MovingFunction Pipeline agg backport to 6.x (#30658) [Docs] Replace InetSocketTransportAddress with TransportAdress (#30673) [TEST] Account for increase in ML C++ memory usage (#30675) User proper write-once semantics for GCS repository (#30438) Deprecate `nGram` and `edgeNGram` names for ngram filters (#30209) Watcher: Fix watch history template for dynamic slack attachments (#30172) Fix _cluster/state to always return cluster_uuid (#30656)

The default behaviour for Apache HTTP client is to mimic the standard browser behaviour of clearing the authentication cache (for a given host) if that host responds with 401. This behaviour is appropriate in a interactive browser environment where the user is given the opportunity to provide alternative credentials, but it is not the preferred behaviour for the ES REST client. X-Pack may respond with a 401 status if a request is made before the node/cluster has recovered sufficient state to know how to handle the provided authentication credentials - for example the security index need to be recovered before we can authenticate native users. In these cases the correct behaviour is to retry with the same credentials (rather than discarding those credentials).

tvernum added review :Clients/Java Low Level REST Client Minimal dependencies Java Client for Elasticsearch v7.0.0 :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) v6.4.0 labels May 14, 2018

tvernum requested review from javanna and jaymode May 14, 2018 07:55

javanna reviewed May 14, 2018

View reviewed changes

jaymode approved these changes May 14, 2018

View reviewed changes

tvernum added 3 commits May 15, 2018 12:43

Address feedback

e32d26e

Merge branch 'master' into rest-client/auth-cache

26cfd20

Merge branch 'master' into rest-client/auth-cache

06ba63f

Merge branch 'master' into rest-client/auth-cache

925679c

tvernum merged commit 8a89306 into elastic:master May 17, 2018

tvernum added backport pending and removed review labels May 17, 2018

tvernum removed the backport pending label May 18, 2018

lcawl added the >enhancement label Aug 20, 2018

jpountz removed the :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) label Jan 29, 2019

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preserve REST client auth despite 401 response #30558

Preserve REST client auth despite 401 response #30558

tvernum commented May 14, 2018

elasticmachine commented May 14, 2018

elasticmachine commented May 14, 2018

tvernum commented May 14, 2018

javanna May 14, 2018

tvernum May 14, 2018

jaymode left a comment

jaymode May 14, 2018

jaymode May 14, 2018

tvernum commented May 16, 2018

javanna commented May 17, 2018

Preserve REST client auth despite 401 response #30558

Preserve REST client auth despite 401 response #30558

Conversation

tvernum commented May 14, 2018

elasticmachine commented May 14, 2018

elasticmachine commented May 14, 2018

tvernum commented May 14, 2018

javanna May 14, 2018

Choose a reason for hiding this comment

tvernum May 14, 2018

Choose a reason for hiding this comment

jaymode left a comment

Choose a reason for hiding this comment

jaymode May 14, 2018

Choose a reason for hiding this comment

jaymode May 14, 2018

Choose a reason for hiding this comment

tvernum commented May 16, 2018

javanna commented May 17, 2018