HLRest: add xpack put user API #32332

jaymode · 2018-07-24T16:13:49Z

This commit adds a security client to the high level rest client, which
includes an implementation for the put user api. As part of these
changes, a new request and response class have been added that are
specific to the high level rest client. One change here is that the response
was previously wrapped inside a user object. The plan is to remove this
wrapping and this PR adds an unwrapped response outside of the user
object so we can remove the user object later on.

See #29827

This commit adds a security client to the high level rest client, which includes an implementation for the put user api. As part of these changes, the request and response classes were moved to the x-pack protocol project and licensed under the Apache 2 license. The PutUserRequest previously performed some validation of the username that really is more fitting for the server validation. This validation has been removed to reduce the amount of logic on the client side and also reduce the number of classes that would need to be moved to the protocol project. The removed validation now happens in the transport action. See elastic#29827

elasticmachine · 2018-07-24T16:13:51Z

Pinging @elastic/es-core-infra

hub-cap

I started a review and then realized that this contains logic that is outside of my comfort zone, especially moving the logic for user validation to transport. Please move those into a separate PR and have someone else w/ more expertise review them.

hub-cap · 2018-07-25T14:58:35Z

client/rest-high-level/src/main/java/org/elasticsearch/client/RequestConverters.java

@@ -1140,6 +1141,15 @@ static Request xpackUsage(XPackUsageRequest usageRequest) {
        return request;
    }

+    static Request putUser(PutUserRequest putUserRequest) throws IOException {


the other request converters have been prepending some xpack stuff to these methods. While im not sure im a big fan of it, its prolly best to stay consistent.

Given the recent decision to move away from xpack naming for new things, I am leaving this as is.

hub-cap · 2018-07-25T15:08:46Z

client/rest-high-level/src/main/java/org/elasticsearch/client/SecurityClient.java

+     * @return the {@link PutUserResponse} parsed
+     * @throws IOException in case there is a problem parsing the response
+     */
+    private static PutUserResponse parsePutUserResponse(XContentParser parser) throws IOException {


yuck. Is it possible to break this in the API for 7.0? Do we want to keep that custom wrapping in RestPutUserAction? Is it necessary?

In this PR I have added a solution that prevents API breakage in 6.x by keeping the wrapping AND having the value unwrapped. For 7.0 we can remove the wrapped value. Does that work for you?

hub-cap · 2018-07-25T15:19:05Z

libs/core/src/main/java/org/elasticsearch/common/CharArrays.java

@@ -1,9 +1,22 @@
 /*


im super weary of moving things that rest depends on to server. The hope is that we will depend even less on server, or not at all. Id rather see this moved into client if it needs to be, or if its just using one or two of those helpers, to just move those.

Until we can remove the need for serialization, I think this should stay as is. Once we can split our internal serialization from the request/response objects then this becomes a non-issue. Currently we already depend on StreamInput and StreamOutput, which pulls in this dependency and lucene too.

This change moves the validation for values of usernames and passwords from the request to the transport action. This is done to prevent the need to move more classes into protocol once we add this API to the high level rest client. Additionally, this resolves an issue where validation depends on settings and we always pass empty settings instead of the actual settings. Relates elastic#32332

hub-cap · 2018-08-03T00:41:21Z

I have just committed #32596 which changes the location of the xpack clients. Its very likely your tests wont compile now that the xpack portion of client.xpack().yourCommercialClient() is removed.

This change cleans up some methods in the CharArrays class from x-pack, which includes the unification of char[] to utf8 and utf8 to char[] conversions that intentionally do not use strings. There was previously an implementation in x-pack and in the reloading of secure settings. The method from the reloading of secure settings was adopted as it handled more scenarios related to the backing byte and char buffers that were used to perform the conversions. The cleaned up class is moved into libs/core to allow it to be used by requests that will be migrated to the high level rest client. Relates elastic#32332

This change moves the validation for values of usernames and passwords from the request to the transport action. This is done to prevent the need to move more classes into protocol once we add this API to the high level rest client. Additionally, this resolves an issue where validation depends on settings and we always pass empty settings instead of the actual settings. Relates #32332

This change cleans up some methods in the CharArrays class from x-pack, which includes the unification of char[] to utf8 and utf8 to char[] conversions that intentionally do not use strings. There was previously an implementation in x-pack and in the reloading of secure settings. The method from the reloading of secure settings was adopted as it handled more scenarios related to the backing byte and char buffers that were used to perform the conversions. The cleaned up class is moved into libs/core to allow it to be used by requests that will be migrated to the high level rest client. Relates #32332

jaymode · 2018-08-16T16:34:10Z

@hub-cap this is ready for review again

albertzaharovits

LGTM 😁

jkakavas

LGTM

client/rest-high-level/src/main/java/org/elasticsearch/client/security/PutUserRequest.java

bizybot · 2018-09-03T03:31:54Z

docs/java-rest/high-level/x-pack/security/put-user.asciidoc

+
+["source","java",subs="attributes,callouts,macros"]
+--------------------------------------------------
+include-tagged::{doc-tests}/WatcherDocumentationIT.java[x-pack-put-user-execute-async]


s/WatcherDocumentation/SecurityDocumentation/

good eye. we should be testing these running build_docs.pl from the docs repo

jkakavas · 2018-09-03T10:23:24Z

...t/rest-high-level/src/test/java/org/elasticsearch/client/SecurityRequestConvertersTests.java

+            expectedParams = Collections.emptyMap();
+        }
+
+        PutUserRequest putUserRequest =new PutUserRequest(username, password, roles, fullName, email, enabled, metadata, refreshPolicy);


s/=new/= new/

bizybot · 2018-09-04T02:38:05Z

client/rest-high-level/src/main/java/org/elasticsearch/client/security/PutUserRequest.java

+ * under the License.
+ */
+
+package org.elasticsearch.client.security;


maybe organize under package org.elasticsearch.client.security.user? As we will keep adding a lot of such classes in the same package for different APIs like role mapping etc.

If you think its too much to put them all in the same package, I wont enforce keeping them in one package. I certainly do not think the number of classes that currently exist in all of the action.* classes is too many for a single package here. But I dont think we need to enforce it one way or the other.

I don't think we need more packages. I think our goal should be to keep the number of packages to a minimum and at this point I do not see justification for it.

bizybot · 2018-09-04T02:38:59Z

client/rest-high-level/src/main/java/org/elasticsearch/client/security/PutUserResponse.java

+ * under the License.
+ */
+
+package org.elasticsearch.client.security;


Same here as PutUserResponse class in package org.elasticsearch.client.security.user

bizybot · 2018-09-04T13:31:48Z

...nt/rest-high-level/src/test/java/org/elasticsearch/client/ESRestHighLevelClientTestCase.java

@@ -136,4 +140,15 @@ protected static void clusterUpdateSettings(Settings persistentSettings,
        request.transientSettings(transientSettings);
        assertOK(client().performRequest(RequestConverters.clusterPutSettings(request)));
    }
+
+    @Override
+    protected Settings restClientSettings() {


For me, I think of our integration tests as examples of API usage using HLRC and I propose the use of HttpClientConfigCallback to set credentials which can be used by HttpClient when security is enabled. WDYT?

that is definitely how the docs read for configuring security on the client. I would be ++ for showing examples of that in the tests.

I am going to leave this as is. It is out of scope for this PR and if we choose to go this way, we should not use ESRestTestCase and instead show all configuration of a HLRC in its own base class.

hub-cap

Looks good. Minor comments / nits.

client/rest-high-level/src/main/java/org/elasticsearch/client/security/PutUserRequest.java

hub-cap · 2018-09-04T19:17:16Z

client/rest-high-level/src/main/java/org/elasticsearch/client/security/PutUserRequest.java

+    @Override
+    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+        builder.startObject();
+        if (password != null) {


does this ever get returned to the user here? im not familiar w security over rest.

The password is never returned to the user. We are simply transferring it to the server for hashing and storage.

hub-cap · 2018-09-04T19:22:26Z

...nt/rest-high-level/src/test/java/org/elasticsearch/client/ESRestHighLevelClientTestCase.java

@@ -136,4 +140,15 @@ protected static void clusterUpdateSettings(Settings persistentSettings,
        request.transientSettings(transientSettings);
        assertOK(client().performRequest(RequestConverters.clusterPutSettings(request)));
    }
+
+    @Override
+    protected Settings restClientSettings() {


that is definitely how the docs read for configuring security on the client. I would be ++ for showing examples of that in the tests.

hub-cap · 2018-09-04T19:23:24Z

...t/rest-high-level/src/test/java/org/elasticsearch/client/SecurityRequestConvertersTests.java

+    public void testPutUser() throws IOException {
+        final String username = randomAlphaOfLengthBetween(4, 12);
+        final char[] password = randomBoolean() ? randomAlphaOfLengthBetween(8, 12).toCharArray() : null;
+        final List<String> roles = Arrays.asList(generateRandomStringArray(randomIntBetween(2, 8), randomIntBetween(8, 16), false, true));


does the case of an empty roles list get tested?

Yes it does. The last parameter is whether or not to allow empty arrays and we set it to true

hub-cap · 2018-09-04T19:23:56Z

...t/rest-high-level/src/test/java/org/elasticsearch/client/SecurityRequestConvertersTests.java

+        final Map<String, Object> metadata;
+        if (randomBoolean()) {
+            metadata = new HashMap<>();
+            for (int i = 0; i < randomIntBetween(1, 10); i++) {


does it make sense to have it 0, 10 to produce an empty hashmap here or is this invalid?

++ I switched to 0, 10 to test a empty map

hub-cap · 2018-09-04T19:25:52Z

docs/java-rest/high-level/x-pack/security/put-user.asciidoc

+
+["source","java",subs="attributes,callouts,macros"]
+--------------------------------------------------
+include-tagged::{doc-tests}/WatcherDocumentationIT.java[x-pack-put-user-execute-async]


good eye. we should be testing these running build_docs.pl from the docs repo

hub-cap · 2018-09-04T19:37:37Z

...in/core/src/main/java/org/elasticsearch/xpack/core/security/action/user/PutUserResponse.java

@@ -49,4 +45,22 @@ public void readFrom(StreamInput in) throws IOException {
        super.readFrom(in);
        this.created = in.readBoolean();
    }
+
+    @Override
+    public boolean equals(Object o) {


hub-cap · 2018-09-04T19:40:16Z

client/rest-high-level/src/main/java/org/elasticsearch/client/security/PutUserRequest.java

+ * under the License.
+ */
+
+package org.elasticsearch.client.security;


If you think its too much to put them all in the same package, I wont enforce keeping them in one package. I certainly do not think the number of classes that currently exist in all of the action.* classes is too many for a single package here. But I dont think we need to enforce it one way or the other.

bizybot · 2018-09-05T00:15:18Z

...high-level/src/test/java/org/elasticsearch/client/documentation/SecurityDocumentationIT.java

+            //end::x-pack-put-user-execute
+
+            //tag::x-pack-put-user-response
+            boolean isCreated = response.isCreated(); // <1>


assert for verification whether it is created

bizybot

LGTM, Thank you.

This commit adds a security client to the high level rest client, which includes an implementation for the put user api. As part of these changes, a new request and response class have been added that are specific to the high level rest client. One change here is that the response was previously wrapped inside a user object. The plan is to remove this wrapping and this PR adds an unwrapped response outside of the user object so we can remove the user object later on. See #29827

...urity/src/main/java/org/elasticsearch/xpack/security/rest/action/user/RestPutUserAction.java

This change removes the wrapping of the created field in the put user response. The created field was added as a top level field in elastic#32332, while also still being wrapped within the `user` object of the response. Since the value is available in both formats in 6.x, we can remove the wrapped version for 7.0.

This change removes the wrapping of the created field in the put user response. The created field was added as a top level field in #32332, while also still being wrapped within the `user` object of the response. Since the value is available in both formats in 6.x, we can remove the wrapped version for 7.0.

albertzaharovits · 2018-10-23T09:45:13Z

client/rest-high-level/src/main/java/org/elasticsearch/client/security/PutUserRequest.java

+        }
+        if (metadata != null) {
+            builder.field("metadata", metadata);
+        }


@jaymode
enabled is missing, so this HL REST API will only be putting enabled=true users.
I have actually pulled out enabled out of the User object on the client side (which I am arduously peddling #33552 (comment)) but the PutUserRequest, containing the User object, has to have an enabled flag that it puts in the XContent.

(Adding a test in SecurityIT introduced by #33552 is one way to test for this)

jaymode added >enhancement v7.0.0 :Core/Java High Level REST Client v6.5.0 labels Jul 24, 2018

jaymode requested review from hub-cap and jkakavas July 24, 2018 16:13

Merge branch 'master' into hl_rest_put_user

3d9c92f

elasticmachine mentioned this pull request Jul 24, 2018

Java high-level REST client completeness for xpack APIs #29827

Closed

98 tasks

Merge branch 'master' into hl_rest_put_user

0d484b5

hub-cap suggested changes Jul 25, 2018

View reviewed changes

jaymode mentioned this pull request Jul 30, 2018

Move validation to server for put user requests #32471

Merged

jaymode mentioned this pull request Aug 14, 2018

Move CharArrays to core lib #32851

Merged

jaymode added 6 commits August 15, 2018 22:56

Merge branch 'master' into hl_rest_put_user

c322f08

Merge branch 'master' into hl_rest_put_user

fc093c7

revert changes to CharArrays

771b074

fixes after updating

577de8e

change put user response so we can remove wrapping in 7.0

488a0b9

revert unnecessary changes

2d5d16b

jaymode added 2 commits August 24, 2018 10:35

Merge branch 'master' into hl_rest_put_user

d743360

move requests back to core

c19ed15

albertzaharovits approved these changes Aug 30, 2018

View reviewed changes

really this time

9d29380

jkakavas approved these changes Aug 31, 2018

View reviewed changes

jkakavas reviewed Aug 31, 2018

View reviewed changes

client/rest-high-level/src/main/java/org/elasticsearch/client/security/PutUserRequest.java Show resolved Hide resolved

bizybot reviewed Sep 3, 2018

View reviewed changes

jkakavas reviewed Sep 3, 2018

View reviewed changes

bizybot reviewed Sep 4, 2018

View reviewed changes

hub-cap approved these changes Sep 4, 2018

View reviewed changes

jaymode added 5 commits September 4, 2018 14:22

Merge branch 'master' into hl_rest_put_user

04f5dee

fix copy paste error

899dd8b

add space

0a65cf1

address feedback from baz

e109dea

put user body has username

8633a9c

bizybot reviewed Sep 5, 2018

View reviewed changes

bizybot approved these changes Sep 5, 2018

View reviewed changes

jaymode added 2 commits September 5, 2018 08:41

Merge branch 'master' into hl_rest_put_user

937a48a

add assert

16014ff

jaymode merged commit ea52277 into elastic:master Sep 5, 2018

jaymode deleted the hl_rest_put_user branch September 5, 2018 17:35

albertzaharovits reviewed Sep 7, 2018

View reviewed changes

...urity/src/main/java/org/elasticsearch/xpack/security/rest/action/user/RestPutUserAction.java Show resolved Hide resolved

jaymode mentioned this pull request Sep 7, 2018

Security: remove wrapping in put user response #33512

Merged

jtibshirani mentioned this pull request Sep 17, 2018

Create a WatchStatus class for the high-level REST client. #33527

Merged

albertzaharovits reviewed Oct 23, 2018

View reviewed changes

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

HLRest: add xpack put user API #32332

HLRest: add xpack put user API #32332

Conversation

jaymode commented Jul 24, 2018 • edited Loading

elasticmachine commented Jul 24, 2018

hub-cap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hub-cap commented Aug 3, 2018

jaymode commented Aug 16, 2018

albertzaharovits left a comment

Choose a reason for hiding this comment

jkakavas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hub-cap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bizybot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaymode commented Jul 24, 2018 •

edited

Loading