Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding x-amz-content-sha256 header for signed requests #339

Merged
merged 8 commits into from
Jan 24, 2023
Merged

Adding x-amz-content-sha256 header for signed requests #339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
bfc1ac1
Adding X-Amz-Content-Sha256 header for signed requests
VachaShah Jan 21, 2023
81f1045
Adding CHANGELOG entry
VachaShah Jan 21, 2023
301e2e4
Adding documentation comment
VachaShah Jan 21, 2023
4d917fe
Adding tests
VachaShah Jan 23, 2023
e88486c
Addressing comments
VachaShah Jan 23, 2023
eed32d5
Addressing comments
VachaShah Jan 24, 2023
57492fc
Removing refresh policy for integ tests for Sigv4
VachaShah Jan 24, 2023
66c4eb1
Updating the developer guide
VachaShah Jan 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Fix version and build ([#254](https://github.com/opensearch-project/opensearch-java/pull/254))
- Update Gradle to 7.6 ([#309](https://github.com/opensearch-project/opensearch-java/pull/309))
- Prevent SPI calls at runtime ([#293](https://github.com/opensearch-project/opensearch-java/pull/293))
- Add support for OpenSearch Serverless ([#339](https://github.com/opensearch-project/opensearch-java/pull/339))

### Deprecated

Expand Down
4 changes: 2 additions & 2 deletions DEVELOPER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,10 @@ Run integration tests after starting OpenSearch cluster:

#### AWS Transport Integration Tests

To run integration tests for the AWS transport client, ensure working AWS credentials and specify your OpenSearch domain and region as follows:
To run integration tests for the AWS transport client, ensure working AWS credentials in `/.aws/credentials` and specify your OpenSearch domain and region as follows:

```
./gradlew integrationTest --tests "*AwsSdk2*" -Dtests.awsSdk2support.domainHost=search-...us-west-2.es.amazonaws.com -Dtests.awsSdk2support.domainRegion=us-west-2
./gradlew integrationTest --tests "*AwsSdk2*" -Dtests.awsSdk2support.domainHost=search-...us-west-2.es.amazonaws.com -Dtests.awsSdk2support.domainRegion=us-west-2 -Dtests.awsSdk2support.serviceName=es
```

For OpenSearch Serverless, change the signing service name.
Expand Down
6 changes: 4 additions & 2 deletions USER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
- [Sample data](#sample-data)
- [IndexData class](#indexdata-class)
- [Create a client](#create-a-client)
- [Create a client using `RestClientTransport`](#create-a-client-using-restclienttransport)
- [Create a client using `ApacheHttpClient5Transport`](#create-a-client-using-apachehttpclient5transport)
- [Create an index](#create-an-index)
- [Index data](#index-data)
- [Search for the documents](#search-for-the-documents)
Expand Down Expand Up @@ -175,7 +177,7 @@ DeleteIndexResponse deleteIndexResponse = client.indices().delete(deleteIndexReq

## Amazon Managed OpenSearch

Use `AwsSdk2Transport` to make requests to Amazon Managed OpenSearch.
Use `AwsSdk2Transport` to make requests to Amazon Managed OpenSearch and OpenSearch Serverless.

```java
SdkHttpClient httpClient = ApacheHttpClient.builder().build();
Expand All @@ -184,7 +186,7 @@ OpenSearchClient client = new OpenSearchClient(
new AwsSdk2Transport(
httpClient,
"search-...us-west-2.es.amazonaws.com", // OpenSearch endpoint, without https://
"es" // signing service name
"es" // signing service name, use "aoss" for OpenSearch Serverless
Region.US_WEST_2, // signing service region
AwsSdk2TransportOptions.builder().build()
)
Expand Down
3 changes: 3 additions & 0 deletions java-client/src/main/java/org/opensearch/client/transport/aws/AwsSdk2Transport.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,9 @@ private <RequestT> SdkHttpFullRequest prepareRequest(
}
req.putHeader("Content-Length", String.valueOf(body.getContentLength()));
req.contentStreamProvider(body::getInputStream);
// To add the "X-Amz-Content-Sha256" header, it needs to set as required.
// It is a required header for Amazon OpenSearch Serverless.
req.putHeader("x-amz-content-sha256", "required");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What a weird API! Took me a while to find https://github.com/aws/aws-sdk-java-v2/blob/0510a17ae41d601cf5f03e7af01e4519a6b3a744/core/auth/src/main/java/software/amazon/awssdk/auth/signer/internal/AbstractAws4Signer.java#L97 that handles this.

Is there a higher level method we can use like https://github.com/aws/aws-sdk-java-v2/blob/0510a17ae41d601cf5f03e7af01e4519a6b3a744/core/auth/src/main/java/software/amazon/awssdk/auth/signer/Aws4UnsignedPayloadSigner.java#L69 so we don't have to hard-code "required"?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it took me a while to find too. We use Aws4Signer which does not have the sign method overriden like Aws4UnsignedPayloadSigner. According to documentation, the Aws4UnsignedPayloadSigner is similar to Aws4Signer but just adds UNSIGNED-PAYLOAD when protocol is HTTPS. Can we use Aws4UnsignedPayloadSigner?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it works we sure can I think.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried the Aws4UnsignedPayloadSigner but it does not work for Amazon OpenSearch Service since it signs the payload with UNSIGNED_PAYLOAD over https protocol. Works for Amazon OpenSearch Serverless. I can PR this change on their repo may be as a new signer class but for now looks like we might have to use the hard-coded "required".

}

boolean responseCompression = Optional.ofNullable(options)
Expand Down
36 changes: 16 additions & 20 deletions ...-client/src/test/java/org/opensearch/client/opensearch/integTest/aws/AwsSdk2SearchIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,7 @@
import org.junit.Assert;
import org.opensearch.client.opensearch.OpenSearchAsyncClient;
import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch._types.OpType;
import org.opensearch.client.opensearch._types.OpenSearchException;
import org.opensearch.client.opensearch._types.Refresh;
import org.opensearch.client.opensearch.core.IndexRequest;
import org.opensearch.client.opensearch.core.IndexResponse;
import org.opensearch.client.opensearch.core.SearchResponse;
Expand Down Expand Up @@ -51,11 +49,14 @@ void testClient(boolean async) throws Exception {
final OpenSearchClient client = getClient(async, null, null);

SimplePojo doc1 = new SimplePojo("Document 1", "The text of document 1");
addDoc(client, "id1", doc1, false);
addDoc(client, "id1", doc1);
SimplePojo doc2 = new SimplePojo("Document 2", "The text of document 2");
addDoc(client, "id2", doc2, false);
addDoc(client, "id2", doc2);
SimplePojo doc3 = getLongDoc("Long Document 3", 1000000);
addDoc(client, "id3", doc3, true);
addDoc(client, "id3", doc3);

// wait for the document to index
Thread.sleep(1000);

SearchResponse<SimplePojo> response = query(client, "NotPresent", null);
Assert.assertEquals(0, response.hits().hits().size());
Expand All @@ -77,12 +78,15 @@ void testClientAsync(boolean async) throws Exception {
final OpenSearchAsyncClient client = getAsyncClient(async, null, null);

SimplePojo doc1 = new SimplePojo("Document 1", "The text of document 1");
CompletableFuture<IndexResponse> add1 = addDoc(client, "id1", doc1, false);
CompletableFuture<IndexResponse> add1 = addDoc(client, "id1", doc1);
SimplePojo doc2 = new SimplePojo("Document 2", "The text of document 2");
CompletableFuture<IndexResponse> add2 = addDoc(client, "id2", doc2, false);
CompletableFuture<IndexResponse> add2 = addDoc(client, "id2", doc2);
SimplePojo doc3 = getLongDoc("Long Document 3", 1000000);
CompletableFuture<IndexResponse> add3 = CompletableFuture.allOf(add1, add2).thenCompose(
unused -> addDoc(client, "id3", doc3, true));
unused -> addDoc(client, "id3", doc3));

// wait for the document to index
Thread.sleep(1000);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a recipe for flakyness ....


List<SearchResponse<SimplePojo>> results = add3.thenCompose(unused -> {
CompletableFuture<SearchResponse<SimplePojo>> r1 = query(client, "NotPresent", null);
Expand All @@ -107,29 +111,21 @@ void testClientAsync(boolean async) throws Exception {
}


private void addDoc(OpenSearchClient client, String id, SimplePojo doc, boolean wait) throws Exception {
private void addDoc(OpenSearchClient client, String id, SimplePojo doc) throws Exception {
IndexRequest.Builder<SimplePojo> req = new IndexRequest.Builder<SimplePojo>()
.index(TEST_INDEX)
.document(doc)
.id(id)
.opType(OpType.Index);
if (wait) {
req.refresh(Refresh.WaitFor);
}
.id(id);
client.index(req.build());
}

private CompletableFuture<IndexResponse> addDoc(
OpenSearchAsyncClient client, String id, SimplePojo doc, boolean wait
OpenSearchAsyncClient client, String id, SimplePojo doc
) {
IndexRequest.Builder<SimplePojo> req = new IndexRequest.Builder<SimplePojo>()
.index(TEST_INDEX)
.document(doc)
.id(id)
.opType(OpType.Index);
if (wait) {
req.refresh(Refresh.WaitFor);
}
.id(id);
try {
return client.index(req.build());
} catch (Exception e) {
Expand Down