Initial commit for the S3 Sink #1048 #2585

deepaksahu562 · 2023-04-24T16:28:54Z

Description

Created s3-sink related configurations.

Issues Resolved

GitHub issue: Support S3 as a Sink #1048

Check List

New functionality includes s3-sink plugin configurations.
New functionality has been documented.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

dlvenable · 2023-04-24T21:05:05Z

data-prepper-plugins/s3-sink/build.gradle

+    classpath = sourceSets.integrationTest.runtimeClasspath
+    systemProperty 'tests.s3source.bucket', System.getProperty('tests.s3source.bucket')
+    systemProperty 'tests.s3source.region', System.getProperty('tests.s3source.region')
+    systemProperty 'tests.s3source.queue.url', System.getProperty('tests.s3source.queue.url')


This should not require an SQS queue.

dlvenable · 2023-04-24T21:06:23Z

data-prepper-plugins/s3-sink/build.gradle

+    useJUnitPlatform()
+
+    classpath = sourceSets.integrationTest.runtimeClasspath
+    systemProperty 'tests.s3source.bucket', System.getProperty('tests.s3source.bucket')


Please update these two properties to tests.s3sink.

dlvenable · 2023-04-24T21:07:36Z

...java/org/opensearch/dataprepper/plugins/sink/configuration/AwsAuthenticationOptionsTest.java

+    }
+
+    @Test
+    void getAwsRegion_returns_null_when_region_is_null() throws NoSuchFieldException, IllegalAccessException {


We should have a test that validates a non-null value as well.

Added test case for non-null value.

dlvenable · 2023-04-24T21:07:50Z

...java/org/opensearch/dataprepper/plugins/sink/configuration/AwsAuthenticationOptionsTest.java

+        reflectivelySetField(awsAuthenticationOptions, "awsRegion", null);
+        assertThat(awsAuthenticationOptions.getAwsRegion(), nullValue());
+    }
+    private void reflectivelySetField(final AwsAuthenticationOptions awsAuthenticationOptions, final String fieldName,


This is available as a utility. Please use that.

dlvenable · 2023-04-24T21:08:29Z

data-prepper-plugins/s3-sink/build.gradle

+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+plugins {


Please remove these boilerplate sections: plugins, repositories.

dlvenable · 2023-04-24T21:09:32Z

...java/org/opensearch/dataprepper/plugins/sink/configuration/AwsAuthenticationOptionsTest.java

+import software.amazon.awssdk.services.sts.StsClientBuilder;
+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+
+class AwsAuthenticationOptionsTest {


This test suite should have a unit test for authenticateAwsConfiguration as well.

Added test case for authenticateAwsConfiguration.

graytaylor0 · 2023-04-24T21:31:29Z

data-prepper-plugins/s3-sink/README.md

+
+## Configuration
+
+- `aws_region` (Optional) : The AWS region to use for credentials. Defaults to [standard SDK behavior to determine the region](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html). Defaults to `none`.


These configurations are misleading, since the parameters are under aws, and not aws_region. Please follow the documentation format from the s3 source that breaks the aws options out (https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/s3-source#aws-configuration)

graytaylor0 · 2023-04-24T21:32:48Z

...ain/java/org/opensearch/dataprepper/plugins/sink/configuration/AwsAuthenticationOptions.java

+                    .build();
+
+            AssumeRoleRequest.Builder assumeRoleRequestBuilder = AssumeRoleRequest.builder()
+                    .roleSessionName("S3-Source-" + UUID.randomUUID())


"S3-sink" as the session name

Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

chenqi0805 · 2023-04-24T19:47:33Z

data-prepper-plugins/s3-sink/README.md

+
+This is the Data Prepper S3 sink plugin that sends records to an S3 bucket via S3Client.
+
+The S3 sink plugin supports OpenSearch 2.0.0 and greater.


QUES: Not following this statement. How is S3 Sink related to OpenSearch engine?

chenqi0805 · 2023-04-24T20:24:08Z

data-prepper-plugins/s3-sink/README.md

+
+## Configuration
+
+- `aws_region` (Optional) : The AWS region to use for credentials. Defaults to [standard SDK behavior to determine the region](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html). Defaults to `none`.


From the context of the PR, we are only supporting AWS Configuration in S3 sink instead of aws_ prefix. We should align the configuration parameter names with what we actually support to avoid confusion. Same applies to aws_sts_role_arn and aws_sts_header_overrides below.

Also by Defaults to none, do you mean Defaults to null, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html).? If yes please also reword that

chenqi0805 · 2023-04-24T21:08:18Z

data-prepper-plugins/s3-sink/README.md

+
+- `aws_region` (Optional) : The AWS region to use for credentials. Defaults to [standard SDK behavior to determine the region](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html). Defaults to `none`.
+
+- `aws_sts_role_arn` (Optional) : The AWS STS role to assume for requests to SQS and S3. Defaults to null, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). Defaults to `none`.


Defaults to none should be deleted

chenqi0805 · 2023-04-24T21:09:55Z

data-prepper-plugins/s3-sink/README.md

+
+- `aws_sts_role_arn` (Optional) : The AWS STS role to assume for requests to SQS and S3. Defaults to null, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). Defaults to `none`.
+
+- `aws_sts_header_overrides`  (Optional) : An optional map of header overrides to make when assuming the IAM role for the sink plugin. Defaults to `none`.


Nit: Defaults to null.

chenqi0805 · 2023-04-24T21:28:45Z

data-prepper-plugins/s3-sink/README.md

+
+- `path_prefix` (Optional) : path_prefix nothing but directory structure inside bucket in-order to store objects. Defaults to `none`.
+
+- `event_count` (Required) : An integer value indicates the maximum number of events required to ingest into s3-bucket as part of threshold.


I suggest making a threshold configuration section for the parameters under threshold: similar as https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/opensearch#aws-configuration

chenqi0805 · 2023-04-24T21:29:46Z

data-prepper-plugins/s3-sink/README.md

+
+- `maximum_size` (Optional) : A String representing the count or size of bytes required to ingest into s3-bucket as part of threshold. Defaults to `50mb`.
+
+- `event_collect` (Required) : A String representing how long events should be collected before ingest into s3-bucket as part of threshold. All Duration values are a string that represents a duration. They support ISO_8601 notation string ("PT20.345S", "PT15M", etc.) as well as simple notation Strings for seconds ("60s") and milliseconds ("1500ms").


Nit: suggest rename into event_collect_timeout or collect_timeout.

I agree - this would be a clearer name.

Modified the name as event_collect_timeout.

chenqi0805 · 2023-04-24T21:30:37Z

data-prepper-plugins/s3-sink/README.md

+pipeline:
+  ...
+  sink:
+    - s3:


Can you also include buffer_type in this example?

Added buffer_type in the yaml configuration example.

chenqi0805 · 2023-04-24T21:43:10Z

...pper-plugins/s3-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/S3SinkConfig.java

+import jakarta.validation.constraints.NotNull;
+
+/**
+ * An implementation class of s3 sink configuration


Nit: Given the class name, this java doc description seems a little redundant. But I am OK to leave it there.

Modified the description.

Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

ashoktelukuntla

Looks good

dlvenable

Thanks @deepaksahu562 !

Github-issue#1048

a1c642b

Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

deepaksahu562 requested review from chenqi0805, engechas, graytaylor0, dinujoh, cmanning09, asifsmohammed, dlvenable and oeyh as code owners April 24, 2023 16:28

deepaksahu562 added 3 commits April 24, 2023 22:36

Github-issue#1048 : s3-sink config related JUnit test-cases.

293bc94

Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

Github-issue#1048 : Added Buffer type options.

784963b

Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

Merge branch 'opensearch-project:main' into s3-sink-config

e8fe7c9

dlvenable requested changes Apr 24, 2023

View reviewed changes

dlvenable assigned graytaylor0 and dinujoh Apr 24, 2023

graytaylor0 reviewed Apr 24, 2023

View reviewed changes

deepaksahu562 added 2 commits April 25, 2023 12:12

Merge branch 'opensearch-project:main' into s3-sink-config

0605b22

Github-issue#1048 Incorporate review comments.

27ce974

Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

chenqi0805 reviewed Apr 25, 2023

View reviewed changes

deepaksahu562 added 2 commits April 26, 2023 15:17

Merge branch 'opensearch-project:main' into s3-sink-config

f9c5e46

Github-issue#1048 Incorporate review comments.

eacc6e2

Signed-off-by: Deepak Sahu <deepak.sahu562@gmail.com>

graytaylor0 approved these changes Apr 26, 2023

View reviewed changes

ashoktelukuntla approved these changes Apr 27, 2023

View reviewed changes

chenqi0805 approved these changes Apr 27, 2023

View reviewed changes

deepaksahu562 requested a review from dlvenable April 27, 2023 19:15

dlvenable approved these changes Apr 28, 2023

View reviewed changes

dlvenable merged commit 8a66e21 into opensearch-project:main Apr 28, 2023

dlvenable changed the title ~~Github-issue#1048~~ Initial commit for the S3 Sink #1048 Apr 28, 2023

dlvenable mentioned this pull request Jun 5, 2023

Support S3 as a Sink #1048

Closed


		## Configuration

		- `aws_region` (Optional) : The AWS region to use for credentials. Defaults to [standard SDK behavior to determine the region](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html). Defaults to `none`.


		This is the Data Prepper S3 sink plugin that sends records to an S3 bucket via S3Client.

		The S3 sink plugin supports OpenSearch 2.0.0 and greater.


		- `aws_region` (Optional) : The AWS region to use for credentials. Defaults to [standard SDK behavior to determine the region](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html). Defaults to `none`.

		- `aws_sts_role_arn` (Optional) : The AWS STS role to assume for requests to SQS and S3. Defaults to null, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). Defaults to `none`.


		- `aws_sts_role_arn` (Optional) : The AWS STS role to assume for requests to SQS and S3. Defaults to null, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). Defaults to `none`.

		- `aws_sts_header_overrides` (Optional) : An optional map of header overrides to make when assuming the IAM role for the sink plugin. Defaults to `none`.


		- `path_prefix` (Optional) : path_prefix nothing but directory structure inside bucket in-order to store objects. Defaults to `none`.

		- `event_count` (Required) : An integer value indicates the maximum number of events required to ingest into s3-bucket as part of threshold.


		- `maximum_size` (Optional) : A String representing the count or size of bytes required to ingest into s3-bucket as part of threshold. Defaults to `50mb`.

		- `event_collect` (Required) : A String representing how long events should be collected before ingest into s3-bucket as part of threshold. All Duration values are a string that represents a duration. They support ISO_8601 notation string ("PT20.345S", "PT15M", etc.) as well as simple notation Strings for seconds ("60s") and milliseconds ("1500ms").

Initial commit for the S3 Sink #1048 #2585

Initial commit for the S3 Sink #1048 #2585

Conversation

deepaksahu562 commented Apr 24, 2023 • edited Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashoktelukuntla left a comment

Choose a reason for hiding this comment

dlvenable left a comment

Choose a reason for hiding this comment

deepaksahu562 commented Apr 24, 2023 •

edited

Loading