GitHub-Issue#2778: Added CouldWatchLogsService, Tests and RetransmissionException #3023

MaGonzalMayedo · 2023-07-12T23:13:31Z

Description

This change adds the CloudWatchLogsService which is the main module to be used by the sink to push logs to CloudWatchLogs Services. It includes a similar iterative style to checking Data-Prepper events for validity and proceeds to wrap them up in InputLogEvent structures before batch sending them to CloudWathcLogs. This change also introduces a new exception for denoting an error that was derived from this plugin (Runtime Exception).

Issues Resolved

This PR will work towards resolving CWL-Sink for Data-Prepper [Issue https://github.com/https://github.com/https://github.com//issues/2778]
Link: #2778 (comment)

Check List

New functionality includes testing.
New functionality has been documented.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…pensearch-project#2910) Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

… syntax) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…ith AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…mer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…tive mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…xRequestSize Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…t the plugin use case more Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…, made fixes to gradle file to include catalog Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…sionLimitException Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…ethods Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…ce and Dispatcher and modified backOffTimeBase Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Signed-off-by:Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…, and added logString pass in parameter for staging log events. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

dlvenable

This looks good to me. Thanks @MaGonzalMayedo !

markkuhn · 2023-07-25T21:52:22Z

...logs/src/main/java/org/opensearch/dataprepper/plugins/sink/client/CloudWatchLogsService.java

+                .backOffTimeBase(backOffTimeBase)
+                .build();
+
+        asyncExecutor.execute(newTaskDispatcher);


I dont think creating an entire CloudWatchLogsDispatcher is the way to go. This class is fairly big and includes a lot of stuff we can run on the same thread and stuff we don't need to re-create all the time.

Agreed, I will try to minimize the the requirements for a log publisher. This way we can have more lightweight objects running on each thread.

…ure (performance enhancement) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

paggynie · 2023-07-26T18:38:10Z

...logs/src/main/java/org/opensearch/dataprepper/plugins/sink/client/CloudWatchLogsService.java

+                addToBuffer(log, logString);
+            }
+
+            bufferLock.unlock();


please put behind finally block
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantLock.html

Thank you! I missed this while reading the documentation, I have made the change in the latest revision.

…ition of Uploader, introduced simple multithread tests for CloudWatchLogsService Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…tests to the CloudWatchLogsService class Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

…d scale, and refactoring changes for better code syntax (renaming, refactoring methods for conciseness, etc...) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

...s/src/main/java/org/opensearch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java

markkuhn · 2023-07-28T21:36:46Z

...s/src/main/java/org/opensearch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java

+                    }
+                }
+            } catch (InterruptedException e) {
+                LOG.warn("Got interrupted while waiting!");


Better log needed here. This doesn't tell the customer anything. Interrupted by what? stacktrace? while waiting for what? in what class?

Thank you, I have added a more appropriate message alongside the actual thrown error message.

…rch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java Co-authored-by: Mark Kuhn <kuhnmar@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

markkuhn · 2023-07-31T18:07:21Z

🚀

graytaylor0 and others added 28 commits July 3, 2023 12:56

GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4)

4387b29

Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added fixes from comments to code (including pathing and nomenclature…

f1c25bb

… syntax) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Refactoring config (#5)

9d640e5

Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Fixed deleted AwsConfig file

b100ee3

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Removed the s3 dependency from build.gradle, replaced the AwsAuth.. w…

35859b0

…ith AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added modifiable back_off_timer, added threshold test for back_off_ti…

7040186

…mer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added fixes to gradle file, added tests to AwsConfig, and used Reflec…

6b13e21

…tive mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added default value test to ThresholdConfig and renamed getter for ma…

3bb125a

…xRequestSize Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Removed unnecessary imports

e5ee1e5

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added cloudwatch-logs to settings.gradle

e90b05a

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Merge branch 'main' into main

43a0a40

Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

Added a quick fix to the back_off_time range

40cb280

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Merge branch 'opensearch-project:main' into main

1aac686

Merge branch 'opensearch-project:main' into main

903ea26

Merge branch 'opensearch-project:main' into main

ffe5cbe

Added Buffer classes, ClientFactory similar to S3, and ThresholdCheck

b56a845

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Removed unnecessary default method from ClientFactory

b06ed0b

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added comments in Buffer Interface, change some default values to sui…

7f5a432

…t the plugin use case more Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Removed unused imports

4576899

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Changed the unused imports, made parameters final in the ThresholdCheck

a539833

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Made changes to the tests and the method signatures in ThresholdCheck…

c89ea17

…, made fixes to gradle file to include catalog Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Removed unused methods/comments

063e1d3

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added CloudWatchLogsService, CloudWatchLogsServiceTest and Retransmis…

1aad0b5

…sionLimitException Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Fixed retransmission logging fixed value

cf1f8e1

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Fixed unused imports

77f6d0f

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Fixed making ThresholdCheck public

75d90fe

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added fixes to ThresholdCheck and CloudWatchLogsService to decouple m…

5f2f511

…ethods Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

MaGonzalMayedo requested review from chenqi0805 and engechas as code owners July 12, 2023 23:13

MaGonzalMayedo force-pushed the adding_cloud_watch_logs_service branch 2 times, most recently from 42a26c1 to 3ba1011 Compare July 25, 2023 20:14

Marcos Gonzalez Mayedo added 3 commits July 25, 2023 13:21

Added CloudWatchLogsDispatcher builder pattern, fixed tests for Servi…

5971190

…ce and Dispatcher and modified backOffTimeBase Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Removed unused imports

53086c4

Signed-off-by:Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added resetBuffer method, removed unnecessary RetransmissionException…

26f18a1

…, and added logString pass in parameter for staging log events. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

MaGonzalMayedo force-pushed the adding_cloud_watch_logs_service branch from 3ba1011 to 26f18a1 Compare July 25, 2023 20:22

dlvenable previously approved these changes Jul 25, 2023

View reviewed changes

markkuhn reviewed Jul 25, 2023

View reviewed changes

Started making changes to the tests to implement the new class struct…

a5b8be7

…ure (performance enhancement) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

paggynie reviewed Jul 26, 2023

View reviewed changes

MaGonzalMayedo dismissed dlvenable’s stale review via 27ba715 July 27, 2023 19:13

MaGonzalMayedo requested review from paggynie and markkuhn July 27, 2023 19:13

MaGonzalMayedo force-pushed the adding_cloud_watch_logs_service branch from 27ba715 to 807c5f8 Compare July 27, 2023 19:15

Refactored the CloudWatchLogsDispatcher into two classes with the add…

070beef

…ition of Uploader, introduced simple multithread tests for CloudWatchLogsService Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

MaGonzalMayedo force-pushed the adding_cloud_watch_logs_service branch from 807c5f8 to 070beef Compare July 27, 2023 22:49

Marcos Gonzalez Mayedo added 2 commits July 27, 2023 16:17

Fixed issues with locking in try block and added final multithreaded …

6130b08

…tests to the CloudWatchLogsService class Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

Added CloudWatchLogsMetricsTest, changed upper back off time bound an…

69320ec

…d scale, and refactoring changes for better code syntax (renaming, refactoring methods for conciseness, etc...) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

MaGonzalMayedo force-pushed the adding_cloud_watch_logs_service branch from 16441d3 to 69320ec Compare July 28, 2023 18:56

Added changes to javadoc

993cbd0

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

MaGonzalMayedo requested a review from dlvenable July 28, 2023 20:16

markkuhn reviewed Jul 28, 2023

View reviewed changes

...s/src/main/java/org/opensearch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java Outdated Show resolved Hide resolved

markkuhn reviewed Jul 28, 2023

View reviewed changes

markkuhn approved these changes Jul 28, 2023

View reviewed changes

MaGonzalMayedo and others added 2 commits July 28, 2023 14:40

Update data-prepper-plugins/cloudwatch-logs/src/main/java/org/opensea…

2b18115

…rch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java Co-authored-by: Mark Kuhn <kuhnmar@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com>

Fixed comment on CloudWatchLogsDispatcher

eee6197

Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>

dlvenable approved these changes Jul 31, 2023

View reviewed changes

dlvenable merged commit 93d06db into opensearch-project:main Jul 31, 2023
24 checks passed

MaGonzalMayedo deleted the adding_cloud_watch_logs_service branch August 1, 2023 22:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub-Issue#2778: Added CouldWatchLogsService, Tests and RetransmissionException #3023

GitHub-Issue#2778: Added CouldWatchLogsService, Tests and RetransmissionException #3023

MaGonzalMayedo commented Jul 12, 2023 •

edited

Loading

dlvenable left a comment

markkuhn Jul 25, 2023

MaGonzalMayedo Jul 25, 2023

paggynie Jul 26, 2023

MaGonzalMayedo Jul 27, 2023

markkuhn Jul 28, 2023

MaGonzalMayedo Jul 28, 2023

markkuhn commented Jul 31, 2023

GitHub-Issue#2778: Added CouldWatchLogsService, Tests and RetransmissionException #3023

GitHub-Issue#2778: Added CouldWatchLogsService, Tests and RetransmissionException #3023

Conversation

MaGonzalMayedo commented Jul 12, 2023 • edited Loading

Description

Issues Resolved

Check List

dlvenable left a comment

Choose a reason for hiding this comment

markkuhn Jul 25, 2023

Choose a reason for hiding this comment

MaGonzalMayedo Jul 25, 2023

Choose a reason for hiding this comment

paggynie Jul 26, 2023

Choose a reason for hiding this comment

MaGonzalMayedo Jul 27, 2023

Choose a reason for hiding this comment

markkuhn Jul 28, 2023

Choose a reason for hiding this comment

MaGonzalMayedo Jul 28, 2023

Choose a reason for hiding this comment

markkuhn commented Jul 31, 2023

MaGonzalMayedo commented Jul 12, 2023 •

edited

Loading