GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Config setup
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
} Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
added 8 commits
June 20, 2023 10:04
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
…als configuration Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com>
MaGonzalMayedo
added a commit
that referenced
this pull request
Jul 3, 2023
Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>
MaGonzalMayedo
added a commit
that referenced
this pull request
Jul 3, 2023
Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos_Gonzalez_Mayedo <alemayed@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>
MaGonzalMayedo
added a commit
that referenced
this pull request
Jul 3, 2023
Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>
MaGonzalMayedo
added a commit
that referenced
this pull request
Jul 5, 2023
…-project#2922) * Elasticsearch client implementation with pit and no context search (opensearch-project#2910) Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added fixes from comments to code (including pathing and nomenclature syntax) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Refactoring config (#5) Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed deleted AwsConfig file Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed the s3 dependency from build.gradle, replaced the AwsAuth.. with AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added modifiable back_off_timer, added threshold test for back_off_timer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added fixes to gradle file, added tests to AwsConfig, and used Reflective mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added default value test to ThresholdConfig and renamed getter for maxRequestSize Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed unnecessary imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added cloudwatch-logs to settings.gradle Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added a quick fix to the back_off_time range Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> --------- Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com> Co-authored-by: Taylor Gray <tylgry@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com>
MaGonzalMayedo
added a commit
that referenced
this pull request
Jul 25, 2023
Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>
MaGonzalMayedo
added a commit
that referenced
this pull request
Jul 25, 2023
…-project#2922) * Elasticsearch client implementation with pit and no context search (opensearch-project#2910) Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added fixes from comments to code (including pathing and nomenclature syntax) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Refactoring config (#5) Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed deleted AwsConfig file Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed the s3 dependency from build.gradle, replaced the AwsAuth.. with AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added modifiable back_off_timer, added threshold test for back_off_timer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added fixes to gradle file, added tests to AwsConfig, and used Reflective mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added default value test to ThresholdConfig and renamed getter for maxRequestSize Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed unnecessary imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added cloudwatch-logs to settings.gradle Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added a quick fix to the back_off_time range Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> --------- Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com> Co-authored-by: Taylor Gray <tylgry@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com>
MaGonzalMayedo
added a commit
that referenced
this pull request
Jul 31, 2023
…ionException (opensearch-project#3023) * Elasticsearch client implementation with pit and no context search (opensearch-project#2910) Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added fixes from comments to code (including pathing and nomenclature syntax) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Refactoring config (#5) Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed deleted AwsConfig file Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed the s3 dependency from build.gradle, replaced the AwsAuth.. with AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added modifiable back_off_timer, added threshold test for back_off_timer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added fixes to gradle file, added tests to AwsConfig, and used Reflective mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added default value test to ThresholdConfig and renamed getter for maxRequestSize Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed unnecessary imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added cloudwatch-logs to settings.gradle Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added a quick fix to the back_off_time range Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added Buffer classes, ClientFactory similar to S3, and ThresholdCheck Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed unnecessary default method from ClientFactory Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added comments in Buffer Interface, change some default values to suit the plugin use case more Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed unused imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Changed the unused imports, made parameters final in the ThresholdCheck Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Made changes to the tests and the method signatures in ThresholdCheck, made fixes to gradle file to include catalog Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed unused methods/comments Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added CloudWatchLogsService, CloudWatchLogsServiceTest and RetransmissionLimitException Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed retransmission logging fixed value Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed unused imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed making ThresholdCheck public Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added fixes to ThresholdCheck and CloudWatchLogsService to decouple methods Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed syntax start import in CloudWatchLogsServiceTest Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Extracted LogPusher and SinkStopWatch classes for code cleanup. Addded fixes to variables and retry logic for InterruptExceptions Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Changed method uses in CloudWatchLogsService and removed logging the batch size in LogPusher Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added Multithreaded CloudWatchLogsDispatcher for handling various async calls to perform PLE's and added tests Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added fixesto test and defaulted the parameters in the config to CloudWatchLogs limits, customer can change this in config file Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added exponential backofftime Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed unused imports Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed up deepcopy of arraylist for service workers in CloudWatchLogsService, and fixed Log calling methods Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added CloudWatchLogsDispatcher builder pattern, fixed tests for Service and Dispatcher and modified backOffTimeBase Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Removed unused imports Signed-off-by:Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added resetBuffer method, removed unnecessary RetransmissionException, and added logString pass in parameter for staging log events. Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Started making changes to the tests to implement the new class structure (performance enhancement) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Refactored the CloudWatchLogsDispatcher into two classes with the addition of Uploader, introduced simple multithread tests for CloudWatchLogsService Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Fixed issues with locking in try block and added final multithreaded tests to the CloudWatchLogsService class Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added CloudWatchLogsMetricsTest, changed upper back off time bound and scale, and refactoring changes for better code syntax (renaming, refactoring methods for conciseness, etc...) Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Added changes to javadoc Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> * Update data-prepper-plugins/cloudwatch-logs/src/main/java/org/opensearch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java Co-authored-by: Mark Kuhn <kuhnmar@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com> * Fixed comment on CloudWatchLogsDispatcher Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> --------- Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com> Co-authored-by: Taylor Gray <tylgry@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com> Co-authored-by: Mark Kuhn <kuhnmar@amazon.com>
MaGonzalMayedo
added a commit
that referenced
this pull request
Aug 1, 2023
GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) --------- Signed-off-by: Taylor Gray <tylgry@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <alemayed@amazon.com> Signed-off-by: Marcos Gonzalez Mayedo <95880281+MaGonzalMayedo@users.noreply.github.com> Co-authored-by: Taylor Gray <tylgry@amazon.com> Co-authored-by: Marcos <alemayed@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change contains the Configuration files for the CloudWatchLogs Sink. It will act as a basis for reading the
user entered configuration yaml file.
Issues Resolved
This PR will work towards resolving CWL-Sink for Data-Prepper [Issue https://github.com/https://github.com/opensearch-project/issues/2778]
Link: opensearch-project#2778 (comment)
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.