-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove blocking calls and change threat intel feed flow to event driven #871
Conversation
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #871 +/- ##
============================================
- Coverage 24.82% 24.81% -0.02%
+ Complexity 1030 1029 -1
============================================
Files 277 277
Lines 12717 12702 -15
Branches 1401 1394 -7
============================================
- Hits 3157 3152 -5
+ Misses 9296 9286 -10
Partials 264 264 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
General comment on removal of the try/catches. Rest LGTM
@@ -103,24 +102,11 @@ public ThreatIntelFeedDataService( | |||
public void getThreatIntelFeedData( | |||
ActionListener<List<ThreatIntelFeedData>> listener | |||
) { | |||
try { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we no longer need this top-level try/catch? My observation has been that calls will hang it exceptions are not handled via the ActionListener
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
listener framework is event driven and no catch is required as ActionListener.onFailure()
would need to implement whatever logic was written in catch block as callback mechanism will not throw an exception
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this code throws an exception, we never make a call to ActionListener.onFailure()
. I believe the original call just hangs in this case, at least that is what I have experienced while developing in this package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me know if I am missing something
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
true.
reverting try-catch removal
if (false == detector.getThreatIntelEnabled() || iocFieldList.isEmpty()) { | ||
listener.onResponse(Collections.emptyList()); | ||
return; | ||
} | ||
|
||
CountDownLatch latch = new CountDownLatch(1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we know why the latches were initially implemented? Seems fine to remove them based on the testing performed but I'm puzzled as to why they would have been added in the first place if they are not required
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bad practice. The right construct to use is a Countdown
Countdown is a simple thread safe count-down class that in contrast to a CountDownLatch never blocks. This class is useful if a certain action has to wait for N concurrent tasks to return or a timeout to occur in order to proceed.
but safer to do it the event-driven way
@@ -116,13 +116,10 @@ private String buildQueryStringQueryWithIocList(Set<String> iocs) { | |||
* Fetches threat intel data and creates doc level queries from threat intel data | |||
*/ | |||
public void createDocLevelQueryFromThreatIntel(List<LogType.IocFields> iocFieldList, Detector detector, ActionListener<List<DocLevelQuery>> listener) { | |||
try { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar try/catch comment as below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
listener framework is event driven and no catch is required as ActionListener.onFailure()
would need to implement whatever logic was written in catch block as callback mechanism will not throw an exception
* @param jobSchedulerParameter the jobSchedulerParameter | ||
*/ | ||
public void updateJobSchedulerParameter(final TIFJobParameter jobSchedulerParameter, final ActionListener<ThreatIntelIndicesResponse> listener) { | ||
jobSchedulerParameter.setLastUpdateTime(Instant.now()); | ||
StashedThreadContext.run(client, () -> { | ||
try { | ||
if (listener != null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this listener never be null?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed it to non-null in all invocations
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
@@ -716,8 +716,6 @@ private IndexMonitorRequest createDocLevelMonitorRequest(List<Pair<String, Rule> | |||
} | |||
|
|||
private void addThreatIntelBasedDocLevelQueries(Detector detector, ActionListener<List<DocLevelQuery>> listener) { | |||
try { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Try/catch still removed here
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
…en (#871) * remove actionGet() and change threat intel feed flow to event driven Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix javadocs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert try catch removals Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * use action listener wrap() in detector threat intel code paths Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add try catch Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> (cherry picked from commit 172d58d)
…en (#871) (#876) * remove actionGet() and change threat intel feed flow to event driven Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix javadocs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert try catch removals Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * use action listener wrap() in detector threat intel code paths Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add try catch Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> (cherry picked from commit 172d58d) Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Joanne Wang <jowg@amazon.com> (cherry picked from commit 4d4f5e3) Co-authored-by: Joanne Wang <jowg@amazon.com> Reduce log level for informative message (opensearch-project#203) (opensearch-project#833) Signed-off-by: Enrico Tröger <enrico.troeger@uvena.de> Co-authored-by: Enrico Tröger <enrico.troeger@uvena.de> Updated alert creation following common-utils PR 584. (opensearch-project#837) (opensearch-project#839) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> (cherry picked from commit 8adb9c3) Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Release notes for 2.12.0 (opensearch-project#834) (opensearch-project#841) * release notes for 2.12 Signed-off-by: Joanne Wang <jowg@amazon.com> * update release notes Signed-off-by: Joanne Wang <jowg@amazon.com> * update release notes Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> (cherry picked from commit 414484a) Co-authored-by: Joanne Wang <jowg@amazon.com> Remove blocking calls and change threat intel feed flow to event driven (opensearch-project#871) (opensearch-project#876) * remove actionGet() and change threat intel feed flow to event driven Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix javadocs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert try catch removals Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * use action listener wrap() in detector threat intel code paths Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add try catch Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> (cherry picked from commit 172d58d) Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> Fail the flow the when detectot type is missing in the log types index (opensearch-project#845) (opensearch-project#857) Signed-off-by: Megha Goyal <goyamegh@amazon.com> (cherry picked from commit 8d19912) Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com> [BUG] ArrayIndexOutOfBoundsException for inconsistent detector index behavior (opensearch-project#843) (opensearch-project#858) * Catch ArrayIndexOutOfBoundsException when detector is missing Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Add a check on SearchHits.getHits() length Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Remove index out of bounds exception Signed-off-by: Megha Goyal <goyamegh@amazon.com> --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> (cherry picked from commit 0ef8543) Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com> Backport opensearch-project#873 and opensearch-project#789 (opensearch-project#895) * support object fields in aggregation based sigma rules (opensearch-project#789) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add finding mappings Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (opensearch-project#786) (opensearch-project#788) (opensearch-project#898) * field mapping changes * add integ test * turn unmappedfieldaliases as set and add integ test * add comments * fix integ tests * moved logic to method for better readability --------- Signed-off-by: Joanne Wang <jowg@amazon.com> Add throw for empty strings in rules with modifier contains, startwith, and endswith (opensearch-project#860) (opensearch-project#896) * add validation for empty strings with contains, startswith and endswith modifiers * throw exception if empty string with contains, startswith, or endswith * change var name * add modifiers to log --------- Signed-off-by: Joanne Wang <jowg@amazon.com> Add an "exists" check for "not" condition in sigma rules (opensearch-project#852) (opensearch-project#897) * test design Signed-off-by: Joanne Wang <jowg@amazon.com> * working version Signed-off-by: Joanne Wang <jowg@amazon.com> * cleaning up Signed-off-by: Joanne Wang <jowg@amazon.com> * testing Signed-off-by: Joanne Wang <jowg@amazon.com> * working version Signed-off-by: Joanne Wang <jowg@amazon.com> * working version Signed-off-by: Joanne Wang <jowg@amazon.com> * refactored querybackend Signed-off-by: Joanne Wang <jowg@amazon.com> * working on tests Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed alerting and finding tests Signed-off-by: Joanne Wang <jowg@amazon.com> * fix correlation tests Signed-off-by: Joanne Wang <jowg@amazon.com> * working all tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved test and changed alias for adldap Signed-off-by: Joanne Wang <jowg@amazon.com> * added more tests Signed-off-by: Joanne Wang <jowg@amazon.com> * cleanup code Signed-off-by: Joanne Wang <jowg@amazon.com> * remove exists flag Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> (cherry picked from commit 656a5fe) Co-authored-by: Joanne Wang <jowg@amazon.com> Add goyamegh as a maintainer (opensearch-project#868) (opensearch-project#899) Signed-off-by: Megha Goyal <goyamegh@amazon.com> Refactor invocation of Action listeners in correlations (opensearch-project#880) (opensearch-project#900) * Refactor invocation of Action listeners in correlations * Close hanging tasks in correlations workflow * Logging finding id and monitor id in error logs --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> Add search request timeouts for correlations workflows (opensearch-project#893) (opensearch-project#901) * Reinstating more leaks plugged-in for correlations workflows Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Add search timeouts to all correlation searches Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Fix logging and exception messages Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Change search timeout to 30 seconds Signed-off-by: Megha Goyal <goyamegh@amazon.com> --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> (cherry picked from commit 75c4429) Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com>
Description
Remove blocking calls and change threat intel feed flow to event driven
Issues Resolved
#870