add rollover & archival mechanism for correlation history indices #670
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sbcd90
requested review from
amsiglan,
AWSHurneyt,
getsaurabh02,
lezzago,
praveensameneni and
eirsep
as code owners
Codecov Report
@@ Coverage Diff @@
## main #670 +/- ##
============================================
- Coverage 25.18% 24.81% -0.37%
Complexity 1021 1021
============================================
Files 275 275
Lines 12458 12670 +212
Branches 1361 1390 +29
============================================
+ Hits 3137 3144 +7
- Misses 9053 9258 +205
Partials 268 268
|
sbcd90
force-pushed
the
rollover_corr
branch
2 times, most recently
from
259808f
to
b38df52
Compare
eirsep
reviewed
Oct 19, 2023
| import static org.opensearch.securityanalytics.settings.SecurityAnalyticsSettings.FINDING_HISTORY_MAX_DOCS; | ||
| import static org.opensearch.securityanalytics.settings.SecurityAnalyticsSettings.FINDING_HISTORY_RETENTION_PERIOD; | ||
| import static org.opensearch.securityanalytics.settings.SecurityAnalyticsSettings.FINDING_HISTORY_ROLLOVER_PERIOD; | ||
| import static org.opensearch.securityanalytics.settings.SecurityAnalyticsSettings.*; |
praveensameneni
previously approved these changes
Oct 25, 2023
amsiglan
previously approved these changes
Oct 25, 2023
eirsep
previously approved these changes
Oct 30, 2023
|
is this failure flaky from before or was this failing from this change? |
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
praveensameneni
approved these changes
Oct 31, 2023
AWSHurneyt
approved these changes
Oct 31, 2023
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Oct 31, 2023
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> (cherry picked from commit 24e94b4)
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Oct 31, 2023
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> (cherry picked from commit 24e94b4)
sbcd90
pushed a commit
that referenced
this pull request
Oct 31, 2023
…) (#707) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
sbcd90
pushed a commit
that referenced
this pull request
Oct 31, 2023
…) (#706) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
riysaxen-amzn
pushed a commit
to riysaxen-amzn/security-analytics
that referenced
this pull request
Mar 25, 2024
…ect#670) (opensearch-project#691) Signed-off-by: Petar Partlov <partlov@gmail.com> Signed-off-by: Petar Partlov <partlov@gmail.com> (cherry picked from commit 9d9f588426b6df8a4a084866f65532baef9268e3) Co-authored-by: Petar Partlov <partlov@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
add rollover & archival mechanism for correlation history indices.
improves scoring mechanism of correlated findings.
The scoring mechanism is dependent on the euclidean distance between 2 vectors. Earlier our vectors were dependent on log types so the dimension of these vectors were set to 100. & vectors grew in size as new log types were added.
But we eliminated it now & fixed dimension of vector to 3 which not only prevents vectors from becoming sparse but also improves correlation scores.
Issues Resolved
[List any issues this PR will resolve]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.