adds new tif source config type - url download #1142

eirsep · 2024-07-06T00:50:07Z

Description

adds new tif source config type - url download
Downloads Iocs from a url of a given format and makes it available for scans and viewing Iocs

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

src/main/java/org/opensearch/securityanalytics/services/STIX2IOCFetchService.java

src/main/java/org/opensearch/securityanalytics/threatIntel/common/SourceConfigDtoValidator.java

AWSHurneyt · 2024-07-09T17:08:35Z

src/main/java/org/opensearch/securityanalytics/services/STIX2IOCFetchService.java

+                    UUID.randomUUID().toString(),
+                    iocType == null ? IOCType.ipv4_addr : IOCType.valueOf(iocType),
+                    iocValue,
+                    "high",


Nitpick: Did @amsiglan mention trying to keep the severities as number-based values?

severity is non-nullable. I didn't know what to set. I think severity should be optional.

src/main/java/org/opensearch/securityanalytics/services/STIX2IOCFetchService.java

...rg/opensearch/securityanalytics/threatIntel/service/DefaultTifSourceConfigLoaderService.java

jowg-amazon · 2024-07-09T16:33:26Z

src/main/java/org/opensearch/securityanalytics/services/STIX2IOCFetchService.java

+                }
+                break;
+            default:
+                // if the feed type doesn't match any of the supporting feed types, throw an exception


add error log

jowg-amazon · 2024-07-09T16:59:53Z

...pensearch/securityanalytics/threatIntel/transport/TransportSearchTIFSourceConfigsAction.java

-        ));
+        StepListener<Void> defaultTifConfigsLoadedListener;
+        try {
+            defaultTifConfigsLoadedListener = new StepListener<>();


For clarification, a user is not able to create a url download source config from the index tif source config rest api but it's created when user calls search/list iocs?

What gets created is system driven. We need triggers for deciding when to create default tif source configs.- search apis imply we have user engagement on the feature

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

AWSHurneyt

Approved. We discussed offline making severity nullable to address this comment; but that can be handled as a follow-up item.
https://github.com/opensearch-project/security-analytics/pull/1142/files#r1670977945

…source configs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

jowg-amazon · 2024-07-09T22:19:06Z

.../opensearch/securityanalytics/threatIntel/transport/TransportIndexTIFSourceConfigAction.java

@@ -94,6 +97,11 @@ private void retrieveLockAndCreateTIFConfig(SAIndexTIFSourceConfigRequest reques
                }
                try {
                    SATIFSourceConfigDto saTifSourceConfigDto = request.getTIFConfigDto();
+                    if (SourceConfigType.URL_DOWNLOAD.equals(saTifSourceConfigDto.getType()) || saTifSourceConfigDto.getSource() instanceof UrlDownloadSource
+                            && request.getMethod().equals(RestRequest.Method.POST)) {


Is there any logic to block users from updating this source config? I think there could still be a case if the user has the url download source already created but then tries to update it through the api.

opensearch-trigger-bot · 2024-07-09T23:56:24Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.x
# Create a new branch
git switch --create backport-1142-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 16bcef3b398f7c20d73d34d03bde80bc4ce1b421
# Push it to GitHub
git push --set-upstream origin backport-1142-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport-1142-to-2.x.

* adds new tif source config type - url download Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * set up create default tif configs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * address review comments Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add check to block create and delete operation url download type tif source configs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> (cherry picked from commit 16bcef3) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* adds new tif source config type - url download * set up create default tif configs * address review comments * add check to block create and delete operation url download type tif source configs --------- (cherry picked from commit 16bcef3) Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…) (opensearch-project#1155) * adds new tif source config type - url download * set up create default tif configs * address review comments * add check to block create and delete operation url download type tif source configs --------- (cherry picked from commit 16bcef3) Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* adds new tif source config type - url download Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * set up create default tif configs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * address review comments Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add check to block create and delete operation url download type tif source configs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* adds new tif source config type - url download (#1142) * adds new tif source config type - url download Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * set up create default tif configs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * address review comments Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add check to block create and delete operation url download type tif source configs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * Adjusted IOCTypes usage. (#1156) * Removed TODOs. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Refactored how STIX2IOCGenerator creates IOCs of specific types. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Added additional integration tests. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Changed IOCType usage as it's no longer an enum in SA commons. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Updated jar file. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed unit tests. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed tests. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Refactored build.gradle to exclude redundant dependencies from SA commons, instead of the SA commons jar being generated withhout those dependencies. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Updated jar. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Reverted changes to build.gradle. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Updated jar. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed tests. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed tests. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed tests. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed IOCType usage. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed log message. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed tests. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Addressed PR feedback. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> --------- Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Fixed jar. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: AWSHurneyt <hurneyt@amazon.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>

eirsep requested review from amsiglan, AWSHurneyt, getsaurabh02, lezzago, praveensameneni, sbcd90, jowg-amazon, engechas, goyamegh and riysaxen-amzn as code owners July 6, 2024 00:50

eirsep force-pushed the url branch 2 times, most recently from 024bd22 to bbe005d Compare July 9, 2024 09:19

AWSHurneyt reviewed Jul 9, 2024

View reviewed changes

jowg-amazon reviewed Jul 9, 2024

View reviewed changes

eirsep added 2 commits July 9, 2024 12:14

adds new tif source config type - url download

4573483

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

set up create default tif configs

00fcd33

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

eirsep force-pushed the url branch from bfe330d to c6aab82 Compare July 9, 2024 19:27

address review comments

242a7aa

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

eirsep force-pushed the url branch from c6aab82 to 242a7aa Compare July 9, 2024 19:43

AWSHurneyt previously approved these changes Jul 9, 2024

View reviewed changes

eirsep dismissed AWSHurneyt’s stale review via 11fd837 July 9, 2024 21:44

add check to block create and delete operation url download type tif …

f18da76

…source configs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

eirsep force-pushed the url branch from 11fd837 to f18da76 Compare July 9, 2024 21:52

jowg-amazon reviewed Jul 9, 2024

View reviewed changes

jowg-amazon approved these changes Jul 9, 2024

View reviewed changes

AWSHurneyt approved these changes Jul 9, 2024

View reviewed changes

eirsep merged commit 16bcef3 into opensearch-project:main Jul 9, 2024
10 of 16 checks passed

eirsep added backport 2.x backport 2.15 labels Jul 9, 2024

opensearch-trigger-bot bot added the failed backport label Jul 9, 2024

opensearch-trigger-bot bot mentioned this pull request Jul 9, 2024

[Backport 2.15] adds new tif source config type - url download #1155

Merged

AWSHurneyt mentioned this pull request Jul 12, 2024

[Backport 2.x] PR 1142 & PR 1156 #1158

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adds new tif source config type - url download #1142

adds new tif source config type - url download #1142

eirsep commented Jul 6, 2024

AWSHurneyt Jul 9, 2024

eirsep Jul 9, 2024

jowg-amazon Jul 9, 2024

jowg-amazon Jul 9, 2024

eirsep Jul 9, 2024

AWSHurneyt left a comment

jowg-amazon Jul 9, 2024

opensearch-trigger-bot bot commented Jul 9, 2024

adds new tif source config type - url download #1142

adds new tif source config type - url download #1142

Conversation

eirsep commented Jul 6, 2024

Description

Issues Resolved

Check List

AWSHurneyt Jul 9, 2024

Choose a reason for hiding this comment

eirsep Jul 9, 2024

Choose a reason for hiding this comment

jowg-amazon Jul 9, 2024

Choose a reason for hiding this comment

jowg-amazon Jul 9, 2024

Choose a reason for hiding this comment

eirsep Jul 9, 2024

Choose a reason for hiding this comment

AWSHurneyt left a comment

Choose a reason for hiding this comment

jowg-amazon Jul 9, 2024

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jul 9, 2024