add search ioc findings api #6

sbcd90 · 2024-06-17T03:36:18Z

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Joanne Wang <jowg@amazon.com>

…project/security-analytics into feature/threat_intel

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* Rough draft of IOC data model. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Changed IOC value from a list to a string. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Added validation for IOC type, value, and feedId fields. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Refactored IocType to for ipv4, and ipv6. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Refactored IocType. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> * Added unit tests. Signed-off-by: AWSHurneyt <hurneyt@amazon.com> --------- Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

* create tif source config api implementation Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up Signed-off-by: Joanne Wang <jowg@amazon.com> * tif/source Signed-off-by: Joanne Wang <jowg@amazon.com> * fix uri Signed-off-by: Joanne Wang <jowg@amazon.com> * comments Signed-off-by: Joanne Wang <jowg@amazon.com> * fix error message Signed-off-by: Joanne Wang <jowg@amazon.com> * moved createIndex invocation and other comments Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com>

* create tif source config api implementation Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up Signed-off-by: Joanne Wang <jowg@amazon.com> * getTIFSourceConfig API Signed-off-by: Joanne Wang <jowg@amazon.com> * clean up Signed-off-by: Joanne Wang <jowg@amazon.com> * more cleanup Signed-off-by: Joanne Wang <jowg@amazon.com> * remove runner Signed-off-by: Joanne Wang <jowg@amazon.com> * add unit serialization tests Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com>

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

eirsep · 2024-06-17T07:04:18Z

src/main/java/org/opensearch/securityanalytics/threatIntel/action/GetIocFindingsAction.java

+public class GetIocFindingsAction extends ActionType<GetIocFindingsResponse> {
+
+    public static final GetIocFindingsAction INSTANCE = new GetIocFindingsAction();
+    public static final String NAME = "cluster:admin/opensearch/securityanalytics/ioc/findings/get";


plz use prefix THREAT_INTEL_BASE_URI

this is action name.

eirsep · 2024-06-17T07:04:39Z

src/main/java/org/opensearch/securityanalytics/threatIntel/iocscan/dao/IocMatchService.java

-    public IocMatchService(final Client client, final ClusterService clusterService) {
+    private final NamedXContentRegistry xContentRegistry;
+
+    public IocMatchService(final Client client, final ClusterService clusterService, final NamedXContentRegistry xContentRegistry) {


plz rename to IocFindingService (and model to ioc findings

eirsep

plz add rest tests

eirsep · 2024-06-17T07:29:56Z

.../java/org/opensearch/securityanalytics/threatIntel/resthandler/RestGetIocFindingsAction.java

+
+    @Override
+    protected RestChannelConsumer prepareRequest(RestRequest request, NodeClient client) throws IOException {
+        String threatIntelMonitorId = request.param("monitor_id");


is this mandatory?

removed it.

eirsep · 2024-06-17T07:46:56Z

.../java/org/opensearch/securityanalytics/threatIntel/resthandler/RestGetIocFindingsAction.java

+
+        String sortString = request.param("sortString", "timestamp");
+        String sortOrder = request.param("sortOrder", "asc");
+        String missing = request.param("missing");


what is this?

https://www.elastic.co/guide/en/elasticsearch/reference/current/sort-search-results.html#_missing_values

eirsep · 2024-06-17T07:47:46Z

src/main/java/org/opensearch/securityanalytics/threatIntel/iocscan/dao/IocMatchService.java

+            }
+
+            @Override
+            public void onFailure(Exception e) {


return empty list for indexnotfound exception

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 · 2024-06-17T17:06:49Z

plz add rest tests

added a rest test.

AWSHurneyt · 2024-06-19T23:13:31Z

@sbcd90 We'd like to use this API in the ListIOCs API to return the number of matches/findings for each of the IOCs returned in the list. There could possibly be thousands of IOCs returned in that list response. Is there a way to return the findings for just those IOCs?

eirsep · 2024-06-19T23:16:21Z

@sbcd90 We'd like to use this API in the ListIOCs API to return the number of matches/findings for each of the IOCs returned in the list. There could possibly be thousands of IOCs returned in that list response. Is there a way to return the findings for just those IOCs?

+1
plz add a param for querying findings for a list of IoC's

AWSHurneyt · 2024-06-19T23:21:41Z

...va/org/opensearch/securityanalytics/threatIntel/transport/TransportGetIocFindingsAction.java

+                            QueryBuilders.boolQuery()
+                                    .must(
+                                            QueryBuilders
+                                                    .queryStringQuery(tableProp.getSearchString())


@sbcd90 I'm wondering if the way to go about the question in #6 (comment) is to supply the IOC value in the searchString? If so, this query builder would need to also search the finding iocValue field.

That might not be the best approach though since it would also return hits with tags, or names that coincidentally match the value.

Remove duplicate ioc types

jowg-amazon and others added 12 commits May 21, 2024 13:30

refactored classes (opensearch-project#1037)

0e77770

Signed-off-by: Joanne Wang <jowg@amazon.com>

refactored classes (opensearch-project#1037)

b2b156e

Signed-off-by: Joanne Wang <jowg@amazon.com>

Merge branch 'feature/threat_intel' of https://github.com/opensearch-…

cadd169

…project/security-analytics into feature/threat_intel

ioc match model (opensearch-project#1038)

76de7ad

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

Threat Intel Feed Config Model (opensearch-project#1028)

a99ca4d

ioc scan business logic

d956c12

ioc match service

e4c3cbd

test failing due to node closed

42372a0

add search ioc findings api

2976be0

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 requested a review from eirsep as a code owner June 17, 2024 03:36

eirsep reviewed Jun 17, 2024

View reviewed changes

eirsep requested changes Jun 17, 2024

View reviewed changes

eirsep reviewed Jun 17, 2024

View reviewed changes

address comments

b09afc1

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

AWSHurneyt reviewed Jun 19, 2024

View reviewed changes

eirsep changed the base branch from ioc to tim June 20, 2024 10:42

eirsep force-pushed the tim branch from 4fd974c to ec14132 Compare June 20, 2024 10:46

eirsep changed the base branch from tim to ioc June 20, 2024 10:57

eirsep force-pushed the ioc branch from 42372a0 to c29a5a8 Compare June 28, 2024 00:28

sbcd90 closed this Jul 1, 2024

eirsep pushed a commit that referenced this pull request Jul 10, 2024

Merge pull request #6 from jowg-amazon/remove_ioc_duplicates

ca1e116

Remove duplicate ioc types

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add search ioc findings api #6

add search ioc findings api #6

sbcd90 commented Jun 17, 2024

eirsep Jun 17, 2024

sbcd90 Jun 17, 2024

eirsep Jun 17, 2024

sbcd90 Jun 17, 2024

eirsep left a comment

eirsep Jun 17, 2024

sbcd90 Jun 17, 2024

eirsep Jun 17, 2024

sbcd90 Jun 17, 2024

eirsep Jun 17, 2024

sbcd90 Jun 17, 2024

sbcd90 commented Jun 17, 2024

AWSHurneyt commented Jun 19, 2024

eirsep commented Jun 19, 2024

AWSHurneyt Jun 19, 2024 •

edited

Loading

add search ioc findings api #6

add search ioc findings api #6

Conversation

sbcd90 commented Jun 17, 2024

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbcd90 commented Jun 17, 2024

AWSHurneyt commented Jun 19, 2024

eirsep commented Jun 19, 2024

AWSHurneyt Jun 19, 2024 • edited Loading

Choose a reason for hiding this comment

AWSHurneyt Jun 19, 2024 •

edited

Loading