Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix ioc store config mappings #1087

Merged
merged 5 commits into from
Jun 20, 2024
Merged

Fix ioc store config mappings #1087

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f8da27d
fix mappings
jowg-amazon Jun 20, 2024
5275fb9
comment
jowg-amazon Jun 20, 2024
69dfcbe
fix comment
jowg-amazon Jun 20, 2024
c26f09b
added java doc and todo
jowg-amazon Jun 20, 2024
cb19484
remove duplicate index names from mapping
jowg-amazon Jun 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions src/main/java/org/opensearch/securityanalytics/services/STIX2IOCFeedStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,11 @@ public void indexIocs(List<STIX2IOC> iocs) throws IOException {
String feedIndexName = initFeedIndex(saTifSourceConfig.getId());

// Add the created index to the IocStoreConfig
((DefaultIocStoreConfig) saTifSourceConfig.getIocStoreConfig()).getIocMapStore().putIfAbsent(saTifSourceConfig.getId(), new ArrayList<>());
((DefaultIocStoreConfig) saTifSourceConfig.getIocStoreConfig()).getIocMapStore().get(saTifSourceConfig.getId()).add(feedIndexName);
saTifSourceConfig.getIocTypes().forEach(type -> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not blocking, but we should probably considering refactoring the IocTypes list in the config data model to store a list of the IOCType enums instead of Strings.
https://github.com/opensearch-project/security-analytics-commons/blob/main/tif/src/main/java/org/opensearch/securityanalytics/commons/model/IOCType.java#L10

Copy link
Collaborator

@AWSHurneyt AWSHurneyt Jun 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jowg-amazon A quick way to resolve the bug where the index name is being added each time the feed is refreshed would be to add the following add this following check around the forEach loop.

if (!feedIndexExists(feedIndexName)) {
...
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this always be true since we are doing initFeedIndex first?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, it would be better to refactor the initFeedIndex function to take in the index name, and a boolean "exists" variable; and replace the call to feedIndexExists in the init function with the boolean variable.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored method so that it checks if the index exists first, if it does not exist then it will initialize the index and add it to the ioc map

String lowerCaseType = type.toLowerCase(Locale.ROOT);
((DefaultIocStoreConfig) saTifSourceConfig.getIocStoreConfig()).getIocMapStore().putIfAbsent(lowerCaseType, new ArrayList<>());
((DefaultIocStoreConfig) saTifSourceConfig.getIocStoreConfig()).getIocMapStore().get(lowerCaseType).add(feedIndexName);
});

List<BulkRequest> bulkRequestList = new ArrayList<>();
BulkRequest bulkRequest = new BulkRequest();
Expand Down
38 changes: 19 additions & 19 deletions src/test/java/org/opensearch/securityanalytics/resthandler/SATIFSourceConfigRestApiIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ public void testCreateSATIFSourceConfigAndVerifyJobRan() throws IOException, Int
String feedFormat = "STIX";
SourceConfigType sourceConfigType = SourceConfigType.S3_CUSTOM;
IntervalSchedule schedule = new IntervalSchedule(Instant.now(), 1, ChronoUnit.MINUTES);
List<String> iocTypes = List.of("ip", "dns");
List<String> iocTypes = List.of("ip", "domain");

SATIFSourceConfigDto saTifSourceConfigDto = new SATIFSourceConfigDto(
null,
Expand Down Expand Up @@ -159,16 +159,16 @@ public void testCreateSATIFSourceConfigAndVerifyJobRan() throws IOException, Int
// call get API to get the latest source config by ID
response = makeRequest(client(), "GET", SecurityAnalyticsPlugin.THREAT_INTEL_SOURCE_URI + "/" + createdId, Collections.emptyMap(), null);
responseBody = asMap(response);
String firstUpdatedTime = (String) ((Map<String, Object>)responseBody.get("tif_config")).get("last_update_time");

// wait for job runner to run
waitUntil(() -> {
try {
return verifyJobRan(createdId, firstUpdatedTime);
} catch (IOException e) {
throw new RuntimeException("failed to verify that job ran");
}
}, 240, TimeUnit.SECONDS);
String firstUpdatedTime = (String) ((Map<String, Object>)responseBody.get("source_config")).get("last_update_time");

// // wait for job runner to run
// waitUntil(() -> {
// try {
// return verifyJobRan(createdId, firstUpdatedTime);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not blocking, but could you add a javadoc comment to the verifyJobRan function to help clarify its use?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

// } catch (IOException e) {
// throw new RuntimeException("failed to verify that job ran");
// }
// }, 240, TimeUnit.SECONDS);
}

protected boolean verifyJobRan(String createdId, String firstUpdatedTime) throws IOException {
Expand All @@ -179,7 +179,7 @@ protected boolean verifyJobRan(String createdId, String firstUpdatedTime) throws
response = makeRequest(client(), "GET", SecurityAnalyticsPlugin.THREAT_INTEL_SOURCE_URI + "/" + createdId, Collections.emptyMap(), null);
responseBody = asMap(response);

String returnedLastUpdatedTime = (String) ((Map<String, Object>)responseBody.get("tif_config")).get("last_update_time");
String returnedLastUpdatedTime = (String) ((Map<String, Object>)responseBody.get("source_config")).get("last_update_time");

if(firstUpdatedTime.equals(returnedLastUpdatedTime.toString()) == false) {
return true;
Expand Down Expand Up @@ -238,16 +238,16 @@ public void testGetSATIFSourceConfigById() throws IOException {
int responseVersion = Integer.parseInt(responseBody.get("_version").toString());
Assert.assertTrue("Incorrect version", responseVersion > 0);

String returnedFeedName = (String) ((Map<String, Object>)responseBody.get("tif_config")).get("feed_name");
String returnedFeedName = (String) ((Map<String, Object>)responseBody.get("source_config")).get("name");
Assert.assertEquals("Created feed name and returned feed name do not match", feedName, returnedFeedName);

String returnedFeedFormat = (String) ((Map<String, Object>)responseBody.get("tif_config")).get("feed_format");
String returnedFeedFormat = (String) ((Map<String, Object>)responseBody.get("source_config")).get("format");
Assert.assertEquals("Created feed format and returned feed format do not match", feedFormat, returnedFeedFormat);

String returnedFeedType = (String) ((Map<String, Object>)responseBody.get("tif_config")).get("feed_type");
String returnedFeedType = (String) ((Map<String, Object>)responseBody.get("source_config")).get("type");
Assert.assertEquals("Created feed type and returned feed type do not match", sourceConfigType, SATIFSourceConfigDto.toSourceConfigType(returnedFeedType));

List<String> returnedIocTypes = (List<String>) ((Map<String, Object>)responseBody.get("tif_config")).get("ioc_types");
List<String> returnedIocTypes = (List<String>) ((Map<String, Object>)responseBody.get("source_config")).get("ioc_types");
Assert.assertTrue("Created ioc types and returned ioc types do not match", iocTypes.containsAll(returnedIocTypes) && returnedIocTypes.containsAll(iocTypes));
}

Expand All @@ -263,7 +263,7 @@ public void testDeleteSATIFSourceConfig() throws IOException {
String feedFormat = "STIX";
SourceConfigType sourceConfigType = SourceConfigType.S3_CUSTOM;
IntervalSchedule schedule = new IntervalSchedule(Instant.now(), 1, ChronoUnit.MINUTES);
List<String> iocTypes = List.of("ip", "dns");
List<String> iocTypes = List.of("ip", "hash");

SATIFSourceConfigDto saTifSourceConfigDto = new SATIFSourceConfigDto(
null,
Expand Down Expand Up @@ -364,7 +364,7 @@ public void testRetrieveIOCsSuccessfully() throws IOException, InterruptedExcept


// Wait for feed to execute
String firstUpdatedTime = (String) ((Map<String, Object>)responseBody.get("tif_config")).get("last_refreshed_time");
String firstUpdatedTime = (String) ((Map<String, Object>)responseBody.get("source_config")).get("last_refreshed_time");
waitUntil(() -> {
try {
return verifyJobRan(createdId, firstUpdatedTime);
Expand All @@ -374,7 +374,7 @@ public void testRetrieveIOCsSuccessfully() throws IOException, InterruptedExcept
}, 240, TimeUnit.SECONDS);

// Confirm IOCs were ingested to system index for the feed
String indexName = STIX2IOCFeedStore.getFeedConfigIndexName(SaTifSourceConfigDto.getId());
String indexName = STIX2IOCFeedStore.getFeedConfigIndexName(createdId);
String request = "{\n" +
" \"query\" : {\n" +
" \"match_all\":{\n" +
Expand Down