Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for OCSF field mapping and correlation engine API #4549

Merged
merged 18 commits into from
Jul 17, 2023

Conversation

cwillum
Copy link
Contributor

@cwillum cwillum commented Jul 12, 2023

Description

Adds documentation for OCSF field mapping types, and adds APIs for correlation engine.

Issues Resolved

Fixes #4500

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum cwillum added 2 - In progress Issue/PR: The issue or PR is in progress. release-notes PR: Include this PR in the automated release notes security-analytics v2.9.0 labels Jul 12, 2023
@cwillum cwillum self-assigned this Jul 12, 2023
Signed-off-by: cwillum <cwmmoore@amazon.com>
Signed-off-by: cwillum <cwmmoore@amazon.com>
@sbcd90
Copy link
Contributor

sbcd90 commented Jul 13, 2023

the ocsf logs integration piece looks good but correlation engine api documentation is missing.

Signed-off-by: cwillum <cwmmoore@amazon.com>
Comment on lines 27 to 28
| `query` | String | TBD. |
| `category` | String | TBD. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we provide description for TBDs here?

| :--- | :--- |:--- |
| `finding` | String | The finding ID. |
| `detector_type` | String | The log type for the detector. |
| `nearby_findings` | Number | TBD. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
Copy link
Contributor Author

cwillum commented Jul 13, 2023

@sbcd90 @getsaurabh02 Yes, I could use some help with a few field/parameter descriptions. Those will be conspicuous with the "TBD" placeholder. The last thing I need is a screenshot that shows the log type selectors with VPC_Flow (or however that is labeled). Thanks for looking.

Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum cwillum added 3 - Tech Review PR: Tech review in progress 4 - Doc Review PR: Doc review in progress and removed 2 - In progress Issue/PR: The issue or PR is in progress. 3 - Tech Review PR: Tech review in progress labels Jul 14, 2023
Copy link
Collaborator

@Naarcha-AWS Naarcha-AWS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small style question. Otherwise, LGTM

_security-analytics/api-tools/rule-api.md Outdated Show resolved Hide resolved
sbcd90 and others added 4 commits July 14, 2023 12:48
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Signed-off-by: cwillum <cwmmoore@amazon.com>
This reverts commit c6db296.

merge conflict with same edits on another branch#
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
Copy link
Contributor Author

cwillum commented Jul 14, 2023

Merge conflicts from same edits on different branches.

@cwillum cwillum added 5 - Final Editorial Review PR: Editorial Review in progress and removed 4 - Doc Review PR: Doc review in progress labels Jul 14, 2023
This reverts commit 7a70dc3.

merge conflict with second branch
:wq
Signed-off-by: cwillum <cwmmoore@amazon.com>
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cwillum Please see my comments and changes and let me know if you have any questions. Thanks!


# Correlation engine APIs

Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain span of time, and perform other tasks.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain span of time, and perform other tasks.
Correlation engine APIs allow you to create new correlation rules, view findings and correlations within a certain time window, and perform other tasks.

_security-analytics/api-tools/correlation-eng.md Outdated Show resolved Hide resolved
_security-analytics/api-tools/correlation-eng.md Outdated Show resolved Hide resolved
_security-analytics/api-tools/correlation-eng.md Outdated Show resolved Hide resolved

## List all findings and their correlations within a time window

Provides a list of all findings and their correlations within a specified window of time.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Provides a list of all findings and their correlations within a specified window of time.
Provides a list of all findings and their correlations within a specified time window:

@@ -11,6 +11,17 @@ The following APIs can be used for a number of tasks related to mappings, from c

## Get Mappings View

Returns a view of the fields contained in an index used as a log source.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rephrase as a complete sentence naming the noun (what is used to do this?).

@@ -11,7 +11,7 @@ The following APIs can be used for a number of tasks related to rules, from sear

## Create Custom Rule

The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information on how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see the information provided in [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).

@@ -60,6 +60,15 @@ Security Analytics takes advantage of prepackaged Sigma rules for security event

Although the ECS rule field names are largely self-explanatory, you can find predefined mappings of the Sigma rule field names to ECS rule field names, for all supported log types, in the GitHub Security Analytics repository. Navigate to the [OSMappings](https://github.com/opensearch-project/security-analytics/tree/main/src/main/resources/OSMapping) folder, choose the folder named for the log type, and open the `fieldmappings.yml` file. For example, to see the Sigma rule fields that correspond to ECS rule fields for the Windows log type, open the [fieldmappings.yml file](https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows/fieldmappings.yml) in the **windows** folder.

#### Amazon Security Lake logs

[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) service converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) service converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).
[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).


[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) service converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

The log types from Security Lake that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, since logs such as CloudTrail can be captured conceivably in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The log types from Security Lake that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, since logs such as CloudTrail can be captured conceivably in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.

@@ -11,7 +11,7 @@ The following APIs can be used for a number of tasks related to rules, from sear

## Create Custom Rule

The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information on how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
The Create custom rule API uses Sigma security rule formatting to create a custom rule. For information about how to write a rule in Sigma format, see information provided at [Sigma's GitHub repository](https://github.com/SigmaHQ/sigma).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirm capitalization of the API name.

Signed-off-by: cwillum <cwmmoore@amazon.com>
Signed-off-by: cwillum <cwmmoore@amazon.com>
Signed-off-by: cwillum <cwmmoore@amazon.com>
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum cwillum removed the 5 - Final Editorial Review PR: Editorial Review in progress label Jul 17, 2023
@cwillum cwillum merged commit 49e8391 into main Jul 17, 2023
3 of 4 checks passed
@prudhvigodithi prudhvigodithi added release-notes PR: Include this PR in the automated release notes and removed release-notes PR: Include this PR in the automated release notes labels Oct 3, 2023
harshavamsi pushed a commit to harshavamsi/documentation-website that referenced this pull request Oct 31, 2023
…pensearch-project#4549)

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* add missing param descriptions (opensearch-project#4555)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* Revert "fix#4500 ocsf fields and api"

This reverts commit c6db296.

merge conflict with same edits on another branch#

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* Revert "fix#4500 ocsf fields and api"

This reverts commit 7a70dc3.

merge conflict with second branch
:wq

* Revert "fix#4500 ocsf fields and api"

This reverts commit ef13a74.

* fix#4500 edits post merge conflict

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

---------

Signed-off-by: cwillum <cwmmoore@amazon.com>
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
vagimeli pushed a commit that referenced this pull request Dec 21, 2023
…4549)

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* add missing param descriptions (#4555)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* Revert "fix#4500 ocsf fields and api"

This reverts commit c6db296.

merge conflict with same edits on another branch#

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* Revert "fix#4500 ocsf fields and api"

This reverts commit 7a70dc3.

merge conflict with second branch
:wq

* Revert "fix#4500 ocsf fields and api"

This reverts commit ef13a74.

* fix#4500 edits post merge conflict

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4500 ocsf fields and api

Signed-off-by: cwillum <cwmmoore@amazon.com>

---------

Signed-off-by: cwillum <cwmmoore@amazon.com>
Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>
Co-authored-by: Subhobrata Dey <sbcd90@gmail.com>
@hdhalter hdhalter deleted the fix#4500-sec-analytics-ocsf branch March 28, 2024 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-notes PR: Include this PR in the automated release notes security-analytics v2.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[DOC]: Update Mappings to support Open Cybersecurity Schema Framework (OCSF)
6 participants