Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add documentation that supports custom log types #4969

Merged
merged 30 commits into from
Sep 19, 2023
Merged

Add documentation that supports custom log types #4969

merged 30 commits into from
Sep 19, 2023

Conversation

cwillum
Copy link
Contributor

@cwillum cwillum commented Sep 6, 2023

Description

Adds documentation for custom log type enhancements.

Issues Resolved

Fixes #4741

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@cwillum
fix#4741 custom logtype updates
4b87dfc 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum cwillum added 2 - In progress Issue/PR: The issue or PR is in progress. release-notes PR: Include this PR in the automated release notes security-analytics v2.10.0 labels Sep 6, 2023
@cwillum cwillum self-assigned this Sep 6, 2023
@cwillum
fix#4741 custom logtype updates
f49a578 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
77931c2 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
bf8020e 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
6b9912b 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
4f43a9b 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
9d4f9db 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
10a5a58 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
b17657b 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
cf15866 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
5f09eeb 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
1669fee 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
3383f9f 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
60fc156 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
461a7e8 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
3107ae6 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
0a70c29 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
Copy link
Contributor Author

cwillum commented Sep 11, 2023

@sbcd90 @amsiglan Could you look at this documentation for custom log types? I also have a few questions about the "Search custom log types" API:

  1. For the Search custom log types API, what kind of information goes in the "match_all" object? How are the values formatted: separated by commas? I’ve tried names and ids - neither work.
  2. What are the other request fields that can be used with this call? (In addition to "match_all")
  3. This appears to provide results for all log types, not only custom. Any problems just calling it "Search log types" API? Or are we intending users to only process custom log types with it?

Thank you.

@cwillum
fix#4741 custom logtype updates
02ef654 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
db25a9e 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
829ab68 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
f5210c3 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
d443444 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
0134802 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
9eae221 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum
fix#4741 custom logtype updates
8969c9c 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum cwillum added 4 - Doc Review PR: Doc review in progress and removed 2 - In progress Issue/PR: The issue or PR is in progress. labels Sep 18, 2023
@cwillum
Copy link
Contributor Author

cwillum commented Sep 18, 2023 •
edited
Loading

Will create a separate PR to address "Search custom log type API" questions. Pushing this to Documentation team review. (this was tech reviewed via direct communications)

Naarcha-AWS
Naarcha-AWS approved these changes Sep 19, 2023
View reviewed changes
Copy link
Collaborator

@Naarcha-AWS Naarcha-AWS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of small questions, otherwise LGTM.

_security-analytics/sec-analytics-config/custom-log-type.md Show resolved Hide resolved
_security-analytics/sec-analytics-config/custom-log-type.md Outdated
After selecting **Create log type** in the **Log types** page, the **Create log type** page opens and provides the necessary fields to create a new log type:
1. Enter a name for the log type.

The log type name supports characters a-z (lower case), 0--9, hyphens, and underscores.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this note format correctly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does.

_security-analytics/sec-analytics-config/detectors-config.md Outdated
@@ -69,7 +69,7 @@ Although the ECS rule field names are largely self-explanatory, you can find pre

[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is DNS lowercased and bolded in this context? Is it supposed to match a UI element?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. But this representation is opposed by the PM. UX and front-end dev will eventually replace these log type names with revised names. For the time being, this is the way the log type is represented in the UI.

cwillum
cwillum commented Sep 19, 2023
View reviewed changes
Copy link
Contributor Author

@cwillum cwillum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Addressed comments.

_security-analytics/sec-analytics-config/detectors-config.md Outdated
@@ -69,7 +69,7 @@ Although the ECS rule field names are largely self-explanatory, you can find pre

[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. But this representation is opposed by the PM. UX and front-end dev will eventually replace these log type names with revised names. For the time being, this is the way the log type is represented in the UI.

_security-analytics/sec-analytics-config/custom-log-type.md Outdated
After selecting **Create log type** in the **Log types** page, the **Create log type** page opens and provides the necessary fields to create a new log type:
1. Enter a name for the log type.

The log type name supports characters a-z (lower case), 0--9, hyphens, and underscores.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does.

@cwillum
fix#4741 custom logtype updates
0833900 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum cwillum added 5 - Final Editorial Review PR: Editorial Review in progress and removed 4 - Doc Review PR: Doc review in progress labels Sep 19, 2023
natebower
natebower reviewed Sep 19, 2023
View reviewed changes
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cwillum Please see my comments and changes and let me know if you have any questions. Thanks!

_security-analytics/api-tools/log-type-api.md Outdated Show resolved Hide resolved
_security-analytics/api-tools/log-type-api.md Outdated Show resolved Hide resolved
_security-analytics/api-tools/log-type-api.md Outdated Show resolved Hide resolved
_security-analytics/index.md Outdated Show resolved Hide resolved
_security-analytics/sec-analytics-config/custom-log-type.md Outdated Show resolved Hide resolved
_security-analytics/sec-analytics-config/detectors-config.md Outdated Show resolved Hide resolved
_security-analytics/sec-analytics-config/index.md Outdated Show resolved Hide resolved
_security-analytics/sec-analytics-config/index.md Outdated Show resolved Hide resolved
_security-analytics/sec-analytics-config/index.md Outdated Show resolved Hide resolved
_security-analytics/usage/overview.md Outdated Show resolved Hide resolved
@cwillum
fix#4741 custom logtype updates
9f54189 
Signed-off-by: cwillum <cwmmoore@amazon.com>
cwillum
cwillum commented Sep 19, 2023
View reviewed changes
Copy link
Contributor Author

@cwillum cwillum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All set. Thanks for the thorough review.

_security-analytics/sec-analytics-config/detectors-config.md Outdated Show resolved Hide resolved
_security-analytics/sec-analytics-config/index.md Outdated Show resolved Hide resolved
@cwillum
fix#4741 custom logtype updates
0154048 
Signed-off-by: cwillum <cwmmoore@amazon.com>
@cwillum cwillum merged commit dde7c02 into main Sep 19, 2023
4 checks passed
@cwillum cwillum added 3 - Done Issue is done/complete and removed 5 - Final Editorial Review PR: Editorial Review in progress labels Sep 19, 2023
vagimeli pushed a commit that referenced this pull request Sep 19, 2023
@cwillum @vagimeli
Add documentation that supports custom log types (#4969)
4ad6fef 
* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

---------

Signed-off-by: cwillum <cwmmoore@amazon.com>
vagimeli added a commit that referenced this pull request Sep 19, 2023
@vagimeli
Revert "Add documentation that supports custom log types (#4969)"
fab0952 
This reverts commit 4ad6fef.
harshavamsi pushed a commit to harshavamsi/documentation-website that referenced this pull request Oct 31, 2023
@cwillum @harshavamsi
Add documentation that supports custom log types (opensearch-project#…
1949c98 
…4969)

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

---------

Signed-off-by: cwillum <cwmmoore@amazon.com>
vagimeli pushed a commit that referenced this pull request Dec 21, 2023
@cwillum @vagimeli
Add documentation that supports custom log types (#4969)
67b3c2c 
* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <cwmmoore@amazon.com>

---------

Signed-off-by: cwillum <cwmmoore@amazon.com>
@hdhalter hdhalter deleted the fix#4741-custom-log-type branch March 28, 2024 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@natebower natebower natebower left review comments

@Naarcha-AWS Naarcha-AWS Naarcha-AWS approved these changes

@hdhalter hdhalter Awaiting requested review from hdhalter hdhalter is a code owner

@kolchfa-aws kolchfa-aws Awaiting requested review from kolchfa-aws kolchfa-aws is a code owner

@vagimeli vagimeli Awaiting requested review from vagimeli vagimeli is a code owner

@ananzh ananzh Awaiting requested review from ananzh

@seanneumann seanneumann Awaiting requested review from seanneumann

@AMoo-Miki AMoo-Miki Awaiting requested review from AMoo-Miki AMoo-Miki is a code owner

Assignees

@cwillum cwillum

Labels
3 - Done Issue is done/complete release-notes PR: Include this PR in the automated release notes security-analytics v2.10.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[DOC] Custom Log Type support for Security Analytics
3 participants
@cwillum @Naarcha-AWS @natebower