DeepSecurity support #27

EijiSugiura · 2020-12-15T10:41:20Z

Issue #, if available:

Description of changes:

Supports DeepSecurity CEF format logs.

CEF logs in JSON samples,

{
    "message":"Oct 28 14:46:28 ip-10-0-XX-XX CEF: 0|Trend Micro|Deep Security Agent|20.0.0.1194|1000552|Generic Cross Site Scripting(XSS) Prevention|10|cn1=2312751 cn1Label=Host ID dvc=10.0.XX.XX TrendMicroDsTenant=tenant name TrendMicroDsTenantId=XXXXXX dmac=0E:37:8C:B4:47:10 smac=0E:28:7B:CF:58:50 TrendMicroDsFrameType=IP src=10.0.XX.XX dst=10.0.YY.YY in=5770 cs3=DF 0 cs3Label=Fragmentation Bits proto=TCP spt=54056 dpt=8080 cs2=0x00 ACK PSH cs2Label=TCP Flags cnt=1 act=IDS:Log cn3=1472 cn3Label=DPI Packet Position cs5=60218 cs5Label=DPI Stream Position cs1=XSS Attack cs1Label=DPI Note cs6=8 cs6Label=DPI Flags TrendMicroDsPacketData=IDwh...YXku\\=\\=",
    "hostname":"ip-10-0-XX-XX",
    "timestamp":"2020-10-28T14:46:28+09:00"
}

{
    "message":"Nov 28 10:33:58 ip-10-0-XX-XX CEF: 0|Trend Micro|Deep Security Agent|20.0.0.1194|3002831|Unix - Syslog|6|cn1=2313513 cn1Label=Host ID dvc=10.0.XX.XX TrendMicroDsTenant=tenant name TrendMicroDsTenantId=XXXXXX cs1=Non standard syslog message (size too large) cs1Label=LI Description fname=/var/log/messages shost=ip-10-0-XX-XX msg=Nov 28 10:33:57 ip-10-0-XX-XX error: message:Could not push log to Elasticsearch: {\"took\"\\=>329, \"errors\"\\=>true, ...",
    "hostname":"ip-10-0-XX-XX",
    "timestamp":"2020-11-28T10:33:58+09:00"
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

EijiSugiura · 2020-12-15T10:43:16Z

source/lambda/es_loader/siem/sf_deepsecurity.py

+        'rule.category': 'cs1',
+        'host.id': 'cn1',
+        'event.original': 'msg',
+    }


These should be done with ecs settings.

EijiSugiura · 2020-12-15T10:44:16Z

source/lambda/es_loader/siem/sf_deepsecurity.py

+
+    return logdata
+
+


Belows are copied from __init__.py.

nakajiak

プルリクありがとうございます！いくつか修正して欲しいので対応をお願いします。

nakajiak · 2020-12-16T00:22:58Z

source/lambda/es_loader/README_deepsecurity_ja.md

@@ -0,0 +1,165 @@
+# es_loaderでDeepSecurityのログを取り込む


プルリクありがとうございます。
このファイルはこちらに保存をお願いします。
/docs/contributed/deepsecurity_ja.md
es-loaderの下に置くと、lambdaのパッケージサイズが増えるのと管理が煩雑になるので、docsは一カ所にまとめようと思います。

了解です。移動します！

nakajiak · 2020-12-16T00:41:14Z

source/lambda/es_loader/README_deepsecurity_ja.md

+    "index_patterns" : [
+      "log-deepsecurity*"
+    ],
+    "mappings" : {


log-* ですでに定義済みのフィールドは、再度の定義は不要なので削除をお願いします。
destination,ip, destination.port, source.ip, souce.port, timestamp 等
文字列のtypeはデフォルトでkeywordとしてるので、数字になる可能性等がなければこちらも設定不要です。
https://github.com/aws-samples/siem-on-amazon-elasticsearch/blob/dc942d03c64c1130b9add025f32de0595fad0abc/source/lambda/deploy_es/data.ini#L42-L46

重複していたものと、keywordを指定したものを削除しました。

nakajiak · 2020-12-16T00:44:25Z

source/lambda/es_loader/README_deepsecurity_ja.md

+index = log-deepsecurity
+s3_key = ds_agent
+format = json
+scripted_fields = True


scrpted_fieldsはobsoletedする予定なので、こちらのフィールドに置き換えをお願いします。
https://github.com/aws-samples/siem-on-amazon-elasticsearch/blob/dc942d03c64c1130b9add025f32de0595fad0abc/source/lambda/es_loader/aws.ini#L184-L191

更新しておきました。

nakajiak · 2020-12-16T00:46:42Z

source/lambda/es_loader/aws.ini

@@ -634,3 +634,30 @@ event.kind = event
 script_ecs = cloud.instance.id event.action event.category event.outcome source.ip source.port user.name user.id
 geoip = source

+


source/lambda/es_loader/user.ini.sample への追加でお願いします。

移動しました！

nakajiak · 2020-12-16T00:47:11Z

source/lambda/es_loader/aws.ini

@@ -634,3 +634,30 @@ event.kind = event
 script_ecs = cloud.instance.id event.action event.category event.outcome source.ip source.port user.name user.id
 geoip = source

+
+[deepsecurity]
+# https://cloudone.trendmicro.com/docs/workload-security/event-syslog-message-formats/


こちらのコメントの追加をお願いします

# See README for more details # https://github.com/aws-samples/siem-on-amazon-elasticsearch/blob/main/docs/contributed/deepsecurity_ja.md

追加しておきました！

nakajiak · 2020-12-16T00:49:31Z

source/lambda/es_loader/siem/sf_deepsecurity.py

+import base64
+import json
+import ipaddress
+


この方法で関数のimport をして、106行以降の削除お願いします

from siem import merge, put_value_into_dict, get_value_from_dict

importして、削除しておきました！

nakajiak · 2020-12-16T00:55:18Z

source/lambda/es_loader/README_deepsecurity_ja.md

+[deepsecurity]
+index = log-deepsecurity
+s3_key = ds_agent
+format = json


CEFフォーマットのサポートのissueを作りました。+1をお願いします
#28

nakajiak · 2020-12-16T09:33:12Z

マージしました。ありがとうございます！

Dashboard for S3 accesslog #27

DeepSecurity support

269f3b4

EijiSugiura commented Dec 15, 2020

View reviewed changes

source/lambda/es_loader/siem/sf_deepsecurity.py Outdated

return logdata

Copy link

Author

EijiSugiura Dec 15, 2020 •

edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Belows are copied from __init__.py.

nakajiak requested changes Dec 16, 2020

View reviewed changes

EijiSugiura added 5 commits December 16, 2020 13:47

Move README for DeepSecurity

b5d5f8b

Update documents around script_ecs

74813a8

Move ini sample to use.init.sample

4677248

fix attribute name

62ce0e4

import from siem

30c3faa

nakajiak merged commit 2918074 into aws-samples:develop Dec 16, 2020

nakajiak added this to the v2.2.0 milestone Dec 16, 2020

nakajiak modified the milestones: v2.2.0, v2.1.1 Dec 25, 2020

nakajiak added the enhancement New feature or request label Dec 25, 2020

fshuhe added a commit to fshuhe/siem-on-amazon-opensearch-service that referenced this pull request Feb 7, 2022

Dashboard for S3 access log aws-samples#27

45bcc12

fshuhe added a commit to fshuhe/siem-on-amazon-opensearch-service that referenced this pull request Feb 7, 2022

Dashboard for S3 access log aws-samples#27

2be2e24

nakajiak added a commit that referenced this pull request Feb 7, 2022

Merge pull request #199 from fshuhe/develop

7ad67a6

Dashboard for S3 accesslog #27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DeepSecurity support #27

DeepSecurity support #27

EijiSugiura commented Dec 15, 2020

EijiSugiura Dec 15, 2020

EijiSugiura Dec 15, 2020 •

edited

nakajiak left a comment

nakajiak Dec 16, 2020

EijiSugiura Dec 16, 2020

nakajiak Dec 16, 2020

EijiSugiura Dec 16, 2020

nakajiak Dec 16, 2020

EijiSugiura Dec 16, 2020

nakajiak Dec 16, 2020

EijiSugiura Dec 16, 2020

nakajiak Dec 16, 2020

EijiSugiura Dec 16, 2020

nakajiak Dec 16, 2020

EijiSugiura Dec 16, 2020

nakajiak Dec 16, 2020

nakajiak commented Dec 16, 2020

		@@ -0,0 +1,165 @@
		# es_loaderでDeepSecurityのログを取り込む

		@@ -634,3 +634,30 @@ event.kind = event
		script_ecs = cloud.instance.id event.action event.category event.outcome source.ip source.port user.name user.id
		geoip = source

DeepSecurity support #27

DeepSecurity support #27

Conversation

EijiSugiura commented Dec 15, 2020

Choose a reason for hiding this comment

EijiSugiura Dec 15, 2020 • edited

Choose a reason for hiding this comment

nakajiak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nakajiak commented Dec 16, 2020

EijiSugiura Dec 15, 2020 •

edited