Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

checkpoint: improve support for Checkpoint 81 #5184

Merged
merged 2 commits into from Feb 6, 2023
Merged

checkpoint: improve support for Checkpoint 81 #5184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3562c30
checkpoint: improve support for Checkpoint 81
efd6 Feb 6, 2023
04b8fe5
address pr comment
efd6 Feb 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/checkpoint/changelog.yml
View file
Open in desktop
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.13.0"
changes:
- description: Improve support for Checkpoint 81.
type: enhancement
link: https://github.com/elastic/integrations/pull/5184
- version: "1.12.0"
changes:
- description: Allow configuration of time zones.
Expand Down
2 changes: 0 additions & 2 deletions ...point/data_stream/firewall/_dev/test/pipeline/test-checkpoint-with-time.log-expected.json
View file
Open in desktop
Expand Up @@ -160,7 +160,6 @@
"rule_action": "Accept",
"scheme": ":\"IKE",
"session_uid": "{6389E8E3-0000-0000-AC10-0209F7730000}",
"src_user_dn": "",
"src_user_name": "srcuser",
"user": "srcuser",
"vpn_feature_name": "VPN"
Expand Down Expand Up @@ -242,7 +241,6 @@
"rule_action": "Accept",
"scheme": ":\"IKE",
"session_uid": "{6389E8E3-0000-0000-AC10-0209F7730000}",
"src_user_dn": "",
"src_user_name": "srcuser",
"user": "srcuser",
"vpn_feature_name": "VPN"
Expand Down
2 changes: 1 addition & 1 deletion ...ages/checkpoint/data_stream/firewall/_dev/test/pipeline/test-checkpoint.log-expected.json
View file
Open in desktop
Expand Up @@ -1301,7 +1301,7 @@
"original": "\u003c134\u003e1 2020-03-30T07:20:35 gw-da58d3 CheckPoint 8363 - [action:\"Accept\"; flags:\"444676\"; ifdir:\"outbound\"; ifname:\"eth0\"; logid:\"0\"; loguid:\"{0x5e819dc3,0x0,0x353707c7,0xee78a1dc}\"; origin:\"192.168.1.100\"; originsicname:\"cn=cp_mgmt,o=gw-da58d3..tmn8s8\"; sequencenum:\"1\"; version:\"5\"; __policy_id_tag:\"product=VPN-1 \u0026 FireWall-1[db_tag={880771B0-FD92-2C4F-82FC-B96FC3DE5A07};mgmt=gw-da58d3;date=1585502566;policy_name=Standard\\]\"; dst:\"192.168.1.153\"; inzone:\"Local\"; layer_name:\"Network\"; layer_uuid:\"63b7fe60-76d2-4287-bca5-21af87337b0a\"; match_id:\"1\"; parent_rule:\"0\"; rule_action:\"Accept\"; rule_uid:\"1fde807b-6300-4b1a-914f-f1c1f3e2e7d2\"; outzone:\"External\"; product:\"VPN-1 \u0026 FireWall-1\"; proto:\"17\"; s_port:\"43103\"; service:\"514\"; service_id:\"syslog\"; src:\"192.168.1.100\"]",
"outcome": "success",
"sequence": 1,
"timezone": "+0500",
"timezone": "+05:00",
"type": [
"allowed",
"connection"
Expand Down
1 change: 0 additions & 1 deletion packages/checkpoint/data_stream/firewall/_dev/test/pipeline/test-r80x.log-expected.json
View file
Open in desktop
Expand Up @@ -2,7 +2,6 @@
"expected": [
{
"@timestamp": "2022-07-06T15:53:08.000Z",
"checkpoint": {},
"ecs": {
"version": "8.6.0"
},
Expand Down
3 changes: 2 additions & 1 deletion packages/checkpoint/data_stream/firewall/_dev/test/pipeline/test-r81x.log
View file
Open in desktop
@@ -1 +1,2 @@
<134>1 2022-07-16T18:51:20Z fw1 CheckPoint 15190 - [action:"Accept"; contextnum:"1"; flags:"802832"; ifdir:"inbound"; ifname:"eth0.11"; logid:"6"; loguid:"{0x8f6ff124,0xbeef4db4,0xbad40b,0xa9525929}"; origin:"81.2.69.144"; originsicname:"CN=fwcp1,O=Client_Server_1.company.com.bg7ujf"; sequencenum:"8"; time:"1657997480"; version:"5"; __nsons:"0"; __p_dport:"0"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={2A2FD8C0-A383-3DE4-A515-13D2CB28A798};mgmt=Client_Server_1;date=1657919347;policy_name=Standard\]"; __pos:"7"; bytes:"11930"; client_inbound_bytes:"7475"; client_inbound_interface:"eth0.11"; client_inbound_packets:"15"; client_outbound_bytes:"6345"; client_outbound_packets:"22"; context_num:"1"; elapsed:"0"; hll_key:"12347634786232348735"; packets:"37"; product:"Log Update"; segment_time:"1657997472"; server_inbound_bytes:"6345"; server_inbound_packets:"11"; server_outbound_bytes:"7475"; server_outbound_interface:"eth0.10"; server_outbound_packets:"30"; start_time:"1657997472"]
<134>1 2022-07-16T18:51:20Z fw1 CheckPoint 15190 - [action:"Accept"; contextnum:"1"; flags:"802832"; ifdir:"inbound"; ifname:"eth0.11"; logid:"6"; loguid:"{0x8f6ff124,0xbeef4db4,0xbad40b,0xa9525929}"; origin:"81.2.69.144"; originsicname:"CN=fwcp1,O=Client_Server_1.company.com.bg7ujf"; sequencenum:"8"; time:"1657997480"; version:"5"; __nsons:"0"; __p_dport:"0"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={2A2FD8C0-A383-3DE4-A515-13D2CB28A798};mgmt=Client_Server_1;date=1657919347;policy_name=Standard\]"; __pos:"7"; bytes:"11930"; client_inbound_bytes:"7475"; client_inbound_interface:"eth0.11"; client_inbound_packets:"15"; client_outbound_bytes:"6345"; client_outbound_packets:"22"; context_num:"1"; elapsed:"0"; hll_key:"12347634786232348735"; packets:"37"; product:"Log Update"; segment_time:"1657997472"; server_inbound_bytes:"6345"; server_inbound_packets:"11"; server_outbound_bytes:"7475"; server_outbound_interface:"eth0.10"; server_outbound_packets:"30"; start_time:"1657997472"]
<85>1 2023-01-13T10:10:16--5:00 172.16.2.9 CP-GW - Log [Fields@1.3.6.1.4.1.2620 inzone="DMZ" outzone="External" service_id="https" src="81.2.69.192" dst="81.2.69.142" proto="6" xlatesrc="81.2.69.144" xlatedst="" NAT_rulenum="1195" NAT_addtnl_rulenum="0" user="" src_user_name="" src_machine_name="" src_user_dn="" snid="" dst_user_name="" dst_machine_name="" dst_user_dn="" UP_match_table="TABLE_START" ROW_START="0" match_id="762" layer_uuid="e2117254-df10-4a5d-8d42-cacc362e077b" layer_name="DETOMPERMFW_2021 Security" rule_uid="82434e2a-a72c-4a59-8422-a0269fc32e93" rule_name="" ROW_END="0" UP_match_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="443" sport_svc="56530" xlatedport_svc="" xlatesport_svc="28493" ProductFamily="Network" ]
81 changes: 81 additions & 0 deletions packages/checkpoint/data_stream/firewall/_dev/test/pipeline/test-r81x.log-expected.json
View file
Open in desktop
Expand Up @@ -49,6 +49,87 @@
"tags": [
"preserve_original_event"
]
},
{
"@timestamp": "2023-01-13T10:10:16.000-05:00",
"checkpoint": {
"fields": "1.3.6.1.4.1.2620",
"match_id": "762",
"nat_addtnl_rulenum": "0",
"nat_rulenum": "1195",
"row_start": "0",
"up_match_table": "TABLE_START"
},
"destination": {
"geo": {
"city_name": "London",
"continent_name": "Europe",
"country_iso_code": "GB",
"country_name": "United Kingdom",
"location": {
"lat": 51.5142,
"lon": -0.0931
},
"region_iso_code": "GB-ENG",
"region_name": "England"
},
"ip": "81.2.69.142"
},
"ecs": {
"version": "8.6.0"
},
"event": {
"category": [
"network"
],
"kind": "event",
"original": "\u003c85\u003e1 2023-01-13T10:10:16--5:00 172.16.2.9 CP-GW - Log [Fields@1.3.6.1.4.1.2620 inzone=\"DMZ\" outzone=\"External\" service_id=\"https\" src=\"81.2.69.192\" dst=\"81.2.69.142\" proto=\"6\" xlatesrc=\"81.2.69.144\" xlatedst=\"\" NAT_rulenum=\"1195\" NAT_addtnl_rulenum=\"0\" user=\"\" src_user_name=\"\" src_machine_name=\"\" src_user_dn=\"\" snid=\"\" dst_user_name=\"\" dst_machine_name=\"\" dst_user_dn=\"\" UP_match_table=\"TABLE_START\" ROW_START=\"0\" match_id=\"762\" layer_uuid=\"e2117254-df10-4a5d-8d42-cacc362e077b\" layer_name=\"DETOMPERMFW_2021 Security\" rule_uid=\"82434e2a-a72c-4a59-8422-a0269fc32e93\" rule_name=\"\" ROW_END=\"0\" UP_match_table=\"TABLE_END\" ProductName=\"VPN-1 \u0026 FireWall-1\" svc=\"443\" sport_svc=\"56530\" xlatedport_svc=\"\" xlatesport_svc=\"28493\" ProductFamily=\"Network\" ]",
"timezone": "-05:00"
},
"network": {
"application": "https",
"iana_number": "6",
"name": "DETOMPERMFW_2021",
"transport": "tcp"
},
"observer": {
"egress": {
"zone": "External"
},
"ingress": {
"zone": "DMZ"
},
"type": "firewall",
"vendor": "Checkpoint"
},
"related": {
"ip": [
"81.2.69.192",
"81.2.69.144",
"81.2.69.142"
]
},
"source": {
"geo": {
"city_name": "London",
"continent_name": "Europe",
"country_iso_code": "GB",
"country_name": "United Kingdom",
"location": {
"lat": 51.5142,
"lon": -0.0931
},
"region_iso_code": "GB-ENG",
"region_name": "England"
},
"ip": "81.2.69.192",
"nat": {
"ip": "81.2.69.144"
}
},
"tags": [
"preserve_original_event"
]
}
]
}
62 changes: 58 additions & 4 deletions packages/checkpoint/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Expand Up @@ -11,12 +11,13 @@ processors:
- grok:
field: event.original
patterns:
- '%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-)
- '%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP}|-)
+(?:%{IPORHOST:syslog5424_host}|-) +(-|%{SYSLOG5424PRINTASCII:syslog5424_app})
+(-|%{SYSLOG5424PRINTASCII:syslog5424_proc}) +(?::-|%{SYSLOG5424PRINTASCII:syslog5424_msgid})
+\[%{GREEDYDATA:syslog5424_sd}\]'
pattern_definitions:
TIMESTAMP_ISO8601: "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE:_temp_.tz}?"
TIMESTAMP: "%{TIMESTAMP_ISO8601:syslog5424_ts}(?:-?%{ISO8601_TIMEZONE:_temp_.tz})?"
TIMESTAMP_ISO8601: "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?"
- kv:
field: syslog5424_sd
field_split: "; "
Expand All @@ -35,6 +36,32 @@ processors:
- rounded_bytes
- db_tag
- update_service
- kv:
if: ctx.checkpoint == null
field: syslog5424_sd
field_split: " "
value_split: "[=@]"
trim_key: " "
trim_value: " "
prefix: checkpoint.
strip_brackets: true
ignore_failure: true
exclude_keys:
- flags
- layer_uuid
- originsicname
- __policy_id_tag
- version
- rounded_bytes
- db_tag
- update_service
- foreach:
field: checkpoint
ignore_missing: true
ignore_failure: true
processor:
lowercase:
field: "_ingest._key"
- remove:
field:
- syslog5424_sd
Expand Down Expand Up @@ -72,6 +99,16 @@ processors:
- set:
field: event.timezone
copy_from: _temp_.tz
- gsub:
field: event.timezone
pattern: '([+-][0-9]{2})([0-9]{2})'
replacement: '$1:$2'
ignore_missing: true
- gsub:
field: event.timezone
pattern: '([+-])([0-9]):?([0-9]{2})'
replacement: '$10$2:$3'
ignore_missing: true
- date:
field: "syslog5424_ts"
timezone: "{{{ event.timezone }}}"
Expand Down Expand Up @@ -112,7 +149,7 @@ processors:
- rename:
field: checkpoint.xlatesrc
target_field: source.nat.ip
if: "ctx.checkpoint?.xlatesrc != '0.0.0.0'"
if: "ctx.checkpoint?.xlatesrc != '0.0.0.0' && ctx.checkpoint?.xlatesrc != ''"
ignore_missing: true
- rename:
field: checkpoint.dst
Expand All @@ -121,7 +158,7 @@ processors:
- rename:
field: checkpoint.xlatedst
target_field: destination.nat.ip
if: "ctx.checkpoint?.xlatedst != '0.0.0.0'"
if: "ctx.checkpoint?.xlatedst != '0.0.0.0' && ctx.checkpoint?.xlatedst != ''"
ignore_missing: true
- rename:
field: checkpoint.uid
Expand Down Expand Up @@ -944,6 +981,23 @@ processors:
- _temp_
- _conf
ignore_missing: true
- script:
description: Drops null/empty values recursively.
lang: painless
source:
boolean dropEmptyFields(Object object) {
if (object == null || object == '') {
return true;
} else if (object instanceof Map) {
((Map) object).values().removeIf(value -> dropEmptyFields(value));
return (((Map) object).size() == 0);
} else if (object instanceof List) {
((List) object).removeIf(value -> dropEmptyFields(value));
return (((List) object).length == 0);
}
return false;
}
dropEmptyFields(ctx);
- remove:
field: event.original
if: "ctx?.tags == null || !(ctx.tags.contains('preserve_original_event'))"
Expand Down
10 changes: 10 additions & 0 deletions packages/checkpoint/data_stream/firewall/fields/fields.yml
View file
Open in desktop
Expand Up @@ -475,6 +475,10 @@
type: keyword
description: |
Connected user name on the destination IP.
- name: dst_user_dn
type: keyword
description: |
User distinguished name connected to the destination IP address.
- name: dstkeyid
type: keyword
description: |
Expand Down Expand Up @@ -626,6 +630,8 @@
type: keyword
description: |
MTA failure description.
- name: fields
type: keyword
- name: file_direction
type: keyword
description: |
Expand Down Expand Up @@ -1226,6 +1232,8 @@
type: keyword
description: |
The role of identity.
- name: row_start
type: keyword
- name: rpc_prog
type: integer
description: |
Expand Down Expand Up @@ -1508,6 +1516,8 @@
- name: update_status
type: keyword
description: Status of database update
- name: up_match_table
type: keyword
- name: url
type: keyword
description: |
Expand Down
12 changes: 6 additions & 6 deletions packages/checkpoint/data_stream/firewall/sample_event.json
View file
Open in desktop
@@ -1,8 +1,8 @@
{
"@timestamp": "2020-03-29T13:19:20.000Z",
"agent": {
"ephemeral_id": "74e7438d-866f-44b9-b40c-80a5987ae772",
"id": "12c4ec6d-c3eb-4f2b-a572-1f056e85e632",
"ephemeral_id": "28303adf-5c5f-43e2-ac02-b30fc103f3f9",
"id": "6132a211-d170-415c-ab43-998f223485b8",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.5.1"
Expand All @@ -19,7 +19,7 @@
"version": "8.6.0"
},
"elastic_agent": {
"id": "12c4ec6d-c3eb-4f2b-a572-1f056e85e632",
"id": "6132a211-d170-415c-ab43-998f223485b8",
"snapshot": false,
"version": "8.5.1"
},
Expand All @@ -28,10 +28,10 @@
"category": [
"network"
],
"created": "2023-02-01T04:30:52.977Z",
"created": "2023-02-06T20:45:58.591Z",
"dataset": "checkpoint.firewall",
"id": "{0x5e80a059,0x0,0x6401a8c0,0x3c7878a}",
"ingested": "2023-02-01T04:30:53Z",
"ingested": "2023-02-06T20:45:59Z",
"kind": "event",
"sequence": 1,
"timezone": "UTC"
Expand All @@ -41,7 +41,7 @@
},
"log": {
"source": {
"address": "192.168.80.4:43660"
"address": "172.18.0.4:52602"
}
},
"network": {
Expand Down
16 changes: 10 additions & 6 deletions packages/checkpoint/docs/README.md
View file
Open in desktop
Expand Up @@ -23,8 +23,8 @@ An example event for `firewall` looks as following:
{
"@timestamp": "2020-03-29T13:19:20.000Z",
"agent": {
"ephemeral_id": "74e7438d-866f-44b9-b40c-80a5987ae772",
"id": "12c4ec6d-c3eb-4f2b-a572-1f056e85e632",
"ephemeral_id": "28303adf-5c5f-43e2-ac02-b30fc103f3f9",
"id": "6132a211-d170-415c-ab43-998f223485b8",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.5.1"
Expand All @@ -41,7 +41,7 @@ An example event for `firewall` looks as following:
"version": "8.6.0"
},
"elastic_agent": {
"id": "12c4ec6d-c3eb-4f2b-a572-1f056e85e632",
"id": "6132a211-d170-415c-ab43-998f223485b8",
"snapshot": false,
"version": "8.5.1"
},
Expand All @@ -50,10 +50,10 @@ An example event for `firewall` looks as following:
"category": [
"network"
],
"created": "2023-02-01T04:30:52.977Z",
"created": "2023-02-06T20:45:58.591Z",
"dataset": "checkpoint.firewall",
"id": "{0x5e80a059,0x0,0x6401a8c0,0x3c7878a}",
"ingested": "2023-02-01T04:30:53Z",
"ingested": "2023-02-06T20:45:59Z",
"kind": "event",
"sequence": 1,
"timezone": "UTC"
Expand All @@ -63,7 +63,7 @@ An example event for `firewall` looks as following:
},
"log": {
"source": {
"address": "192.168.80.4:43660"
"address": "172.18.0.4:52602"
}
},
"network": {
Expand Down Expand Up @@ -209,6 +209,7 @@ An example event for `firewall` looks as following:
| checkpoint.drops_amount | Amount of multicast packets dropped. | integer |
| checkpoint.dst_country | Destination country. | keyword |
| checkpoint.dst_phone_number | Destination IP-Phone. | keyword |
| checkpoint.dst_user_dn | User distinguished name connected to the destination IP address. | keyword |
| checkpoint.dst_user_name | Connected user name on the destination IP. | keyword |
| checkpoint.dstkeyid | Responder Spi ID. | keyword |
| checkpoint.duplicate | Log marked as duplicated, when mail is split and the Security Gateway sees it twice. | keyword |
Expand Down Expand Up @@ -248,6 +249,7 @@ An example event for `firewall` looks as following:
| checkpoint.extracted_file_verdict | Verdict of extracted files in case of an archive. | keyword |
| checkpoint.failure_impact | The impact of update service failure. | keyword |
| checkpoint.failure_reason | MTA failure description. | keyword |
| checkpoint.fields | | keyword |
| checkpoint.file_direction | File direction. Possible options: upload/download. | keyword |
| checkpoint.file_name | Malicious file name. | keyword |
| checkpoint.files_names | List of files requested by FTP. | keyword |
Expand Down Expand Up @@ -398,6 +400,7 @@ An example event for `firewall` looks as following:
| checkpoint.reply_status | ICAP reply status code, e.g. 200 or 204. | integer |
| checkpoint.risk | Risk level we got from the engine. | keyword |
| checkpoint.roles | The role of identity. | keyword |
| checkpoint.row_start | | keyword |
| checkpoint.rpc_prog | Log for new RPC state - prog values. | integer |
| checkpoint.rule | Matched rule number. | integer |
| checkpoint.rule_action | Action of the matched rule in the access policy. | keyword |
Expand Down Expand Up @@ -468,6 +471,7 @@ An example event for `firewall` looks as following:
| checkpoint.unique_detected_day | Detected virus for a specific host during the last day. | integer |
| checkpoint.unique_detected_hour | Detected virus for a specific host during the last hour. | integer |
| checkpoint.unique_detected_week | Detected virus for a specific host during the last week. | integer |
| checkpoint.up_match_table | | keyword |
| checkpoint.update_status | Status of database update | keyword |
| checkpoint.url | Translated URL. | keyword |
| checkpoint.user | Source user name. | keyword |
Expand Down