[Juniper SRX] Issues with System message groks

[P1llus]

Currently the parsing of system messages for Juniper SRX is quite new, however there is a few main issues that has to be resolved.

1. It is possible for the initial grok created for traffic structured in default.yml to hit the system-structured data as well, this causes issues. The grok should be rewritten into 3 components instead of the current 2:

• One grok to handle structured traffic data, should have a custom pattern that lists all the possible juniper process names (the field used to decide on which pipeline to send the data).
• One grok to handle structured system data, should re-use the custom pattern as a negative lookahead, to support mapping the process name but ignore any of the traffic ones.
• One grok to handle unstructured system data, this exists today, but some has too many optional values, structured and unstructured should be split into multiple.

2. The grok processor in system.yml has issues with certain data formats, here as well we should have more explicit grok patterns for structured and unstructured data, as unstructured data is still hitting the pattern for structured.

• Create one pattern for structured (they are always inside square brackets).
• One pattern for unstructured.
• One catchall pattern that we have today.

3. Currently the _temp_.to_be_parsed field is not actually parsed anywhere, the TAG field at the start of the grok also support only values without spaces, so things like "IKE negotiation failed with error" does not work.

Dissect for this is not working because of missing tag, its in the test data:
<27>1 2023-05-04T15:19:33.984+10:00 AB1234-A-AB-AB01C-ABC kmd 9159 asd2 asd IKE negotiation failed with error: Timed out. IKE Version: 1, VPN: IPSEC-AAAAA-AAA1-PROD-VPN Gateway: IKE-AAAAA-AAA1-GW, Local: 89.160.20.112/500, Remote: 67.43.156.1/500, Local IKE-ID: Not-Available, Remote IKE-ID: Not-Available, VR-ID: 5: Role: Initiator

Other sample data that hits traffic-structured on default.yml rather than the system pattern:
<37>1 2023-05-10T00:10:24.232+10:00 AB1234-A-AB-AB01C-ABC snmpd 8959 SNMPD_AUTH_FAILURE [junos@1111.1.1.1.1.111 function-name="nsa_log_community" message="unauthorized SNMP community" source-address="216.160.83.56" destination-address="89.160.20.128" index1="j5Cx6eSkKF7A"]

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

Additional tests that should pass from #6837.

<27>1 2023-07-04T12:22:36.461+10:00 AC004-PR-VPN01-DMZ kmd 9812 - - IKE negotiation failed with error: Timed out. IKE Version: 1, VPN: IPSEC-HORSEFACTS-TUN1-PROD-VPN Gateway: IKE-HORSEFACTS-TUN1-GW, Local: 10.11.22.444/500, Remote: 198.1.124.8/500, Local IKE-ID: Not-Available, Remote IKE-ID: Not-Available, VR-ID: 5: Role: Initiator
<30>1 2023-07-04T10:21:11.590+10:00 AC004-PR-VPN01-DMZ kmd 9812 - - IKE negotiation successfully completed. IKE Version: 1, VPN: IPSEC-NIKON-TUN1-PROD-VPN Gateway: IKE-NIKON-TUN1-GW, Local: 10.8.10.115/9001, Remote: 89.160.20.112/9001, Local IKE-ID: 81.2.69.192, Remote IKE-ID: 89.160.20.112, VR-ID: 6, Role: Responder
<27>1 2023-07-04T11:48:31.702+10:00 AC004-PR-VPN01-DMZ kmd 9812 - - IPSec negotiation failed with error: Peer proposed traffic-selectors are not in configured range. IKE Version: 2, VPN: IPSEC-INT-ORMB-TUN2-VPN Gateway: IKE-INT-ORMB-TUN2-GW, Local: 10.32.64.128/9001, Remote: 89.160.20.112/9001, Local IKE-ID: 89.160.20.112, Remote IKE-ID: 89.160.20.112, VR-ID: 6

[kcreddy]

In the PR fix: #7280, there are several instances where logs are not correctly formatted, or deviating from the standard.

Examples from test-system.log https://github.com/elastic/integrations/pull/7280/files#diff-1eb55a2372d47f06bd91f28e99332584946da09269defe2e7103b33a1e42758a:

• line 2: , Traffic-selector: FC Name: has no comma between fields as separator.
• line 3: , VR-ID: 5: Role: Initiator has : instead of , for field separator.
• line 6: , VPN: IPSEC-NIKON-TUN1-PROD-VPN Gateway: IKE-NIKON-TUN1-GW, has instead of , for field separator. Also present in multiple places.

These are leading to incorrect parsing.

[kcreddy]

Hey @cai-elastic,

Based on above comment, could you please confirm if these logs are correctly formatted? Because of this we are seeing incorrect output in such cases. https://github.com/elastic/integrations/pull/7280/files#diff-0db8ac6cad8ca10c7bcf6c12ed7588cc181c0c7876d9701a31195e56a66b3b99

[cai-elastic]

Hi Kcreddy,

I think they are exceptional cases that can be ignored in the test coverage. It's fine as long as we support correct formats.

Best Regards,
Cai

[kcreddy]

I think they are exceptional cases that can be ignored in the test coverage. It's fine as long as we support correct formats.

Thanks Cai, since it can be ignored, I will make the PR ready for review.