Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Meraki Syslog Parser not parsing some events #9471

Closed
joseff8 opened this issue Nov 24, 2023 · 31 comments
Closed

Meraki Syslog Parser not parsing some events #9471

joseff8 opened this issue Nov 24, 2023 · 31 comments
Assignees
Labels
enhancement New feature or request Parser Parser specialty review needed

Comments

@joseff8
Copy link

joseff8 commented Nov 24, 2023

Is your feature request related to a problem? Please describe.
Meraki Syslog Parser not parsing some events

Solutions/CiscoMeraki/Parsers/CiscoMeraki.txt

Describe the solution you'd like
Would like the parser to handle additonal "LogType" events like:
"firewall"
"vpn_firewall"
"ip_flow_start"
"ip_flow_end"

Additional context
Not very good in KQL, however i was able to add the basics:
srcipaddr example:
LogType has "ip_flow_start", extract(@"src=([0-9.]+)\s",1,Substring),
LogType has "ip_flow_end", extract(@"src=([0-9.]+)\s",1,Substring),
LogType has "vpn_firewall", extract(@"src=([0-9.]+)\s",1,Substring),
LogType has "firewall", extract(@"src=([0-9.]+)\s",1,Substring),

Looks like the extract is the same for "flows" in each case

Added the LogType Events here as will
LogType !in ("urls", "airmarshal_events","security_event","ids-alerts", "events", "vpn_firewall", "firewall", "ip_flow_start", "ip_flow_end") and LogType !contains "flows", iif(isempty(LogType), "", LogType),

Copy link
Contributor

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

@v-muuppugund
Copy link
Contributor

Hi @joseff8 , Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 29Nov23. Thanks!

@v-muuppugund v-muuppugund self-assigned this Nov 24, 2023
@v-muuppugund v-muuppugund added the Parser Parser specialty review needed label Nov 24, 2023
@v-sudkharat v-sudkharat added the enhancement New feature or request label Dec 7, 2023
@v-sudkharat v-sudkharat self-assigned this Dec 7, 2023
@v-sudkharat
Copy link
Contributor

Hi @joseff8, Sorry for the delay in response. Could you please share more detailed information about the issue? It would be helpful if you could add screenshots of the error you are encountering.

Thanks!

@v-sudkharat
Copy link
Contributor

Hi @joseff8, we are waiting for your response on above comment. thanks!

@joseff8
Copy link
Author

joseff8 commented Dec 11, 2023

Hi @v-sudkharat, Could you please let me know what more information is needed? As stated the Meraki parser does not parse the events mentioned.
"firewall"
"vpn_firewall"
"ip_flow_start"
"ip_flow_end"

I added example changes to how the parser could include the events.

@v-muuppugund
Copy link
Contributor

Hi @joseff8 ,Could you please share sample data in csv format for this parser issue and also screen shot of the error details.

@joseff8
Copy link
Author

joseff8 commented Dec 15, 2023

Hi @v-muuppugund,
Here is a sample.

Dec 15 07:38:10 10.47.129.253 171122330.11122336118 MX_Firewall vpn_firewall src=x.x.x.x dst=x.x.x.x protocol=udp sport=60940 dport=161 pattern: allow all

Dec 15 07:38:10 10.47.255.253 1711223390.202233603 MX_Firewall firewall src=x.x.x.x dst=x.x.x.x mac=00:xx:xx:xx:xx:xx protocol=udp sport=60944 dport=161 pattern: allow (dst x.x.x.x/24 || dst x.x.x.x/24 || (dst x.x.x.x/23 || dst x.x.x.x/24 || dst x.x.x.x/23) || dst x.x.x.x/26) && (src x.x.x.x/24)

Dec 15 07:38:10 10.47.255.253 1112233890.211223390 MX_Firewall ip_flow_start src=x.x.x.x dst=x.x.x.x protocol=udp sport=60944 dport=161 translated_dst_ip=x.x.x.x translated_port=161

As you can see the "type" of log is vpn_firewall, firewall and ip_flow_start
I was able to edit the parser to some degree to get these event "Types" parsed as well:
See attached file.
CiscoMeraki.txt

@v-muuppugund
Copy link
Contributor

v-muuppugund commented Dec 15, 2023

Hi @joseff8 ,Thanks ,will check details and do detailed analysis and will get back to you with an update

@v-muuppugund
Copy link
Contributor

v-muuppugund commented Dec 18, 2023

Hi @joseff8 ,After Analyzing the above sample data ,we are not parsing the above log types in parser,so its unable to detect,I am unable to replicate the same issue at my environment,Please add LogType has "firewall", extract(@"pattern: ([\S\s]+)", 1, Substring),
LogType has "ip_flow_start", extract(@"pattern: ([\S\s]+)", 1, Substring),
LogType has "ip_flow_end", extract(@"pattern: ([\S\s]+)", 1, Substring) and this will work ,need check whole logs can you share whole dump in csv format to v-muuppugund@microsoft.com,so will modify the query and share with you.

@v-muuppugund
Copy link
Contributor

v-muuppugund commented Dec 21, 2023

Hi @joseff8 ,I have updated the parser attached
Updated.Cisco.Meraki.txt

based on data,Could you please check and let me know if any issues

@antigrammatik
Copy link

@v-muuppugund That was really good and got most of the parsing working for the flow start and flow end events, thank you for putting this out here. There were a couple columns that weren't working correctly. The source and destination ports needed to be converted to int and the regex for the new fields needed to be updated to get the right value. Here is my updated version. It still needs a bit of work to make it perfect but it's probably 90% of the way there.

Updated Cisco Meraki 12-21-23.txt

@v-sudkharat
Copy link
Contributor

@antigrammatik, thank you for your response.

@v-sudkharat
Copy link
Contributor

Hi @joseff8, Could you please have a look on this comment- #9471 (comment) and let us know if your issue gets resolved. Thanks!

@v-muuppugund
Copy link
Contributor

Hi @joseff8 / @antigrammatik , I have cross checked the parser ,above attached is the updated one,Gentle Reminder,Please check and let me know if any issues

@v-muuppugund
Copy link
Contributor

v-muuppugund commented Jan 2, 2024

Hi @joseff8 / @antigrammatik ,Genle Reminder,I have cross checked the parser ,above attached is the updated one, Please check and let me know if any issues, Proceeding with the changes, will raise pr

@joseff8
Copy link
Author

joseff8 commented Jan 3, 2024

@v-muuppugund, Thanks for the update. Sorry for the late reply, i have been on vacation. The code works better. We are still missing the changes for "pattern".

    Pattern = case(
                        isnotempty(column_ifexists("Pattern", "")), column_ifexists("Pattern", ""),
                        LogType has "flows", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "vpn_firewall", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "firewall", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "ip_flow_start", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "ip_flow_end", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        ""
                   ), 

@v-muuppugund
Copy link
Contributor

v-muuppugund commented Jan 3, 2024

@joseff8 ,Thanks, Will check on these changes ,will work on it and push it.

@v-muuppugund
Copy link
Contributor

@joseff8 ,I have done the changes,testing with large dump of data for any issues as part of standard practice once done,will raise the PR and push it.

@v-muuppugund
Copy link
Contributor

v-muuppugund commented Jan 16, 2024

@joseff8 ,Still data ingesting is happening, Will share updates once Testing is completed and pr is pushed.

@azvella
Copy link

azvella commented Mar 7, 2024

This really should make it into the production solution parser file. This was a great find and super helpful for me.

@abrbon
Copy link

abrbon commented Mar 11, 2024

Another issue with this parser is that it does not work for IPv6 addresses.

I fixed this by changing the following lines of code:

LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)

into

LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)

Made the same changes for all "dst=" IP addresses.

And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes.
I fixed that also by simply changing all "flows" to "flow".

And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:

NetworkProtocol = toupper(NetworkProtocol)

and added the following to parse ICMP type protocol:

ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

@v-muuppugund
Copy link
Contributor

Another issue with this parser is that it does not work for IPv6 addresses.

I fixed this by changing the following lines of code:

LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)

into

LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)

Made the same changes for all "dst=" IP addresses.

And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow".

And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:

NetworkProtocol = toupper(NetworkProtocol)

and added the following to parse ICMP type protocol:

ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

@v-muuppugund
Copy link
Contributor

v-muuppugund commented Mar 19, 2024

Another issue with this parser is that it does not work for IPv6 addresses.
I fixed this by changing the following lines of code:
LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)
into
LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)
Made the same changes for all "dst=" IP addresses.
And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow".
And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:
NetworkProtocol = toupper(NetworkProtocol)
and added the following to parse ICMP type protocol:
ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

Will be doing further analysis of Ip4 address ,Will update you post my analysis and changes.

@v-muuppugund
Copy link
Contributor

Another issue with this parser is that it does not work for IPv6 addresses.
I fixed this by changing the following lines of code:
LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)
into
LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)
Made the same changes for all "dst=" IP addresses.
And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow".
And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:
NetworkProtocol = toupper(NetworkProtocol)
and added the following to parse ICMP type protocol:
ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

Will be doing further analysis of Ip4 address ,Will update you post my analysis and changes.

Hi @abrbon / @joseff8 ,I have done the requested changes ,working on testing it,will update you

@v-muuppugund
Copy link
Contributor

HI @joseff8 / @abrbon ,just want to update ,earlier data has been lost due to data retention period,so working on testing the new changes done once data ingestion completed

@v-rusraut
Copy link
Contributor

Hi @joseff8,
We are still working on data ingestion, we will update you once testing complete.
Thanks

@v-rusraut
Copy link
Contributor

Hi @joseff8,
We tried to ingest data, but we are not able to ingest data in few tables.
Please run below parser in your workspace where data available.
CiscoMeraki.txt and share working screen shots with us.
Thanks

@v-rusraut
Copy link
Contributor

Hi @joseff8,
Please provide update on above comment.
Thanks

@v-rusraut
Copy link
Contributor

Hi @joseff8 , Gentle Reminder: We are waiting for your response on this issue. If you still need to keep this issue active, please respond to it in the next 2 days. If we don't receive a response by 13-05-2024 date, we will be closing this issue.
Thanks!

@abrbon
Copy link

abrbon commented May 10, 2024

Hi @joseff8 , Gentle Reminder: We are waiting for your response on this issue. If you still need to keep this issue active, please respond to it in the next 2 days. If we don't receive a response by 13-05-2024 date, we will be closing this issue. Thanks!

Hi,
I checked and ran the parser on data from our client but unfortunately have not any data in our tables for “Pattern”.

However I still see the same issues I had with this parser like not recognizing IPv6 addresses and not parsing the “ip_flow_start” and “ip_flow_end” logtypes like I mentioned in my earlier posts.

@v-rusraut
Copy link
Contributor

Hi @joseff8,
We have tried to resolve this issue in existing parser, but as it is enhancement/feature request in existing parser, so we have noted this feature request and will share this with our respective concern team. Currently we can't provide any ETA for this. but once we get any further info from our team will share update to you. So, closing this issue from GitHub. If you still need support for this issue, feel free to re-open it any time. Thank you for your co-operation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Parser Parser specialty review needed
Projects
None yet
Development

No branches or pull requests

7 participants