suricata logs can't parser? #594
Comments
|
Transferring to the hub as more appropriate location |
|
For the suricata fast log we only support text format not JSON, is this configurable? plus which logs is it? is it one below stated? |
LaurenceJJones
added
the
need-more-info
|
Stale issue closing please reopen if you find time to respond to questions |
|
Hi team, well, I exactly face the same problem. I run suricata version 5.0.10 (cannot upgrade to version 6, since I need to use it together with prelude OSS). The suricata logs do not get parsed. FYI: Acquisition Metrics: Local Api Metrics: Local Api Machines Metrics: Local Api Bouncers Metrics: Local Api Decisions: Local Api Alerts: This is one example line from eve.json {"timestamp":"2022-12-18T18:30:53.263350+0200","flow_id":989059031607521,"in_iface":"eth0","event_type":"tls","src_ip":"84.44.200.23","src_port":47470,"dest_ip":"95.216.205.51","dest_port":443,"proto":"TCP","tls":{"sni":"tube.xy-space.de","version":"TLS 1.3","ja3":{"hash":"579ccef312d18482fc42e2b822ca2430","string":"771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-34-51-43-13-45-28-21,29-23-24-25-256-257,0"},"ja3s":{"hash":"15af977ce25de452b96affa2addb1036","string":"771,4866,43-51"}}} Any ideas? |
|
Hey so the filter is json based so the event_type needs to be "alert" the one you shared is "tls" Here the filter |
|
Well, this was an example line. {"timestamp":"2022-12-18T22:13:41.010414+0200","flow_id":739629681939368,"in_iface":"eth0","event_type":"alert","src_ip":"135.181.5.2","src_port":38917,"dest_ip":"95.216.205.51","dest_port":8042,"proto":"TCP","metadata":{"flowints":{"applayer.anomaly.count":1,"http.anomaly.count":1}},"tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":2221042,"rev":1,"signature":"SURICATA HTTP Request line incomplete","category":"Generic Protocol Command Decode","severity":3},"http":{"http_port":0,"status":400,"length":0},"app_proto":"http","app_proto_ts":"failed","flow":{"pkts_toserver":4,"pkts_toclient":7,"bytes_toserver":247,"bytes_toclient":657,"start":"2022-12-18T22:13:41.005032+0200"}} which should be parsed, shouldn't they? |
|
Reopening issue will look into it Monday. Would be helpful if you could run the line through |
cscli explain --file /var/log/suricata/eve.json --type suricata-evelogs --verbose yields many lines, e. g. line: {"timestamp":"2022-12-18T23:31:09.003158+0200","flow_id":2159103788454115,"in_iface":"eth0","event_type":"flow","src_ip":"132.226.212.242","src_port":41784,"dest_ip":"95.216.205.51","dest_port":443,"proto":"TCP","app_proto":"tls","flow":{"pkts_toserver":14,"pkts_toclient":13,"bytes_toserver":3137,"bytes_toclient":7441,"start":"2022-12-18T23:30:00.921827+0200","end":"2022-12-18T23:30:06.410477+0200","age":6,"state":"closed","reason":"timeout","alerted":false},"tcp":{"tcp_flags":"1b","tcp_flags_ts":"1b","tcp_flags_tc":"1b","syn":true,"fin":true,"psh":true,"ack":true,"state":"closed"}} line: {"timestamp":"2022-12-18T23:36:58.012295+0200","flow_id":2041696589825413,"in_iface":"eth0","event_type":"tls","src_ip":"95.216.205.51","src_port":59106,"dest_ip":"172.67.202.221","dest_port":443,"proto":"TCP","tls":{"sni":"relay.national-defence.network","version":"TLS 1.3","ja3":{"hash":"c199b43d41b470f8f68c5561f8f1ce3e","string":"771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-35-22-23-13-43-45-51-21,29-23-30-25-24,0-1-2"},"ja3s":{"hash":"907bf3ecef1c987c889946b737b43de8","string":"771,4866,51-43"}}} |
|
Is that a like for like copy of the output? Cause its missing s00 section Helps if I have the full output also wrap the lines in ``` allows for better formatting But it's clear something not parsing so I can debug on Monday |
|
O. K. so there seems to be a different problem!? |
|
This is my acquis.yaml file content: #Generated acquisition file - wizard.sh (service: sshd) / files :
Note that I removed every third minus sign when posting the content in order to avoid formatting issues. |
|
Yes your missing the syslog parser? If you do |
|
Well, I did so, now I receive |
|
My updated acquis.yaml file looks like this: #Added by Martin Stenzel :
|
|
The crowdsec.log tells me: time="19-12-2022 00:20:15" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/apache2-logs.yaml stage=s01-parse No errors or warnings, but still "s01-parse" only yields red dots... |
|
That's because the log is stating the yaml syntax is correct doesn't mean the parser is correct for your log sample as stated will investigate Monday as its not just a syslog parser missing issue |
|
So with the alert above with |
|
So when you pass it through a json prettifier you can see it more clearly. So the parser only parses the line when the |
|
Well, first of all thank you for digging into it! I verified with the console, and receive alerts - as expected. So after all I can state that the latest stable version of crowdsec (1.4.3) works with suricata version 5.0.10. Thank you again. |
|
Always best to get confirmation that it is working 👍🏻 |
What happened?
suricata logs can't parser?
What did you expect to happen?
parser suricata work
How can we reproduce it (as minimally and precisely as possible)?
suricata logs :
{"timestamp":"2022-11-14T10:41:13.512844+0800","flow_id":171518089232986,"in_iface":"eth0","event_type":"anomaly","src_ip":"61.141.64.67","src_port":55191,"dest_ip":"103.164.63.78","dest_port":40189,"proto":"TCP","anomaly":{"app_proto":"dcer
pc","type":"applayer","event":"APPLAYER_DETECT_PROTOCOL_ONLY_ONE_DIRECTION","layer":"proto_detect"}}
{"timestamp":"2022-11-14T10:41:15.259165+0800","flow_id":1902553118510911,"in_iface":"eth0","event_type":"tls","src_ip":"103.164.63.78","src_port":37314,"dest_ip":"212.64.63.190","dest_port":443,"proto":"TCP","tls":{"subject":"CN=.gitee.com
","issuerdn":"C=CN, O=TrustAsia Technologies, Inc., OU=Domain Validated SSL, CN=TrustAsia TLS RSA CA","serial":"0E:9C:10:54:AA:30:0B:61:4B:19:82:19:4B:12:E0:B9","fingerprint":"73:77:c8:87:1a:3d:3b:5f:68:18:d8:3b:11:5a:e6:92:32:3a:5c:54","sni
":"gitee.com","version":"TLS 1.2","notbefore":"2022-02-21T00:00:00","notafter":"2023-03-06T23:59:59","ja3":{},"ja3s":{}}}
{"timestamp":"2022-11-14T10:41:15.263897+0800","flow_id":2097420079606221,"in_iface":"eth0","event_type":"anomaly","src_ip":"61.141.64.67","src_port":55223,"dest_ip":"103.164.63.78","dest_port":40189,"proto":"TCP","anomaly":{"app_proto":"dce
rpc","type":"applayer","event":"APPLAYER_DETECT_PROTOCOL_ONLY_ONE_DIRECTION","layer":"proto_detect"}}
{"timestamp":"2022-11-14T10:41:15.279867+0800","flow_id":664576040082216,"in_iface":"eth0","event_type":"tls","src_ip":"103.164.63.78","src_port":37316,"dest_ip":"212.64.63.190","dest_port":443,"proto":"TCP","tls":{"subject":"CN=.gitee.com"
,"issuerdn":"C=CN, O=TrustAsia Technologies, Inc., OU=Domain Validated SSL, CN=TrustAsia TLS RSA CA","serial":"0E:9C:10:54:AA:30:0B:61:4B:19:82:19:4B:12:E0:B9","fingerprint":"73:77:c8:87:1a:3d:3b:5f:68:18:d8:3b:11:5a:e6:92:32:3a:5c:54","sni"
:"gitee.com","version":"TLS 1.2","notbefore":"2022-02-21T00:00:00","notafter":"2023-03-06T23:59:59","ja3":{},"ja3s":{}}}
{"timestamp":"2022-11-14T10:41:15.280621+0800","flow_id":1021023965817414,"in_iface":"eth0","event_type":"anomaly","src_ip":"61.141.64.67","src_port":55224,"dest_ip":"103.164.63.78","dest_port":40189,"proto":"TCP","anomaly":{"app_proto":"dce
rpc","type":"applayer","event":"APPLAYER_DETECT_PROTOCOL_ONLY_ONE_DIRECTION","layer":"proto_detect"}}
{"timestamp":"2022-11-14T10:41:15.307745+0800","event_type":"stats","stats":{"uptime":16,"capture":{"kernel_packets":1964,"kernel_drops":0,"errors":0},"decoder":{"pkts":2298,"bytes":1348785,"invalid":144,"ipv4":2286,"ipv6":3,"ethernet":2298,
"chdlc":0,"raw":0,"null":0,"sll":0,"tcp":2112,"udp":33,"sctp":0,"icmpv4":0,"icmpv6":0,"ppp":0,"pppoe":0,"geneve":0,"gre":0,"vlan":0,"vlan_qinq":0,"vxlan":0,"vntag":0,"ieee8021ah":0,"teredo":0,"ipv4_in_ipv6":0,"ipv6_in_ipv6":0,"mpls":0,"avg_p
kt_size":586,"max_pkt_size":1534,"max_mac_addrs_src":0,"max_mac_addrs_dst":0,"erspan":0,"event":{"ipv4":{"pkt_too_small":0,"hlen_too_small":0,"iplen_smaller_than_hlen":0,"trunc_pkt":144,"opt_invalid":0,"opt_invalid_len":0,"opt_malformed":0,"
opt_pad_required":0,"opt_eol_required":0,"opt_duplicate":0,"opt_unknown":0,"wrong_ip_version":0,"icmpv6":0,"frag_pkt_too_large":0,"frag_overlap":0,"frag_ignored":0},"icmpv4":{"pkt_too_small":0,"unknown_type":0,"unknown_code":0,"ipv4_trunc_pk
t":0,"ipv4_unknown_ver":0},"icmpv6":{"unknown_type":0,"unknown_code":0,"pkt_too_small":0,"ipv6_unknown_version":0,"ipv6_trunc_pkt":0,"mld_message_with_invalid_hl":0,"unassigned_type":0,"experimentation_type":0},"ipv6":{"pkt_too_small":0,"tru
nc_pkt":0,"trunc_exthdr":0,"exthdr_dupl_fh":0,"exthdr_useless_fh":0,"exthdr_dupl_rh":0,"exthdr_dupl_hh":0,"exthdr_dupl_dh":0,"exthdr_dupl_ah":0,"exthdr_dupl_eh":0,"exthdr_invalid_optlen":0,"wrong_ip_version":0,"exthdr_ah_res_not_null":0,"hop
opts_unknown_opt":0,"hopopts_only_padding":0,"dstopts_unknown_opt":0,"dstopts_only_padding":0,"rh_type_0":0,"zero_len_padn":0,"fh_non_zero_reserved_field":0,"data_after_none_header":0,"unknown_next_header":0,"icmpv4":0,"frag_pkt_too_large":0
,"frag_overlap":0,"frag_invalid_length":0,"frag_ignored":0,"ipv4_in_ipv6_too_small":0,"ipv4_in_ipv6_wrong_version":0,"ipv6_in_ipv6_too_small":0,"ipv6_in_ipv6_wrong_version":0},"tcp":{"pkt_too_small":0,"hlen_too_small":0,"invalid_optlen":0,"o
pt_invalid_len":0,"opt_duplicate":0},"udp":{"pkt_too_small":0,"hlen_too_small":0,"hlen_invalid":0},"sll":{"pkt_too_small":0},"ethernet":{"pkt_too_small":0},"ppp":{"pkt_too_small":0,"vju_pkt_too_small":0,"ip4_pkt_too_small":0,"ip6_pkt_too_sma
ll":0,"wrong_type":0,"unsup_proto":0},"pppoe":{"pkt_too_small":0,"wrong_code":0,"malformed_tags":0},"gre":{"pkt_too_small":0,"wrong_version":0,"version0_recur":0,"version0_flags":0,"version0_hdr_too_big":0,"version0_malformed_sre_hdr":0,"ver
sion1_chksum":0,"version1_route":0,"version1_ssr":0,"version1_recur":0,"version1_flags":0,"version1_no_key":0,"version1_wrong_protocol":0,"version1_malformed_sre_hdr":0,"version1_hdr_too_big":0},"vlan":{"header_too_small":0,"unknown_type":0,
"too_many_layers":0},"ieee8021ah":{"header_too_small":0},"vntag":{"header_too_small":0,"unknown_type":0},"ipraw":{"invalid_ip_version":0},"ltnull":{"pkt_too_small":0,"unsupported_type":0},"sctp":{"pkt_too_small":0},"mpls":{"header_too_small"
:0,"pkt_too_small":0,"bad_label_router_alert":0,"bad_label_implicit_null":0,"bad_label_reserved":0,"unknown_payload_type":0},"vxlan":{"unknown_payload_type":0},"geneve":{"unknown_payload_type":0},"erspan":{"header_too_small":0,"unsupported_v
ersion":0,"too_many_vlan_layers":0},"dce":{"pkt_too_small":0},"chdlc":{"pkt_too_small":0}},"too_many_layers":0},"flow":{"memcap":0,"tcp":38,"udp":6,"icmpv4":0,"icmpv6":0,"tcp_reuse":0,"get_used":0,"get_used_eval":0,"get_used_eval_reject":0,"
get_used_eval_busy":0,"get_used_failed":0,"wrk":{"spare_sync_avg":100,"spare_sync":4,"spare_sync_incomplete":0,"spare_sync_empty":0,"flows_evicted_needs_work":0,"flows_evicted_pkt_inject":0,"flows_evicted":0,"flows_injected":0},"mgr":{"full_
hash_pass":1,"closed_pruned":0,"new_pruned":0,"est_pruned":0,"bypassed_pruned":0,"rows_maxlen":0,"flows_checked":0,"flows_notimeout":0,"flows_timeout":0,"flows_timeout_inuse":0,"flows_evicted":0,"flows_evicted_needs_work":0},"spare":9600,"em
erg_mode_entered":0,"emerg_mode_over":0,"memuse":7394304},"defrag":{"ipv4":{"fragments":0,"reassembled":0,"timeouts":0},"ipv6":{"fragments":0,"reassembled":0,"timeouts":0},"max_frag_hits":0},"flow_bypassed":{"local_pkts":0,"local_bytes":0,"l
ocal_capture_pkts":0,"local_capture_bytes":0,"closed":0,"pkts":0,"bytes":0},"tcp":{"sessions":29,"ssn_memcap_drop":0,"pseudo":0,"pseudo_failed":0,"invalid_checksum":0,"no_flow":0,"syn":29,"synack":22,"rst":15,"midstream_pickups":0,"pkt_on_wr
ong_thread":0,"segment_memcap_drop":0,"stream_depth_reached":0,"reassembly_gap":8,"overlap":2,"overlap_diff_data":0,"insert_data_normal_fail":0,"insert_data_overlap_fail":0,"insert_list_fail":0,"memuse":2424992,"reassembly_memuse":424024},"d
etect":{"engines":[{"id":0,"last_reload":"2022-11-14T10:40:59.216508+0800","rules_loaded":0,"rules_failed":0}],"alert":0,"alert_queue_overflow":0,"alerts_suppressed":0},"app_layer":{"flow":{"http":0,"ftp":0,"smtp":0,"tls":9,"ssh":1,"imap":0,
"smb":0,"dcerpc_tcp":10,"dns_tcp":0,"nfs_tcp":0,"ntp":1,"ftp-data":0,"tftp":0,"ikev2":0,"krb5_tcp":0,"dhcp":0,"snmp":0,"sip":0,"rfb":0,"mqtt":0,"rdp":0,"failed_tcp":0,"dcerpc_udp":0,"dns_udp":1,"nfs_udp":0,"krb5_udp":0,"failed_udp":4},"tx":{
"http":0,"ftp":0,"smtp":0,"tls":0,"ssh":0,"imap":0,"smb":0,"dcerpc_tcp":0,"dns_tcp":0,"nfs_tcp":0,"ntp":1,"ftp-data":0,"tftp":0,"ikev2":0,"krb5_tcp":0,"dhcp":0,"snmp":0,"sip":0,"rfb":0,"mqtt":0,"rdp":0,"dcerpc_udp":0,"dns_udp":2,"nfs_udp":0,
"krb5_udp":0},"expectations":0},"http":{"memuse":0,"memcap":0},"ftp":{"memuse":0,"memcap":0},"file_store":{"open_files":0}}}
Anything else we need to know?
No response
Crowdsec version
OS version
Enabled collections and parsers
Acquisition config
On Windows:
C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
paste output here
Config show
Prometheus metrics
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
The text was updated successfully, but these errors were encountered: