Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Next/20190124/v4 #3625

Merged
merged 7 commits into from
Jan 31, 2019
Merged

Next/20190124/v4 #3625

merged 7 commits into from
Jan 31, 2019

Conversation

victorjulien
Copy link
Member

Most intrusive here is the eve.stats decoder events handling to fix ticket 2225. In existing setups it should keep behaving unchanged, although it will show warnings.

For new setups using the shipped yaml, the decoder events prefix in the logs will be 'decoder.event' (e.g. decoder.event.ipv6.unknown_next_header).

Warnings are added to indicate that in 5.0 this will become the overall default.

cc: @satta

Link to redmine ticket:
https://redmine.openinfosecfoundation.org/issues/2225
https://redmine.openinfosecfoundation.org/issues/2770

Describe changes:

PRScript output (if applicable):

victorjulien and others added 7 commits January 23, 2019 22:22
@victorjulien
decoder: add gre over ipv6 support
c140505
@victorjulien
hash: move string hash funcs into util files
0f1fc1f
@victorjulien
eve: fix missing decoder-events in stats
932c2a7 
In the eve log the decoder events are added as optional counters. This
behaviour is enabled by default. However, lots of the counters are
missing, as the names colide with other counters.

E.g.

decoder.ipv6 counts ipv6 packets
decoder.ipv6.unknown_next_header counts how often an unknown next
    header is encountered.

In this example 'ipv6' would be both a json integer and a json object.
It appears that jansson favours the first that is generated, so the
event counters are mostly missing.

This patch registers them as 'decoder.events.<event>' instead. As
these names are generated on the fly, a hash table to contain the
allocated strings was added as well.
@victorjulien
eve.stats: make decoder event prefix configurable
0d86263
@victorjulien
eve.stats: warn that output might miss decoder-events
fb18a16
@victorjulien
stream: fix false negative on bad RST
d8634da 
If a bad RST was received the stream inspection would not happen
for that packet, but it would still move the 'raw progress' tracker
forward. Following good packets would then fail to detect anything
before the 'raw progress' position.

Bug OISF#2770

Reported-by: Alexey Vishnyakov
@jingleyang @victorjulien
source-pcap:set PktAcqBreakLoop as pcap_breakloop
bb26e62
@victorjulien victorjulien requested a review from a team as a code owner January 25, 2019 14:17
@jasonish
Copy link
Member

On a first quick look, I don't like seeing the configuration option being in the config and not just in the documentation. But I suppose that is to keep 4.1.x releases behaving the same? And we could remove it in 5.0 and just have it in the documentation?

@pevma
Copy link
Member

pevma commented Jan 29, 2019

This is how a full run on a busy box looks like in terms of stats. Seems the formatting is off in some places due to the longer names.


------------------------------------------------------------------------------------
Date: 1/29/2019 -- 08:40:59 (uptime: 0d, 15h 19m 52s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 79119493860
capture.kernel_drops                       | Total                     | 3211114510
decoder.pkts                               | Total                     | 75907750281
decoder.bytes                              | Total                     | 78172804309131
decoder.invalid                            | Total                     | 36417
decoder.ipv4                               | Total                     | 69251153090
decoder.ipv6                               | Total                     | 8947627648
decoder.ethernet                           | Total                     | 75907750819
decoder.tcp                                | Total                     | 67763375493
decoder.udp                                | Total                     | 7623328894
decoder.icmpv4                             | Total                     | 179169959
decoder.icmpv6                             | Total                     | 6883013
decoder.ppp                                | Total                     | 7402598
decoder.gre                                | Total                     | 2322737532
decoder.vlan                               | Total                     | 75907728499
decoder.teredo                             | Total                     | 332253
decoder.ipv4_in_ipv6                       | Total                     | 76489004
decoder.avg_pkt_size                       | Total                     | 1029
decoder.max_pkt_size                       | Total                     | 1514
flow.tcp                                   | Total                     | 1782157180
flow.udp                                   | Total                     | 195786196
flow.icmpv4                                | Total                     | 10048524
flow.icmpv6                                | Total                     | 642933
defrag.ipv4.fragments                      | Total                     | 35136634
defrag.ipv4.reassembled                    | Total                     | 17390072
defrag.ipv6.fragments                      | Total                     | 771399
defrag.ipv6.reassembled                    | Total                     | 368553
decoder.event.ipv4.opt_pad_required        | Total                     | 1839
decoder.event.icmpv4.pkt_too_small         | Total                     | 5
decoder.event.icmpv4.unknown_type          | Total                     | 3795
decoder.event.icmpv4.unknown_code          | Total                     | 278
decoder.event.icmpv4.ipv4_trunc_pkt        | Total                     | 1
decoder.event.icmpv4.ipv4_unknown_ver      | Total                     | 1515
decoder.event.icmpv6.unknown_code          | Total                     | 1
decoder.event.icmpv6.unassigned_type       | Total                     | 4
decoder.event.ipv6.exthdr_useless_fh       | Total                     | 1752
decoder.event.ipv6.zero_len_padn           | Total                     | 1000
decoder.event.ipv6.fh_non_zero_reserved_field | Total                     | 5
decoder.event.ipv6.data_after_none_header  | Total                     | 225312
decoder.event.ipv6.unknown_next_header     | Total                     | 10791
decoder.event.ipv6.icmpv4                  | Total                     | 16
decoder.event.tcp.hlen_too_small           | Total                     | 25450
decoder.event.tcp.invalid_optlen           | Total                     | 2637
decoder.event.tcp.opt_invalid_len          | Total                     | 11211
decoder.event.tcp.opt_duplicate            | Total                     | 113
decoder.event.udp.pkt_too_small            | Total                     | 255
decoder.event.udp.hlen_invalid             | Total                     | 88
decoder.event.ppp.wrong_type               | Total                     | 7035179
decoder.event.ppp.unsup_proto              | Total                     | 29236
decoder.event.gre.version0_recur           | Total                     | 12153
decoder.event.gre.version0_flags           | Total                     | 17251300
decoder.event.ipv4.frag_overlap            | Total                     | 1500
decoder.event.ipv6.frag_overlap            | Total                     | 1
decoder.event.ipv6.ipv4_in_ipv6_wrong_version | Total                     | 4
decoder.event.ipv6.ipv6_in_ipv6_wrong_version | Total                     | 2
stream.3whs_right_seq_wrong_ack_evasion    | Total                     | 36166
stream.3whs_synack_in_wrong_direction      | Total                     | 2384
stream.3whs_synack_resend_with_diff_ack    | Total                     | 1035
stream.3whs_synack_toserver_on_syn_recv    | Total                     | 3356
stream.3whs_synack_with_wrong_ack          | Total                     | 130658
stream.3whs_synack_flood                   | Total                     | 2524
stream.3whs_syn_resend_diff_seq_on_syn_recv | Total                     | 129862
stream.3whs_syn_toclient_on_syn_recv       | Total                     | 281
stream.3whs_wrong_seq_wrong_ack            | Total                     | 17196055
stream.4whs_synack_with_wrong_ack          | Total                     | 1
stream.4whs_synack_with_wrong_syn          | Total                     | 18
stream.4whs_wrong_seq                      | Total                     | 2040
stream.4whs_invalid_ack                    | Total                     | 136
stream.closewait_ack_out_of_window         | Total                     | 3077199
stream.closewait_fin_out_of_window         | Total                     | 68808
stream.closewait_pkt_before_last_ack       | Total                     | 1647391
stream.closewait_invalid_ack               | Total                     | 1309036
stream.est_packet_out_of_window            | Total                     | 882206646
stream.est_pkt_before_last_ack             | Total                     | 136932248
stream.est_synack_resend                   | Total                     | 31871
stream.est_synack_resend_with_diff_ack     | Total                     | 10329
stream.est_synack_resend_with_diff_seq     | Total                     | 10055
stream.est_synack_toserver                 | Total                     | 1493
stream.est_syn_resend                      | Total                     | 8864
stream.est_syn_resend_diff_seq             | Total                     | 131976
stream.est_syn_toclient                    | Total                     | 11
stream.est_invalid_ack                     | Total                     | 467681659
stream.fin_invalid_ack                     | Total                     | 127970
stream.fin1_ack_wrong_seq                  | Total                     | 8942
stream.fin1_fin_wrong_seq                  | Total                     | 8362
stream.fin1_invalid_ack                    | Total                     | 5265
stream.fin2_ack_wrong_seq                  | Total                     | 485461
stream.fin2_fin_wrong_seq                  | Total                     | 1623
stream.fin2_invalid_ack                    | Total                     | 286331
stream.fin_but_no_session                  | Total                     | 76381514
stream.fin_out_of_window                   | Total                     | 111499
stream.lastack_ack_wrong_seq               | Total                     | 886
stream.lastack_invalid_ack                 | Total                     | 52377
stream.rst_but_no_session                  | Total                     | 29082495
stream.timewait_ack_wrong_seq              | Total                     | 25238
stream.timewait_invalid_ack                | Total                     | 814
stream.shutdown_syn_resend                 | Total                     | 59708
stream.pkt_invalid_timestamp               | Total                     | 3421946
stream.pkt_invalid_ack                     | Total                     | 469620368
stream.pkt_broken_ack                      | Total                     | 9498484
stream.rst_invalid_ack                     | Total                     | 156688
stream.pkt_retransmission                  | Total                     | 3428533
stream.pkt_bad_window_update               | Total                     | 9126811
stream.suspected_rst_inject                | Total                     | 18383
stream.wrong_thread                        | Total                     | 865
stream.reassembly_segment_before_base_seq  | Total                     | 22
stream.reassembly_seq_gap                  | Total                     | 2927797
tcp.sessions                               | Total                     | 1555726345
tcp.pseudo                                 | Total                     | 1228288
tcp.syn                                    | Total                     | 1583408745
tcp.synack                                 | Total                     | 199060539
tcp.rst                                    | Total                     | 185144369
tcp.pkt_on_wrong_thread                    | Total                     | 22570
tcp.stream_depth_reached                   | Total                     | 2167458
tcp.reassembly_gap                         | Total                     | 2927797
tcp.overlap                                | Total                     | 4263044726
tcp.insert_list_fail                       | Total                     | 22
detect.alert                               | Total                     | 84337387
app_layer.flow.http                        | Total                     | 15691004
app_layer.tx.http                          | Total                     | 34999943
app_layer.flow.ftp                         | Total                     | 124497
app_layer.flow.smtp                        | Total                     | 2056613
app_layer.tx.smtp                          | Total                     | 2098124
app_layer.flow.tls                         | Total                     | 74524249
app_layer.flow.ssh                         | Total                     | 4013008
app_layer.flow.imap                        | Total                     | 2789
app_layer.flow.smb                         | Total                     | 5074
app_layer.tx.smb                           | Total                     | 276048
app_layer.flow.dcerpc_tcp                  | Total                     | 63909
app_layer.flow.dns_tcp                     | Total                     | 548679
app_layer.tx.dns_tcp                       | Total                     | 1106397
app_layer.flow.nfs_tcp                     | Total                     | 46
app_layer.tx.nfs_tcp                       | Total                     | 766
app_layer.flow.ntp                         | Total                     | 3885831
app_layer.flow.ftp-data                    | Total                     | 100710
app_layer.flow.tftp                        | Total                     | 756878
app_layer.flow.ikev2                       | Total                     | 27310
app_layer.flow.krb5_tcp                    | Total                     | 9447
app_layer.tx.krb5_tcp                      | Total                     | 9438
app_layer.flow.dhcp                        | Total                     | 4758
app_layer.flow.failed_tcp                  | Total                     | 9282345
app_layer.flow.dcerpc_udp                  | Total                     | 5149
app_layer.flow.dns_udp                     | Total                     | 76726664
app_layer.tx.dns_udp                       | Total                     | 157830585
app_layer.tx.ntp                           | Total                     | 5695357
app_layer.tx.tftp                          | Total                     | 789898
app_layer.tx.ikev2                         | Total                     | 141563
app_layer.flow.krb5_udp                    | Total                     | 52457
app_layer.tx.krb5_udp                      | Total                     | 2141
app_layer.tx.dhcp                          | Total                     | 70648
app_layer.flow.failed_udp                  | Total                     | 114327149
flow_mgr.closed_pruned                     | Total                     | 102732712
flow_mgr.new_pruned                        | Total                     | 1761810999
flow_mgr.est_pruned                        | Total                     | 119362206
flow_mgr.bypassed_pruned                   | Total                     | 3863694
flow.spare                                 | Total                     | 80084619
flow.tcp_reuse                             | Total                     | 217654
flow_mgr.flows_checked                     | Total                     | 103183
flow_mgr.flows_notimeout                   | Total                     | 67836
flow_mgr.flows_timeout                     | Total                     | 35347
flow_mgr.flows_timeout_inuse               | Total                     | 3411
flow_mgr.flows_removed                     | Total                     | 31936
flow_mgr.rows_checked                      | Total                     | 16388608
flow_mgr.rows_skipped                      | Total                     | 16258536
flow_mgr.rows_empty                        | Total                     | 31742
flow_mgr.rows_maxlen                       | Total                     | 14
flow_bypassed.closed                       | Total                     | 7699532
flow_bypassed.pkts                         | Total                     | 806673539
flow_bypassed.bytes                        | Total                     | 813537302112
tcp.memuse                                 | Total                     | 2240053144
tcp.reassembly_memuse                      | Total                     | 1490446704
http.memuse                                | Total                     | 239080478
ftp.memuse                                 | Total                     | 6919359
app_layer.expectations                     | Total                     | 1956
flow.memuse                                | Total                     | 7562635000

@inliniac inliniac merged commit bb26e62 into OISF:master Jan 31, 2019
@victorjulien
Copy link
Member Author

Merged this with a doc update and a fix for the stats.log layout. Thanks @jasonish @pevma

@victorjulien victorjulien mentioned this pull request Jan 31, 2019
Closed
3 tasks
@victorjulien victorjulien deleted the next/20190124/v4 branch February 12, 2019 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
5 participants
@victorjulien @jasonish @pevma @inliniac @jingleyang