Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Segfault in dbProcessEventInformation() #63

Closed
reedloden opened this Issue · 12 comments

3 participants

@reedloden

Getting a fairly consistent segfault with latest barnyard2 using the database output plugin connected to mysql.

Compile options:
./configure --prefix=/usr --sysconfdir=/etc/barnyard2 --with-mysql-libraries=/usr/lib/x86_64-linux-gnu --enable-mysql-ssl-support --enable-debug

Startup arguments:
-D -c /etc/barnyard2/barnyard.conf -d /var/log/barnyard2 -w /var/log/barnyard2/barnyard2.waldo -l /var/log/barnyard2 -f unified2.alert -i eth1

$ barnyard2 -V

  ______   -*> Barnyard2 <*-
 / ,,_  \  Version 2.1.11 (Build 318) DEBUG
 |o"  )~|  By Ian Firns (SecurixLive): http://www.securixlive.com/
 + '''' +  (C) Copyright 2008-2012 Ian Firns <firnsy@securixlive.com>

Output from gdb:

0x000000000043953a in dbProcessEventInformation (data=0x7ffff7f87010, p=0x3a79cc0, event=0x3a78760, event_type=7, i_sig_id=375) at spo_database.c:2001
2001    spo_database.c: No such file or directory.
(gdb) bt full
#0  0x000000000043953a in dbProcessEventInformation (data=0x7ffff7f87010, p=0x3a79cc0, event=0x3a78760, event_type=7, i_sig_id=375) at spo_database.c:2001
        SQLQueryPtr = 0x33e4400 ""
        i = 0
        __FUNCTION__ = "dbProcessEventInformation"
#1  0x000000000043a4bc in Database (p=0x3a79cc0, event=0x3a78760, event_type=7, arg=0x7ffff7f87010) at spo_database.c:2494
        data = 0x7ffff7f87010
        CurrentQuery = 0x0
        sig_id = 375
        itr = 0
        SQLMaxQuery = 0
        __FUNCTION__ = "Database"
        sid = 2200035
        gid = 1
        revision = 1
        event_id = 2473594
        event_second = 1357679520
        event_microsecond = 25801
#2  0x000000000041d9ac in CallOutputPlugins (out_type=OUTPUT_TYPE__SPECIAL, packet=0x3a79cc0, event=0x3a78760, event_type=7) at plugbase.c:573
        idx = 0xb4c540
#3  0x000000000041ebf3 in spoolerProcessRecord (spooler=0x2caa0f0, fire_output=1) at spooler.c:701
        event_id = 2473594
        pkth = {ts = {tv_sec = 1357679520, tv_usec = 25801}, caplen = 136, len = 136}
        type = 2
        ernCache = 0x3a78650
#4  0x000000000041e731 in ProcessContinuous (dirpath=0x6bbacc "/var/log/barnyard2", filebase=0x6bbecc "unified2.alert", record_start=0, timestamp=1357678214) at spooler.c:533
        spooler = 0x2caa0f0
        ret = 0
        pc_ret = 0
        new_file_available = 0
        waiting_logged = 0
        skipped = 0
        extension = 1357678213
        waldo_timestamp = 1357621019
#5  0x000000000041e956 in ProcessContinuousWithWaldo (waldo=0x6bb6c4) at spooler.c:621
No locals.
#6  0x0000000000403d62 in Barnyard2Main (argc=14, argv=0x7fffffffe178) at barnyard2.c:374
No locals.
#7  0x0000000000403b8a in main (argc=14, argv=0x7fffffffe178) at barnyard2.c:285
No locals.

Rule that is causing the problem:

alert pkthdr any any -> any any (msg:"SURICATA TCP invalid option length"; decode-event:tcp.invalid_optlen; sid:2200035; rev:1;)

Assistance getting this resolved would be appreciated.

@reedloden
(gdb) print *p
$12 = {pkth = 0x7fffffffdf50, pkt = 0x3a787bc "\324\256R\255z\223(\300\332z\024\023\b", ah = 0x0, eh = 0x3a787bc, vh = 0x0, ehllc = 0x0, ehllcother = 0x0, greh = 0x0, mpls = 0x0, iph = 0x3a787ca,
  orig_iph = 0x0, inner_iph = 0x3a787ca, outer_iph = 0x0, tcph = 0x0, orig_tcph = 0x0, udph = 0x0, orig_udph = 0x0, icmph = 0x0, orig_icmph = 0x0, data = 0x0,
  ip_data = 0x3a787de "\305\232\001\273]\234\210G\001\v\005\004\001\003", outer_ip_data = 0x0, ip_frag_start = 0x0, ip_options_data = 0x0, tcp_options_data = 0x0, ssnptr = 0x0, fragtracker = 0x0,
  flow = 0x0, streamptr = 0x0, ip4h = 0x0, orig_ip4h = 0x0, ip6h = 0x0, orig_ip6h = 0x0, icmp6h = 0x0, orig_icmp6h = 0x0, iph_api = 0x0, orig_iph_api = 0x0, outer_iph_api = 0x0,
  outer_orig_iph_api = 0x0, inner_ip4h = {ip_verhl = 0 '\000', ip_tos = 0 '\000', ip_len = 0, ip_id = 0, ip_off = 0, ip_ttl = 0 '\000', ip_proto = 0 '\000', ip_csum = 0, ip_src = {family = 0,
      bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}, ip_dst = {family = 0, bits = 0, ip = {
        u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}}, inner_orig_ip4h = {ip_verhl = 0 '\000', ip_tos = 0 '\000', ip_len = 0, ip_id = 0,
    ip_off = 0, ip_ttl = 0 '\000', ip_proto = 0 '\000', ip_csum = 0, ip_src = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0,
          0, 0, 0}}}, ip_dst = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}}, inner_ip6h = {vcl = 0, len = 0,
    next = 0 '\000', hop_lmt = 0 '\000', ip_src = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}, ip_dst = {
      family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}}, inner_orig_ip6h = {vcl = 0, len = 0, next = 0 '\000',
    hop_lmt = 0 '\000', ip_src = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}, ip_dst = {family = 0, bits = 0,
      ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}}, outer_ip4h = {ip_verhl = 0 '\000', ip_tos = 0 '\000', ip_len = 0, ip_id = 0,
    ip_off = 0, ip_ttl = 0 '\000', ip_proto = 0 '\000', ip_csum = 0, ip_src = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0,
          0, 0, 0}}}, ip_dst = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}}, outer_orig_ip4h = {
    ip_verhl = 0 '\000', ip_tos = 0 '\000', ip_len = 0, ip_id = 0, ip_off = 0, ip_ttl = 0 '\000', ip_proto = 0 '\000', ip_csum = 0, ip_src = {family = 0, bits = 0, ip = {
        u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}, ip_dst = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>,
        u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}}, outer_ip6h = {vcl = 0, len = 0, next = 0 '\000', hop_lmt = 0 '\000', ip_src = {family = 0, bits = 0, ip = {
        u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}, ip_dst = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>,
        u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}}, outer_orig_ip6h = {vcl = 0, len = 0, next = 0 '\000', hop_lmt = 0 '\000', ip_src = {family = 0, bits = 0, ip = {
        u6_addr8 = '\000' <repeats 15 times>, u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}, ip_dst = {family = 0, bits = 0, ip = {u6_addr8 = '\000' <repeats 15 times>,
        u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}}, mplsHdr = {label = 0, exp = 0 '\000', bos = 0 '\000', ttl = 0 '\000'}, family = 0, orig_family = 0, outer_family = 0,
  bytes_to_inspect = 0, preprocessor_bits = 0, preproc_reassembly_pkt_bits = 0, caplen = 0, http_pipeline_count = 0, packet_flags = 0, proto_bits = 0, dsize = 0, ip_dsize = 102, alt_dsize = 0,
  actual_ip_len = 122, outer_ip_dsize = 0, frag_offset = 0, ip_frag_len = 0, ip_options_len = 0, tcp_options_len = 0, sp = 0, dp = 0, orig_sp = 0, orig_dp = 0, application_protocol_ordinal = 0,
  frag_flag = 0 '\000', mf = 0 '\000', df = 1 '\001', rf = 0 '\000', uri_count = 0 '\000', csum_flags = 0 '\000', encapsulated = 0 '\000', ip_option_count = 0 '\000', tcp_option_count = 0 '\000',
  ip6_extension_count = 0 '\000', ip6_frag_index = 0 '\000', ip_lastopt_bad = 0 '\000', tcp_lastopt_bad = 0 '\000', fddihdr = 0x0, fddisaps = 0x0, fddisna = 0x0, fddiiparp = 0x0, fddiother = 0x0,
  trh = 0x0, trhllc = 0x0, trhmr = 0x0, pf1h = 0x0, pf2h = 0x0, pf3h = 0x0, sllh = 0x0, wifih = 0x0, pppoeh = 0x0, eplh = 0x0, eaph = 0x0, eaptype = 0x0, eapolk = 0x0, ip_options = {{code = 0 '\000',
      len = 0 '\000', data = 0x0} <repeats 40 times>}, tcp_options = {{code = 0 '\000', len = 0 '\000', data = 0x0} <repeats 40 times>}, ip6_extensions = {{type = 0 '\000',
      data = 0x0} <repeats 40 times>}, configPolicyId = 0, linktype = 1}
@binf
Collaborator

Are you able to process your unified2 file with u2spewfoo?
What is your segmentation fault message?

By the later i @ which line does it segfault?

Also whats the impact of the following suricata keyword in event logging data "pkthdr".

/* Addmentum */
I actually reformated the comment you left (localy) and i had some of the previously requested information (IE didin't seem to render it correctly width wise so i was missing some information.

seem's like the decoder is not able to decode the packet but the packet is tagged as tcp packet (from the decoded ip header) but if you look at your second message, there is no tcph pointer defined in the payload which lead to the segfault.

Its mainly due to the way the decoder work for now, it probably bails and does not return any error since their all void returning functions.

I would still be curious to see what u2spewfoo prints and see the offending packet in a pcap, you can probably also use u2boat to extract the packets them self from the unified2 file.

@reedloden

Sadly, I needed to move forward (as this was blocking progress), and before your comment came in yesterday, I disabled all the suricata decoder (http://rules.emergingthreats.net/open/suricata/rules/decoder-events.rules) and stream (http://rules.emergingthreats.net/open/suricata/rules/stream-events.rules) rules and removed all the faulty unified2 files. Barnyard2 is working fine now, but I still have it running under gdb for the time being in case it segfaults again.

I'll be spinning up some new sensors soon, and when I do, I'll re-enable those rules to see if I can get it to segfault again.

@binf
Collaborator

Ok.

You could also set a large enough ulimit and use the coredump for backtracing a potential crash instead of running under gdb.

If you even happen to have a unified2 file you could share with a event that cause a crahs i would also like to see it if possible.

@s-takehana

I've been getting same issue when using Suricata ET rules(http://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz).

Output from Barnyard2:

        --== Initialization Complete ==--

  ______   -*> Barnyard2 

Using waldo file '/etc/suricata/bylog.waldo':
    spool directory = /var/log/suricata
    spool filebase  = unified2.alert
    time_stamp      = 1358335191
    record_idx      = 159
Opened spool file '/var/log/suricata/unified2.alert.1358335191'
01/16-21:06:38.701717  [**] [1:2200035:1] SURICATA TCP invalid option length [**] [Classification ID: (null)] [Priority ID: 3] {TCP} ***.***.***.***:0 -> ***.***.***.***:0
INFO [dbProcessSignatureInformation()]: [Event: 80] with [gid: 1] [sid: 2200035] [rev: 1] [classification: 0] [priority: 3] Signature Message -> "[SURICATA TCP invalid option length]"
         was not found in barnyard2 signature cache, this could mean its is the first time the signature is processed, and will be inserted
         in the database with the above information, this message should only be printed once for each signature that is not  present in the database
         The new inserted signature will not have its information present in the sig_reference table,it should be present on restart
         if the information is present in the sid-msg.map file.
         You can allways update the message via a SQL query if you want it to be displayed correctly by your favorite interface

Segmentation fault (core dumped)

Output from syslog:

kernel: barnyard2[30469]: segfault at 12 ip 00000000004380d9 sp 00007fffc1b46ee0 error 4 in barnyard2[400000+5e000]

Output from gdb:

(gdb) bt
#0  0x00000000004380d9 in dbProcessEventInformation (data=0x7f7402fe9010, p=0x3eb2c40, event=0x39daf70, event_type=7, i_sig_id=1996) at spo_database.c:2001
#1  0x000000000043903e in Database (p=0x3eb2c40, event=0x39daf70, event_type=7, arg=0x7f7402fe9010) at spo_database.c:2494
#2  0x000000000041ce8c in CallOutputPlugins (out_type=OUTPUT_TYPE__SPECIAL, packet=0x3eb2c40, event=0x39daf70, event_type=7) at plugbase.c:573
#3  0x000000000041e0d3 in spoolerProcessRecord (spooler=0x3ba1880, fire_output=1) at spooler.c:701
#4  0x000000000041dc11 in ProcessContinuous (dirpath=0x15beabc "/var/log/suricata", filebase=0x15beebc "unified2.alert", record_start=0, timestamp=1358335192) at spooler.c:533
#5  0x000000000041de36 in ProcessContinuousWithWaldo (waldo=0x15be6b4) at spooler.c:621
#6  0x0000000000403882 in Barnyard2Main (argc=7, argv=0x7fffc1b472e8) at barnyard2.c:374
#7  0x00000000004036aa in main (argc=7, argv=0x7fffc1b472e8) at barnyard2.c:285

#0  0x00000000004380d9 in dbProcessEventInformation (data=0x7f7402fe9010, p=0x3eb2c40, event=0x39daf70, event_type=7, i_sig_id=1996) at spo_database.c:2001
2001                                           ntohs(p->tcph->th_urp))) != SNORT_SNPRINTF_SUCCESS)

(gdb) p p->tcph->th_urp
Cannot access memory at address 0x12

Output from u2spewfoo:

(Event)
    sensor id: 0    event id: 80    event second: 1358337998    event microsecond: 701717
    sig id: 2200035 gen id: 1   revision: 1  classification: 0
    priority: 3 ip source: ***.***.***.***  ip destination: ***.***.***.***
    src port: 0 dest port: 0    protocol: 6 impact_flag: 0  blocked: 0

Packet
    sensor id: 0    event id: 80    event second: 1358337998
    packet second: 1358337998   packet microsecond: 701717
    linktype: 1 packet_length: 66
[    0] 00 A0 DE 67 28 D9 00 12 F2 92 1A 00 81 00 00 5F  ...g(.........._
[   16] 08 00 45 00 00 30 E3 C1 40 00 6F 06 2B 4B 32 12  ..E..0..@.o.+K2.
[   32] 75 B8 CB B2 89 3E 00 00 00 00 55 C3 69 82 00 00  u....>....U.i...
[   48] 00 00 04 28 1F 90 91 39 00 00 99 02 97 05 00 00  ...(...9........
[   64] 00 00                                            ..

This isuue happened with below alert too.

SURICATA UDP packet too small
ET RBN Known Russian Business Network IP (423)

Could you resolve?

@binf
Collaborator
@s-takehana

How about bug-fix.
Could you tell me, how long will it take?

@binf
Collaborator

Sorry for the delay , i honestly forgot about it.

The branch containing the fix can be found there -> https://github.com/binf/barnyard2/tree/bug-fix

Let us know how its goes and again, sorry for the delay.

@s-takehana

I've updated to bug-fix and tried again.
I've confirmed writting to database whithout segfault.

Thank you very much for your help.

@reedloden
@binf
Collaborator
@reedloden

Going to close this out...

@reedloden reedloden closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.