Segmentation fault in cmdline input provider

[bostjan]

Originally reported by @ajzach in issue #39:

Error with master version (ff587e):

#  SIGSEGV (0xb) at pc=0x00007fbd09d93a03, pid=1507, tid=140447328962304
#
# JRE version: Java(TM) SE Runtime Environment (8.0_11-b12) (build 1.8.0_11-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.11-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x4ba03]  _IO_vfprintf+0x1d13
..............................
..............................
..............................
Stack: [0x00007fbc71170000,0x00007fbc71271000],  sp=0x00007fbc7126bc20,  free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x4ba03]  _IO_vfprintf+0x1d13

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.lang.UNIXProcess.forkAndExec(I[B[B[BI[BI[B[IZ)I+0
j  java.lang.UNIXProcess.<init>([B[BI[BI[B[IZ)V+28
J 12594 C1 java.lang.ProcessImpl.start([Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;[Ljava/lang/ProcessBuilder$Redirect;Z)Ljava/lang/Process; (634 bytes) @ 0x00007fbcf7129fd4 [0x00007fbcf7128fc0+0x1014]
J 12304 C1 java.lang.ProcessBuilder.start()Ljava/lang/Process; (303 bytes) @ 0x00007fbcf70184cc [0x00007fbcf7017ac0+0xa0c]
j  hudson.Proc$LocalProc.<init>(Ljava/lang/String;Ljava/lang/ProcessBuilder;Ljava/io/InputStream;Ljava/io/OutputStream;Ljava/io/OutputStream;)V+55
J 12588 C1 hudson.Proc$LocalProc.<init>([Ljava/lang/String;[Ljava/lang/String;Ljava/io/InputStream;Ljava/io/OutputStream;Ljava/io/OutputStream;Ljava/io/File;)V (52 bytes) @ 0x00007fbcf7124f7c [0x00007fbcf7124b40+0x43c]

#2085 __GI_abort () at abort.c:125
---Type <return> to continue, or q <return> to quit---
        act = {__sigaction_handler = {sa_handler = 0x7fbd04146ce4, sa_sigaction = 0x7fbd04146ce4}, sa_mask = {__val = {140449793993955, 
              140449793993955, 140449885647857, 11, 140447328939072, 140449793993956, 0, 0, 0, 0, 140449898820457, 0, 140449883661430, 0, 
              140449898820457, 0}}, sa_flags = 158114934, sa_restorer = 0x7fbc7126a9a0}
        sigs = {__val = {0, 0, 0, 0, 0, 140447328938208, 140447328938928, 140447328938400, 11, 140447328939072, 140447328938928, 140449881522244, 
            206158430248, 140447328938416, 140447328938224, 140449793993956}}
#2086 0x00007fbd096bd339 in os::die() () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2087 0x00007fbd0984f7c2 in VMError::report_and_die() () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2088 0x00007fbd0984fe4e in crash_handler(int, siginfo*, void*) () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2089 <signal handler called>
No locals.
#2090 __GI_abort () at abort.c:125
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {18446744073709551615 <repeats 16 times>}}, 
          sa_flags = 0, sa_restorer = 0x0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2091 0x00007fbd096bf115 in os::abort(bool) () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2092 0x00007fbd0984f9a3 in VMError::report_and_die() () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2093 0x00007fbd096c44e2 in JVM_handle_linux_signal () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2094 0x00007fbd096bb463 in signalHandler(int, siginfo*, void*) () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2095 <signal handler called>
No locals.
#2096 0x00007fbd09d93a03 in _IO_vfprintf_internal (s=s@entry=0x7fbc7126c210, format=<optimized out>, format@entry=0x7fbd0a7578d0 "%s", 
    ap=ap@entry=0x7fbc7126c378) at vfprintf.c:1661
        len = <optimized out>
        string_malloced = <optimized out>
        step0_jumps = {0, -9485, -2676, -2589, 2607, 2694, 2015, 2291, 2998, -574, -238, 3335, 2510, 3607, -2495, -15046, -727, 1092, 1034, 1620, 
          -14278, -28, 1449, -10481, -10405, -15728, 1138, 3510, 3510, 2188}
        space = 0
        is_short = 0
        use_outdigits = 0
        step1_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, -574, -238, 3335, 2510, 3607, -2495, -15046, -727, 1092, 1034, 1620, -14278, -28, 1449, -10481, 
          -10405, -15728, 1138, 3510, 3510, 0}
---Type <return> to continue, or q <return> to quit---
        group = 0
        prec = -1
        step2_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -238, 3335, 2510, 3607, -2495, -15046, -727, 1092, 1034, 1620, -14278, -28, 1449, -10481, 
          -10405, -15728, 1138, 3510, 3510, 0}
        string = <optimized out>
        left = 0
        is_long_double = 0
        width = 0
        step3a_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3238, 0, 0, 0, -2495, -15046, -727, 1092, 1034, 0, 0, 0, 0, -10481, 0, 0, 0, 0, 0, 0}
        alt = 0
        showsign = 0
        is_long = 0
        is_char = 0
        pad = 32 ' '
        step3b_jumps = {0 <repeats 11 times>, 2510, 0, 0, -2495, -15046, -727, 1092, 1034, 1620, -14278, -28, 1449, -10481, -10405, -15728, 1138, 0, 
          0, 0}
        step4_jumps = {0 <repeats 14 times>, -2495, -15046, -727, 1092, 1034, 1620, -14278, -28, 1449, -10481, -10405, -15728, 1138, 0, 0, 0}
        is_negative = <optimized out>
        number = <optimized out>
        base = <optimized out>
        the_arg = {pa_wchar = 175470802 L'\xa7578d2', pa_int = 175470802, pa_long_int = 140449901017298, pa_long_long_int = 140449901017298, 
          pa_u_int = 175470802, pa_u_long_int = 140449901017298, pa_u_long_long_int = 140449901017298, pa_double = 6.9391471054450486e-310, 
          pa_long_double = <invalid float value>, pa_string = 0x7fbd0a7578d2 "", 
          pa_wstring = 0x7fbd0a7578d2 <error: Cannot access memory at address 0x7fbd0a7578d2>, pa_pointer = 0x7fbd0a7578d2, pa_user = 0x7fbd0a7578d2}
        spec = 115 's'
        _buffer = {__routine = 0x3000000018, __arg = 0x7fbc7126c470, __canceltype = 1898365872, __prev = 0x0}
        _avail = <optimized out>
        thousands_sep = 0x0
        grouping = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>
        done = 0
        f = 0x7fbd0a7578d1 "s"
        lead_str_end = <optimized out>
        end_of_spec = <optimized out>
        work_buffer = "(\000\000\000\060\000\000\000\240\304&q\274\177\000\000\340\303&q\274\177\000\000\240\304&q\274\177\000\000\340\303&q\274\177", '\000' <repeats 589 times>...
        workstart = 0x0
        workend = <optimized out>
        ap_save = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7fbc7126c450, reg_save_area = 0x7fbc7126c390}}
---Type <return> to continue, or q <return> to quit---
        nspecs_done = 0
        save_errno = <optimized out>
        readonly_format = 0
        args_malloced = 0x0
        specs = <optimized out>
        specs_malloced = false


        jump_table = "\001\000\000\004\000\016\000\006\000\000\a\002\000\003\t\000\005\b\b\b\b\b\b\b\b\b\000\000\000\000\000\000\000\032\000\031\000\023\023\023\000\035\000\000\f\000\000\000\000\000\000\025\000\000\000\000\022\000\r\000\000\000\000\000\000\032\000\024\017\023\023\023\n\017\034\000\v\030\027\021\026\f\000\025\033\020\000\000\022\000\r"
        __PRETTY_FUNCTION__ = "_IO_vfprintf_internal"
#2097 0x00007fbd09dba579 in _IO_vsnprintf (string=0x7fbcd4030636 "", maxlen=<optimized out>, format=0x7fbd0a7578d0 "%s", 
    args=args@entry=0x7fbc7126c378) at vsnprintf.c:119
        sf = {f = {_sbf = {_f = {_flags = -72515583, _IO_read_ptr = 0x7fbcd4030636 "", _IO_read_end = 0x7fbcd4030636 "", 
                _IO_read_base = 0x7fbcd4030636 "", _IO_write_base = 0x7fbcd4030636 "", _IO_write_ptr = 0x7fbcd4030636 "", 
                _IO_write_end = 0x7fbcd4030642 "", _IO_buf_base = 0x7fbcd4030636 "", _IO_buf_end = 0x7fbcd4030642 "", _IO_save_base = 0x0, 
                _IO_backup_base = 0x0, _IO_save_end = 0x0, _markers = 0x0, _chain = 0x0, _fileno = 1898365760, _flags2 = 0, 
                _old_offset = 140447328944960, _cur_column = 0, _vtable_offset = 38 '&', _shortbuf = "q", _lock = 0x0, _offset = 0, _codecvt = 0x0, 
                _wide_data = 0xffffffffffffffff, _freeres_list = 0x0, _freeres_buf = 0x400000000, _freeres_size = 0, _mode = -1, 
                _unused2 = '\000' <repeats 19 times>}, vtable = 0x7fbd0a104fe0 <_IO_strn_jumps>}, _s = {_allocate_buffer = 0x0, 
              _free_buffer = 0x0}}, 
          overflow_buf = "\377\377\377\377\377\377\377\377\000\000\000\000\000\000\000\000\377\377\377\377", '\000' <repeats 12 times>, "\377\377\377\377\000\000\000\000\340O\020\n\275\177\000\000\000\000\000\000\000\000\000\000\340O\020\n\275\177\000"}
        ret = <optimized out>
#2098 0x00007fbd09d9c532 in __snprintf (s=s@entry=0x7fbcd4030636 "", maxlen=maxlen@entry=13, format=format@entry=0x7fbd0a7578d0 "%s")
    at snprintf.c:33
        arg = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fbc7126c450, reg_save_area = 0x7fbc7126c390}}
        done = 0
#2099 0x00007fbd0a756307 in snprintf (__fmt=0x7fbd0a7578d0 "%s", __n=13, __s=0x7fbcd4030636 "") at /usr/include/x86_64-linux-gnu/bits/stdio2.h:64
No locals.
#2100 snoopy_input_cmdline (input=0x7fbc7126cd20 "", arg=<optimized out>) at cmdline.c:75
        cmdLine = 0x7fbcd40305f0 "git config remote.origin.url http://git.xxxxxxxx.it/git/yyyyy/ (null) "
        cmdLineSize = 83
        argc = <optimized out>
        i = <optimized out>
        n = 70
#2101 0x00007fbd0a7546e4 in snoopy_log_message_generate (
    logMessage=logMessage@entry=0x7fbcd4034110 "[rpname:daemon user:jenkins ip:(undefined) uid:115 sid:1269 tty:(none) cwd:/ filename:/usr/bin/git]: ", logMessageFormat=<optimized out>) at log.c:140
---Type <return> to continue, or q <return> to quit---
        lengthToCopy = <optimized out>
        inputTag = "cmdline\000\000LIENT", '\000' <repeats 18 times>, "B\000\000\000\000\000\000\000\070\345&q\274\177\000\000snoopy:message_format\000\357D\313'k\"\207Մ\020\003\374/\323\037E;NQ\307\"\322Fa\b\257\b#\372", <incomplete sequence \351>
        fmtPos_inputTagArg = <optimized out>
        inputProviderNamePtr = 0x7fbc7126c4b0 "cmdline"
        inputProviderArgPtr = 0x7fbc7126c920 ""
        inputProviderMsg = "\000usr/bin/git\000ername} ip:%{env:SSH_CLIENT} uid:%{uid} sid:%{sid} tty:%{tty} cwd:%{cwd} filename:%{filename}]: %{cmdline}\"\000\317&q\274\177\000\000\360\316&q\274\177", '\000' <repeats 18 times>, "PE\270\n\275\177\000\000h\266\270\n\275\177\000\000ɫ\226\n\275\177\000\000\001", '\000' <repeats 19 times>, "\274\177\000\000h"...
        fmtStaticText = "]: \000ename:\000\b\275\177\000\000\220\345&q\274\177\000\000P\305&q\274\177\000\000\320\313Fq\274\177\000\000P\305&q\274\177\000\000\n\345&q\274\177\000\000@\265Cq\274\177\000\000\270\310Fqit --version\330\016Dqimeout=10\177\000\000(d\031\000\001\000\000\000\300\305&q\274\177\000\000\320QDq\000\000\000\000\060\306&q\274\177\000\000\220\306&q\274\177\000\000(d\031\000\001\000\000\000\360\305&q\274\177\000\000(\204\023p\275\177\000\000H\345Cq\274\177\000\000\000P/\300\274\177\000\000\020\334&q\274\177\000\000 \026Dq\274\177\000\000\000\000\000\000\000\000\000\000"...
        inputProviderArg = "\000LIENT} uid:%{uid} sid:%{sid} tty:%{tty} cwd:%{cwd} filename:%{filename}]: %{cmdline}", '\000' <repeats 891 times>...
        fmtPos_cur = <optimized out>
        fmtPos_nextInputTag = 0x7fbcd402e7cb "%{cmdline}"
        fmtPos_nextInputTagClose = 0x7fbcd402e7d4 "}"
#2102 0x00007fbd0a754a70 in snoopy_log_syscall (syscallName=syscallName@entry=0x7fbd0a757416 "execve") at log.c:385
        logMessage = 0x7fbcd4034110 "[rpname:daemon user:jenkins ip:(undefined) uid:115 sid:1269 tty:(none) cwd:/ filename:/usr/bin/git]: "
#2103 0x00007fbd0a754b33 in snoopy_log_syscall_execve (filename=filename@entry=0x7fbc7126d220 "/usr/bin/git", argv=argv@entry=0x7fbcd402ed30, 
    envp=envp@entry=0x7fbcd402b4f0) at log.c:348
No locals.
#2104 0x00007fbd0a753d25 in execve (filename=0x7fbc7126d220 "/usr/bin/git", argv=0x7fbcd402ed30, envp=0x7fbcd402b4f0) at libsnoopy.c:76
        func = 0x7fbd09e09330 <__execve>
#2105 0x00007fbd084e6e8e in execve_with_shell_fallback () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
No symbol table info available.
#2106 0x00007fbd084e70e1 in JDK_execvpe () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
No symbol table info available.
#2107 0x00007fbd084e74fb in childProcess () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
No symbol table info available.
#2108 0x00007fbd084e4474 in vforkChild () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
No symbol table info available.
#2109 0x00007fbd084e4dc0 in Java_java_lang_UNIXProcess_forkAndExec () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so

b.

[bostjan]

Additional debugging output by @ajzach:

2° error using the master version (ff587e):

#2093 __GI_abort () at abort.c:125
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {18446744073709551615 <repeats 16 times>}}, 
          sa_flags = 0, sa_restorer = 0x0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2094 0x00007f89b4a97115 in os::abort(bool) () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2095 0x00007f89b4c279a3 in VMError::report_and_die() () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2096 0x00007f89b4a9c4e2 in JVM_handle_linux_signal () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2097 0x00007f89b4a93463 in signalHandler(int, siginfo*, void*) () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
No symbol table info available.
#2098 <signal handler called>
No locals.
#2099 strlen () at ../sysdeps/x86_64/strlen.S:106
No locals.
#2100 0x00007f89b5b2e284 in snoopy_input_cmdline (input=0x7f8994210d50 "", arg=<optimized out>) at cmdline.c:65
        cmdLine = 0x0
        cmdLineSize = 6
        argc = 5
        i = <optimized out>
---Type <return> to continue, or q <return> to quit---
        n = <optimized out>
#2101 0x00007f89b5b2c6e4 in snoopy_log_message_generate (
    logMessage=logMessage@entry=0x7f893000ca30 "[rpname:daemon user:jenkins ip:(undefined) uid:115 sid:26418 tty:(none) cwd:/ filename:\240\037]: ", 
    logMessageFormat=<optimized out>) at log.c:140
        lengthToCopy = <optimized out>
        inputTag = "cmdline\000\000LIENT", '\000' <repeats 18 times>, "\250\a\000\000\001\000\000\000\250\a\000\000\001\000\000\000snoopy:message_format\000\000\000\340\005!\224\211\177\000\000\000`\000`\211\177", '\000' <repeats 11 times>, "`\000`"
        fmtPos_inputTagArg = <optimized out>
        inputProviderNamePtr = 0x7f89942104e0 "cmdline"
        inputProviderArgPtr = 0x7f8994210950 ""
        inputProviderMsg = "\000\037\000ne)\000ned)\000sername} ip:%{env:SSH_CLIENT} uid:%{uid} sid:%{sid} tty:%{tty} cwd:%{cwd} filename:%{filename}]: %{cmdline}\"\000\017!\224\211\177\000\000 \017!\224\211\177", '\000' <repeats 18 times>, "P\305\365\265\211\177\000\000\350\342\365\265\211\177\000\000\311+Ե\211\177\000\000\001", '\000' <repeats 19 times>, "\004\000\000\000\350"...
        fmtStaticText = "]: \000ename:\000`\211\177\000\000\000`\000`\211\177\000\000\311s\221\265\211\177\000\000\000\000\000\000\000\000\000\000 ", '\000' <repeats 15 times>, "\340a\000`\211\177\000\000 \000\000\000\000\000\000\000\340\005!\224\211\177\000\000\020&!\224\211\177\000\000B\373\213\263\211\177\000\000\000 \000X\211\177\000\000\340\005!\224\211\177\000\000B\000\000\000\000\000\000\000(&!\224\211\177\000\000 &!\224\211\177\000\000\340\216\006\235\211\177\000\000\301)5%%?\255Kc\340\332I\220\255\352u\bX\351\r\006!;\035\241\376r\315\023\354a\373\340!\000X\211\177\000\000\001\000\000\000\000\000\000\000p&!\224\211\177\000\000"...
        inputProviderArg = "\000LIENT} uid:%{uid} sid:%{sid} tty:%{tty} cwd:%{cwd} filename:%{filename}]: %{cmdline}", '\000' <repeats 891 times>...
        fmtPos_cur = <optimized out>
        fmtPos_nextInputTag = 0x7f89303b732b "%{cmdline}"
        fmtPos_nextInputTagClose = 0x7f89303b7334 "}"
#2102 0x00007f89b5b2ca70 in snoopy_log_syscall (syscallName=syscallName@entry=0x7f89b5b2f416 "execve") at log.c:385
        logMessage = 0x7f893000ca30 "[rpname:daemon user:jenkins ip:(undefined) uid:115 sid:26418 tty:(none) cwd:/ filename:\240\037]: "
#2103 0x00007f89b5b2cb33 in snoopy_log_syscall_execve (filename=filename@entry=0x7f8994211250 "/usr/local/bin/git", argv=argv@entry=0x7f89303229b0, 
    envp=envp@entry=0x7f8930007ce0) at log.c:348
No locals.
#2104 0x00007f89b5b2bd25 in execve (filename=0x7f8994211250 "/usr/local/bin/git", argv=0x7f89303229b0, envp=0x7f8930007ce0) at libsnoopy.c:76
        func = 0x7f89b51e1330 <__execve>
#2105 0x00007f89b38bee8e in execve_with_shell_fallback () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
No symbol table info available.
#2106 0x00007f89b38bf0e1 in JDK_execvpe () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
No symbol table info available.
#2107 0x00007f89b38bf4fb in childProcess () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
No symbol table info available.
#2108 0x00007f89b38bc474 in vforkChild () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
No symbol table info available.
#2109 0x00007f89b38bcdc0 in Java_java_lang_UNIXProcess_forkAndExec () from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
---Type <return> to continue, or q <return> to quit---

[bostjan]

@ajzach: could you please test this with master 5c3578a?

Master is being prepared for Snoopy 2.3.0 release, and cmdline input provider has been reviewed, and a potential buffer overflow has been fixed.

Tnx,
b.

[tmccombs]

I'm still seeing this with snoopy 2.3.1. I also notice that the segfault seems to be coming from strlen. Which makes me think the segfault is happening on line 66 of the current master:

 /* Argument length + space */
        cmdLineSizeSum += strlen(snoopy_inputdatastorage_argv[i]) + 1;

So somehow snoopy_inputdatastorage_argv is either NULL or not a null-terminated string.

[bostjan]

Hi Thayne,

A quote from README:
Only latest released version of Snoopy is "supported"! Any version that is not the latest (stable or development) is not supported. Please do not submit any bugfix/feature/change requests related to old and thus unsupported Snoopy versions.

Retest it with latest version.

PS: Where are you guys getting these old versions from?

[tmccombs]

We are using Ubuntu 16.04. 2.3.1 is the latest version in the Xenial repository. Using a more recent version would be pretty difficult for us.

[bostjan]

There is a contact for Ubuntu build maintainer on the build project page (not maintained by me):
https://packages.ubuntu.com/xenial/admin/snoopy

2.3.1 is not the latest available Snoopy version :)