Test 140: libarchive_test_fuzz_tar fails in VS2015

[hinrichsenhans]

Basic Information
Version of libarchive: pulled from master (8adeb0a - last of 2015)
How you obtained it: (build from source, pre-packaged binary, etc) built from source
Operating system and version: Windows 10 Home Premium 64bit
What compiler and/or IDE you are using (include version): VS2015

Description of the problem you are seeing:
What did you do? Ran build process. Ran tests. (Had to do a number of cleanup things involving VS2015 and snprintf being defined as a macro (will submit pull request soon - it's in the hinrichsenhans/libarchive vs2015 branch).)
What did you expect to happen? Tests pass
What actually happened? Tests failed
What log files or error messages were produced? Pasted below. This is a report for test 140 - there were other failures that I'll look at and report soon.
18:libarchive_test_archive_match_time
140:libarchive_test_fuzz_tar
333:libarchive_test_read_format_zip_utf8_paths
371:libarchive_test_warn_missing_hardlink_target
479:bsdcat_test_error
480:bsdcat_test_error_mixed
506:bsdtar_test_leading_slash
557:bsdcpio_test_option_a

How the libarchive developers can reproduce your problem:
What other software was involved? Just VS2015, cmake, libarchive, and the gnuwin32 binaries/libraries
What other files were involved? none
How can we obtain any of the above?

Test 140 shows an empty test_fuzz_tar.log file. From the LastTest.log:

140/580 Testing: libarchive_test_fuzz_tar
140/580 Test: libarchive_test_fuzz_tar
Command: "C:/libarchive_build/libarchive_bin/bin/Debug/libarchive_test.exe" "-vv" "-r" "C:/libarchive_build/libarchive/libarchive/test" "test_fuzz_tar"
Directory: C:/libarchive_build/libarchive_bin/libarchive/test
"libarchive_test_fuzz_tar" start time: Jan 06 08:43 Central Standard Time
Output:
. ----------------------------------------------------------

If tests fail or crash, details will be in:
C:\Users\Hans\AppData\Local\Temp/libarchive_test.exe.2016-01-06T08.43.16-000

Reference files will be read from: C:/libarchive_build/libarchive/libarchive/test
Exercising:

Test time = 6.43 sec
. ----------------------------------------------------------
Test Failed.
"libarchive_test_fuzz_tar" end time: Jan 06 08:43 Central Standard Time
"libarchive_test_fuzz_tar" time elapsed: 00:00:06
. ----------------------------------------------------------

Folder contents here:
libarchive_test.exe.2016-01-06T08.43.16-000.zip

[kientzle]

Test 140 shows an empty test_fuzz_tar.log file. From the LastTest.

This is expected; test_fuzz_tar doesn't log very much. It just tries to crash libarchive by feeding it randomly corrupted data. Would it be possible for you to run this particular test under a debugger and give us a stack trace?

[ismail]

This test fails for me on Linux with gcc5 (and also with clang too) , how can I run it under the debugger? I didn't see a standalone executable.

[kientzle]

The test suite consists of three standalone executables: libarchive_test, bsdtar_test, and bsdcpio_test.

libarchive_test requires a -r option with the full path to the libarchive/test directory (which contains the various reference files used in the test). I usually run it with -v -v -v to get more output to the console. So I recommend running it under the debugger as:

libarchive_test -r path_to_libarchive/test -vvv test_fuzz_tar

[ismail]

Can't get it to crash with debug enabled but valgrind shows the culprit (crash was in bzip2 code):

==5345== 1 errors in context 1 of 6:
==5345== Use of uninitialised value of size 8
==5345==    at 0x58C1A57: BZ2_decompress (decompress.c:547)
==5345==    by 0x58C4370: BZ2_bzDecompress (bzlib.c:842)
==5345==    by 0x50C7C4: bzip2_filter_read (archive_read_support_filter_bzip2.c:308)
==5345==    by 0x505947: __archive_read_filter_ahead (archive_read.c:1366)
==5345==    by 0x50C5B6: bzip2_reader_bid (archive_read_support_filter_bzip2.c:134)
==5345==    by 0x504520: choose_filters (archive_read.c:568)
==5345==    by 0x504520: archive_read_open1 (archive_read.c:512)
==5345==    by 0x446915: test_fuzz (test_fuzz.c:188)
==5345==    by 0x41CA9B: test_run (main.c:2405)
==5345==    by 0x41CA9B: main (main.c:2871)
==5345==
==5345==
==5345== 1 errors in context 2 of 6:
==5345== Conditional jump or move depends on uninitialised value(s)
==5345==    at 0x58C2EE3: BZ2_decompress (decompress.c:449)
==5345==    by 0x58C4370: BZ2_bzDecompress (bzlib.c:842)
==5345==    by 0x50C7C4: bzip2_filter_read (archive_read_support_filter_bzip2.c:308)
==5345==    by 0x505947: __archive_read_filter_ahead (archive_read.c:1366)
==5345==    by 0x50C5B6: bzip2_reader_bid (archive_read_support_filter_bzip2.c:134)
==5345==    by 0x504520: choose_filters (archive_read.c:568)
==5345==    by 0x504520: archive_read_open1 (archive_read.c:512)
==5345==    by 0x446915: test_fuzz (test_fuzz.c:188)
==5345==    by 0x41CA9B: test_run (main.c:2405)
==5345==    by 0x41CA9B: main (main.c:2871)
==5345==
==5345==
==5345== 1 errors in context 3 of 6:
==5345== Use of uninitialised value of size 8
==5345==    at 0x58C2EDE: BZ2_decompress (decompress.c:448)
==5345==    by 0x58C4370: BZ2_bzDecompress (bzlib.c:842)
==5345==    by 0x50C7C4: bzip2_filter_read (archive_read_support_filter_bzip2.c:308)
==5345==    by 0x505947: __archive_read_filter_ahead (archive_read.c:1366)
==5345==    by 0x50C5B6: bzip2_reader_bid (archive_read_support_filter_bzip2.c:134)
==5345==    by 0x504520: choose_filters (archive_read.c:568)
==5345==    by 0x504520: archive_read_open1 (archive_read.c:512)
==5345==    by 0x446915: test_fuzz (test_fuzz.c:188)
==5345==    by 0x41CA9B: test_run (main.c:2405)
==5345==    by 0x41CA9B: main (main.c:2871)
==5345==
==5345==
==5345== 1 errors in context 4 of 6:
==5345== Conditional jump or move depends on uninitialised value(s)
==5345==    at 0x58C14DD: BZ2_decompress (decompress.c:443)
==5345==    by 0x58C4370: BZ2_bzDecompress (bzlib.c:842)
==5345==    by 0x50C7C4: bzip2_filter_read (archive_read_support_filter_bzip2.c:308)
==5345==    by 0x505947: __archive_read_filter_ahead (archive_read.c:1366)
==5345==    by 0x50C5B6: bzip2_reader_bid (archive_read_support_filter_bzip2.c:134)
==5345==    by 0x504520: choose_filters (archive_read.c:568)
==5345==    by 0x504520: archive_read_open1 (archive_read.c:512)
==5345==    by 0x446915: test_fuzz (test_fuzz.c:188)
==5345==    by 0x41CA9B: test_run (main.c:2405)
==5345==    by 0x41CA9B: main (main.c:2871)
==5345==
==5345==
==5345== 4 errors in context 5 of 6:
==5345== Conditional jump or move depends on uninitialised value(s)
==5345==    at 0x58C14BD: BZ2_decompress (decompress.c:392)
==5345==    by 0x58C4370: BZ2_bzDecompress (bzlib.c:842)
==5345==    by 0x50C7C4: bzip2_filter_read (archive_read_support_filter_bzip2.c:308)
==5345==    by 0x505947: __archive_read_filter_ahead (archive_read.c:1366)
==5345==    by 0x50C5B6: bzip2_reader_bid (archive_read_support_filter_bzip2.c:134)
==5345==    by 0x504520: choose_filters (archive_read.c:568)
==5345==    by 0x504520: archive_read_open1 (archive_read.c:512)
==5345==    by 0x446825: test_fuzz (test_fuzz.c:171)
==5345==    by 0x41CA9B: test_run (main.c:2405)
==5345==    by 0x41CA9B: main (main.c:2871)
==5345==
==5345==
==5345== 4 errors in context 6 of 6:
==5345== Conditional jump or move depends on uninitialised value(s)
==5345==    at 0x58C14B3: BZ2_decompress (decompress.c:390)
==5345==    by 0x58C4370: BZ2_bzDecompress (bzlib.c:842)
==5345==    by 0x50C7C4: bzip2_filter_read (archive_read_support_filter_bzip2.c:308)
==5345==    by 0x505947: __archive_read_filter_ahead (archive_read.c:1366)
==5345==    by 0x50C5B6: bzip2_reader_bid (archive_read_support_filter_bzip2.c:134)
==5345==    by 0x504520: choose_filters (archive_read.c:568)
==5345==    by 0x504520: archive_read_open1 (archive_read.c:512)
==5345==    by 0x446825: test_fuzz (test_fuzz.c:171)
==5345==    by 0x41CA9B: test_run (main.c:2405)
==5345==    by 0x41CA9B: main (main.c:2871)
==5345==

[kientzle]

This could be a red herring: Reading uninitialized data isn't always a bug.

But it is worth checking further:

1. I suggest you look to see if there is a newer version of the BZ2 libraries for your system.
2. test_fuzz.c has several parameters in the source code to determine how many iterations it runs. I suggest increasing those significantly to see if you can reproduce the actual crash. Since test_fuzz generates random test data, it can require some effort to reproduce failures it finds. I suggest increasing the number of iterations and trying to run it for several hours. (I'll do the same here.)

[ismail]

Here is a gdb session with the backtrace, let me know if you need more details:

Program received signal SIGSEGV, Segmentation fault.
BZ2_decompress (s=s@entry=0x4e06570) at decompress.c:448
448                    uc = ppx[nextSym];
(gdb) print *ppx
$1 = 0 '\000'
(gdb) print nextSym
$2 = -1061109569
(gdb) frame 1
#1  0x00007f03a804d371 in BZ2_bzDecompress (strm=<optimized out>) at bzlib.c:842
842              Int32 r = BZ2_decompress ( s );
(gdb) print s
$3 = (DState *) 0x4e06570
(gdb) print *s
$4 = {
  strm = 0x4df64e0,
  state = 36,
  state_out_ch = 192 '\300',
  state_out_len = -1061109568,
  blockRandomised = 0 '\000',
  rNToGo = -1061109568,
  rTPos = -1061109568,
  bsBuff = 25437,
  bsLive = 7,
  blockSize100k = 9,
  smallDecompress = 0 '\000',
  currBlockNo = 1,
  verbosity = 0,
  origPtr = 1,
  tPos = 3233857728,
  k0 = -1061109568,
  unzftab = {[0] = 0 <repeats 256 times>},
  nblock_used = -1061109568,
  cftab = {[0] = -1061109568 <repeats 257 times>},
  cftabCopy = {[0] = -1061109568 <repeats 257 times>},
  tt = 0x7f03a6831010,
  ll16 = 0x0,
  ll4 = 0x0,
  storedBlockCRC = 1286518755,
  storedCombinedCRC = 3233857728,
  calculatedBlockCRC = 3233857728,
  calculatedCombinedCRC = 0,
  nInUse = 30,
  inUse = "\001\000\001\000\000\000\001\000\000\000\001", '\000' <repeats 21 times>, "\001", '\000' <repeats 14 times>, "\001\001\001\001\001\001\001\001\001", '\000' <repeats 12 times>, "\001", '\000' <repeats 22 times>, "\001\000\000\000\000\001\000\000\001\000\000\000\000\000\001\000\001\001\000\001\000\000\000\001\001\001\001\000\000\000\000\001", '\000' <repeats 31 times>, "\001\000\000\000\000\000\000\000\000\000\001", '\000' <repeats 86 times>...,
  inUse16 = "\001\000\001\001\001\001\001\001\000\001\001\000\000\000\000\001",
  seqToUnseq = "\000\002\006\n /01234567D[`ciklnrstuz\232\244\373", '\300' <repeats 226 times>,
  mtfa = '\300' <repeats 3840 times>...,
  mtfbase = {[0] = 3840, [1] = 3856, [2] = 3872, [3] = 3888, [4] = 3904, [5] = 3920, [6] = 3936, [7] = 3952, [8] = 3968, [9] = 3984, [10] = 4000, [11] = 4016, [12] = 4032, [13] = 4048, [14] = 4064, [15] = 4080},
  selector = "\001\000\001", '\300' <repeats 17999 times>,
  selectorMtf = "\001\001\001", '\300' <repeats 17999 times>,
  len = {[0] = "\003\006\005\006\005\005\005\004\005\006\006\005\006\005\006\006\006\006\006\006\006\006\005\006\005\b\a\b\b\005\b\b", '\300' <repeats 226 times>, [1] = "\006\a\b\b\a\t\t\a\t\a\t\t\t\t\t\b", '\t' <repeats 16 times>, '\300' <repeats 226 times>, [2] = '\300' <repeats 258 times>, [3] = '\300' <repeats 258 times>, [4] = '\300' <repeats 258 times>, [5] = '\300' <repeats 258 times>},
  limit = {[0] = {[0] = -1, [1] = -1, [2] = -1, [3] = 0, [4] = 2, [5] = 15, [6] = 45, [7] = 92, [8] = 190, [9] = 2147483647, [10] = -1 <repeats 13 times>, [23] = -1061109568 <repeats 235 times>}, [1] = {[0] = -1, [1] = -1, [2] = -1, [3] = -1, [4] = -1, [5] = -1, [6] = 0, [7] = 5, [8] = 14, [9] = 53, [10] = 2147483647, [11] = -1 <repeats 12 times>, [23] = -1061109568 <repeats 235 times>}, [2] = {[0] = -1061109568 <repeats 258 times>}, [3] = {[0] = -1061109568 <repeats 258 times>}, [4] = {[0] = -1061109568 <repeats 258 times>}, [5] = {[0] = -1061109568 <repeats 258 times>}},
  base = {[0] = {[0] = 0, [1] = 0, [2] = 0, [3] = 0, [4] = 1, [5] = 4, [6] = 20, [7] = 66, [8] = 159, [9] = 0, [10] = 32 <repeats 13 times>, [23] = -1061109568 <repeats 235 times>}, [1] = {[0] = 0, [1] = 0, [2] = 0, [3] = 0, [4] = 0, [5] = 0, [6] = 0, [7] = 1, [8] = 7, [9] = 22, [10] = 0, [11] = 32 <repeats 12 times>, [23] = -1061109568 <repeats 235 times>}, [2] = {[0] = -1061109568 <repeats 258 times>}, [3] = {[0] = -1061109568 <repeats 258 times>}, [4] = {[0] = -1061109568 <repeats 258 times>}, [5] = {[0] = -1061109568 <repeats 258 times>}},
  perm = {[0] = {[0] = 0, [1] = 7, [2] = 2, [3] = 4, [4] = 5, [5] = 6, [6] = 8, [7] = 11, [8] = 13, [9] = 22, [10] = 24, [11] = 29, [12] = 1, [13] = 3, [14] = 9, [15] = 10, [16] = 12, [17] = 14, [18] = 15, [19] = 16, [20] = 17, [21] = 18, [22] = 19, [23] = 20, [24] = 21, [25] = 23, [26] = 26, [27] = 25, [28] = 27, [29] = 28, [30] = 30, [31] = 31, [32] = -1061109568 <repeats 226 times>}, [1] = {[0] = 0, [1] = 1, [2] = 4, [3] = 7, [4] = 9, [5] = 2, [6] = 3, [7] = 15, [8] = 5, [9] = 6, [10] = 8, [11] = 10, [12] = 11, [13] = 12, [14] = 13, [15] = 14, [16] = 16, [17] = 17, [18] = 18, [19] = 19, [20] = 20, [21] = 21, [22] = 22, [23] = 23, [24] = 24, [25] = 25, [26] = 26, [27] = 27, [28] = 28, [29] = 29, [30] = 30, [31] = 31, [32] = -1061109568 <repeats 226 times>}, [2] = {[0] = -1061109568 <repeats 258 times>}, [3] = {[0] = -1061109568 <repeats 258 times>}, [4] = {[0] = -1061109568 <repeats 258 times>}, [5] = {[0] = -1061109568 <repeats 258 times>}},
  minLens = {[0] = 3, [1] = 6, [2] = -1061109568, [3] = -1061109568, [4] = -1061109568, [5] = -1061109568},
  hufcode = {[0] = {[0] = 7680 <repeats 128 times>, [128] = 6663 <repeats 64 times>, [192] = 5634 <repeats 32 times>, [224] = 5636 <repeats 32 times>, [256] = 5637 <repeats 32 times>, [288] = 5638 <repeats 32 times>, [320] = 5640 <repeats 32 times>, [352] = 5643 <repeats 32 times>, [384] = 5645 <repeats 32 times>, [416] = 5654 <repeats 32 times>, [448] = 5656 <repeats 32 times>, [480] = 5661 <repeats 32 times>, [512] = 4609 <repeats 16 times>, [528] = 4611 <repeats 16 times>, [544] = 4617 <repeats 16 times>, [560] = 4618 <repeats 16 times>, [576] = 4620 <repeats 16 times>, [592] = 4622 <repeats 16 times>, [608] = 4623 <repeats 16 times>, [624] = 4624 <repeats 16 times>...}, [1] = {[0] = 4608 <repeats 16 times>, [16] = 3585, [17] = 3585, [18] = 3585, [19] = 3585, [20] = 3585, [21] = 3585, [22] = 3585, [23] = 3585, [24] = 3588, [25] = 3588, [26] = 3588, [27] = 3588, [28] = 3588, [29] = 3588, [30] = 3588, [31] = 3588, [32] = 3591, [33] = 3591, [34] = 3591, [35] = 3591, [36] = 3591, [37] = 3591, [38] = 3591, [39] = 3591, [40] = 3593, [41] = 3593, [42] = 3593, [43] = 3593, [44] = 3593, [45] = 3593, [46] = 3593, [47] = 3593, [48] = 2562, [49] = 2562, [50] = 2562, [51] = 2562, [52] = 2563, [53] = 2563, [54] = 2563, [55] = 2563, [56] = 2575, [57] = 2575, [58] = 2575, [59] = 2575, [60] = 1541, [61] = 1541, [62] = 1542, [63] = 1542, [64] = 1544, [65] = 1544, [66] = 1546, [67] = 1546, [68] = 1547, [69] = 1547, [70] = 1548, [71] = 1548, [72] = 1549, [73] = 1549, [74] = 1550, [75] = 1550, [76] = 1552, [77] = 1552, [78] = 1553, [79] = 1553, [80] = 1554, [81] = 1554, [82] = 1555, [83] = 1555, [84] = 1556, [85] = 1556, [86] = 1557, [87] = 1557, [88] = 1558, [89] = 1558, [90] = 1559, [91] = 1559, [92] = 1560, [93] = 1560, [94] = 1561, [95] = 1561, [96] = 1562, [97] = 1562, [98] = 1563, [99] = 1563, [100] = 1564, [101] = 1564, [102] = 1565, [103] = 1565, [104] = 1566, [105] = 1566, [106] = 1567, [107] = 1567, [108] = 0 <repeats 916 times>}, [2] = {[0] = -16192 <repeats 1024 times>}, [3] = {[0] = -16192 <repeats 1024 times>}, [4] = {[0] = -16192 <repeats 1024 times>}, [5] = {[0] = -16192 <repeats 1024 times>}},
  save_i = 0,
  save_j = 0,
  save_t = 0,
  save_alphaSize = 0,
  save_nGroups = 0,
  save_nSelectors = 0,
  save_EOB = 0,
  save_groupNo = 0,
  save_groupPos = 0,
  save_nextSym = 0,
  save_nblockMAX = 0,
  save_nblock = 0,
  save_es = 0,
  save_N = 0,
  save_curr = 0,
  save_zt = 0,
  save_zn = 0,
  save_zvec = 0,
  save_zj = 0,
  save_gSel = 0,
  save_gMinlen = 0,
  save_gLimit = 0x0,
  save_gBase = 0x0,
  save_gPerm = 0x0,
  save_gHufCode = 0x0
}
(gdb) frame 2
#2  0x000000000050c7c5 in bzip2_filter_read (self=<optimized out>, p=<optimized out>) at ../libarchive/archive_read_support_filter_bzip2.c:308
308                     ret = BZ2_bzDecompress(&(state->stream));
(gdb) print state
$5 = <optimized out>

[kientzle]

Unfortunately, I don't know enough about bz2lib internals to know whether the information you've shown indicates an error.

The real question is whether this is a problem with libarchive or with bz2lib. What version of bz2lib are you using? (I'm not seeing the same problem on OS X with libarchive master and bz2lib 1.0.6.)

[ismail]

The crash seems to be due to https://build.opensuse.org/package/view_file/openSUSE:Factory/bzip2/bzip2-faster.patch , I'll check with our guys (SUSE). Thanks a lot!

[kientzle]

I'm glad you tracked that down. I'll close this report for now; feel free to re-open it if it turns out that the problem is not with your bz2 library.