heap-buffer-overflow in bmp_compress1_row

[YourButterfly]

Description of problem:

There is a heap-buffer-overflow in bmp_compress1_row.

Version-Release number of selected component (if applicable):

<= latest version

The output information is as follows:

$ ./sam2p poc try.bmp
This is sam2p 0.49.4.
Available Loaders: PS PDF JAI PNG JPEG TIFF PNM BMP LBM XPM PCX TGA.
Available Appliers: XWD Meta Empty BMP PNG TIFF6 TIFF6-JAI JPEG-JAI JPEG PNM XPM PSL1C PSL23+PDF PSL2+PDF-JAI P-TrOpBb.
sam2p: Notice: job: read InputFile: poc
sam2p: Notice: SampleFormat: Gray8 would be better than Indexed8
sam2p: appliers.cpp:2137: Rule::Applier::cons_t out_bmp_work(GenBuffer::Writable &, Rule::OutputRule *, Image::SampledInfo *): Assertion `(slen_t)(crow2-crow)<=crowsize-2' failed.
Aborted (core dumped)

The gdb debugging information is listed below:(with asan)

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
This is sam2p 0.49.4.
Available Loaders: PS PDF JAI PNG JPEG TIFF PNM BMP LBM XPM PCX TGA.
Available Appliers: XWD Meta Empty BMP PNG TIFF6 TIFF6-JAI JPEG-JAI JPEG PNM XPM PSL1C PSL23+PDF PSL2+PDF-JAI P-TrOpBb.
sam2p: Notice: job: read InputFile: /home/pwd/Desktop/crashes-2018-06-19-19-02/sam2p000_id_000096\,sig_06\,src_000412\,op_flip1\,pos_12
sam2p: Notice: SampleFormat: Gray8 would be better than Indexed8
=================================================================
==54172==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000cc0 at pc 0x55555559518a bp 0x7fffffffc720 sp 0x7fffffffc710
WRITE of size 1 at 0x604000000cc0 thread T0
    #0 0x555555595189 in bmp_compress1_row /home/pwd/fuzz/fuzz-sam2p/sam2p/appliers.cpp:1968
    #1 0x555555595189 in out_bmp_work(GenBuffer::Writable&, Rule::OutputRule*, Image::SampledInfo*) /home/pwd/fuzz/fuzz-sam2p/sam2p/appliers.cpp:2120
    #2 0x5555555c421e in Rule::applyProfile(GenBuffer::Writable&, Rule::OutputRule*, Image::SampledInfo*) /home/pwd/fuzz/fuzz-sam2p/sam2p/rule.cpp:680
    #3 0x55555557e753 in run_sam2p_engine(Files::FILEW&, Files::FILEW&, char const* const*, bool) /home/pwd/fuzz/fuzz-sam2p/sam2p/sam2p_main.cpp:1079
    #4 0x555555579d50 in main /home/pwd/fuzz/fuzz-sam2p/sam2p/sam2p_main.cpp:1148
    #5 0x7ffff66bcb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #6 0x55555557a9d9 in _start (/home/pwd/fuzz/fuzz-sam2p/sam2p/sam2p+0x269d9)

0x604000000cc0 is located 0 bytes to the right of 48-byte region [0x604000000c90,0x604000000cc0)
allocated by thread T0 here:
    #0 0x7ffff6efa618 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0618)
    #1 0x5555555926e3 in out_bmp_work(GenBuffer::Writable&, Rule::OutputRule*, Image::SampledInfo*) /home/pwd/fuzz/fuzz-sam2p/sam2p/appliers.cpp:2114

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/pwd/fuzz/fuzz-sam2p/sam2p/appliers.cpp:1968 in bmp_compress1_row
Shadow bytes around the buggy address:
  0x0c087fff8140: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c087fff8150: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c087fff8160: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c087fff8170: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x0c087fff8180: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 00
=>0x0c087fff8190: fa fa 00 00 00 00 00 00[fa]fa fa fa fa fa fa fa
  0x0c087fff81a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff81b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff81c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff81d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff81e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==54172==ABORTING
[Inferior 1 (process 54172) exited with code 01]

that is a breif description，i will update
found by pwd@360TeamSerious

[YourButterfly]

poc

[pts]

Thank you for reporting this!

FYI In debug mode it fails with:

sam2p.yes: appliers.cpp:2138: Rule::Applier::cons_t out_bmp_work(GenBuffer::Writable&, Rule::OutputRule*, Image::SampledInfo*): Assertion `(slen_t)(crow2-crow)<=crowsize-2' failed.

It looks like the formula slen_t crowsize=2+ rlen+(rlen+128)*2/255; is wrong, and we need a larger value there.

[pts]

It looks like bmp_compress1_row has a subtle bug, and the implementation is hard to understand, so it needs a full rewrite.

[pts]

Fixed it in 22e7a17 by rewriting bmp_compress1_row from scratch.