Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A memory leak detected. #731

Closed
fCorleone opened this issue Jul 10, 2018 · 2 comments
Closed

A memory leak detected. #731

fCorleone opened this issue Jul 10, 2018 · 2 comments

Comments

@fCorleone
Copy link

fCorleone commented Jul 10, 2018

Sorry for that I didn't reply in my last issue immediately because I had some other stuffs to handle then. Well ,it seems that you have found the SEGV signal and fixed it. I found another problem here ,it is a memory leak issue.
The program I ran is test_bgzf in test directory, the input file is bgziptest.txt.gz in the same directory.
The command line is :./test_bgzf ./bgziptest.txt
The detail information is displayed as :

=================================================================
==11095==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f7e77d58961 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98961)
    #1 0x425d8e in bgzf_getline /home/mfc_fuzz/htslib/bgzf.c:1800
    #2 0x40b885 in test_bgzf_getline test/test_bgzf.c:691
    #3 0x40c72f in main test/test_bgzf.c:774
    #4 0x7f7e770ad82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f7e77d58961 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98961)
    #1 0x425d8e in bgzf_getline /home/mfc_fuzz/htslib/bgzf.c:1800
    #2 0x40b885 in test_bgzf_getline test/test_bgzf.c:691
    #3 0x40c711 in main test/test_bgzf.c:773
    #4 0x7f7e770ad82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f7e77d58961 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98961)
    #1 0x425d8e in bgzf_getline /home/mfc_fuzz/htslib/bgzf.c:1800
    #2 0x40b885 in test_bgzf_getline test/test_bgzf.c:691
    #3 0x40c74d in main test/test_bgzf.c:775
    #4 0x7f7e770ad82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

@fCorleone
Copy link
Author

fCorleone commented Jul 10, 2018

Another issue has been found. It's also a memory leak issue. The command line is ./test_realn -i ./ce#5.sam -o out.sam -f faidx.fa

=================================================================
==10754==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x7f829603879a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    #1 0x40cb6b in fai_read /home/mfc_fuzz/htslib/faidx.c:358
    #2 0x40ef67 in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:627
    #3 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #4 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #5 0x4040e7 in main test/test_realn.c:75
    #6 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 131072 byte(s) in 1 object(s) allocated from:
    #0 0x7f8296038602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x5d523e in bgzf_read_init /home/mfc_fuzz/htslib/bgzf.c:222
    #2 0x5d6717 in bgzf_open /home/mfc_fuzz/htslib/bgzf.c:309
    #3 0x40f0df in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:640
    #4 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #5 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #6 0x4040e7 in main test/test_realn.c:75
    #7 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 4096 byte(s) in 1 object(s) allocated from:
    #0 0x7f8296038602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x413d29 in hfile_init /home/mfc_fuzz/htslib/hfile.c:105
    #2 0x4195d8 in hopen_fd /home/mfc_fuzz/htslib/hfile.c:608
    #3 0x41ffbc in hopen /home/mfc_fuzz/htslib/hfile.c:1041
    #4 0x5d66bb in bgzf_open /home/mfc_fuzz/htslib/bgzf.c:308
    #5 0x40f0df in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:640
    #6 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #7 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #8 0x4040e7 in main test/test_realn.c:75
    #9 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 256 byte(s) in 1 object(s) allocated from:
    #0 0x7f8296038961 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98961)
    #1 0x407281 in kh_resize_s /home/mfc_fuzz/htslib/faidx.c:51
    #2 0x408cd6 in kh_put_s /home/mfc_fuzz/htslib/faidx.c:51
    #3 0x409df0 in fai_insert_index /home/mfc_fuzz/htslib/faidx.c:74
    #4 0x40cfd2 in fai_read /home/mfc_fuzz/htslib/faidx.c:400
    #5 0x40ef67 in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:627
    #6 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #7 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #8 0x4040e7 in main test/test_realn.c:75
    #9 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 128 byte(s) in 1 object(s) allocated from:
    #0 0x7f8296038961 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98961)
    #1 0x40a18a in fai_insert_index /home/mfc_fuzz/htslib/faidx.c:86
    #2 0x40cfd2 in fai_read /home/mfc_fuzz/htslib/faidx.c:400
    #3 0x40ef67 in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:627
    #4 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #5 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #6 0x4040e7 in main test/test_realn.c:75
    #7 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 104 byte(s) in 1 object(s) allocated from:
    #0 0x7f829603879a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    #1 0x5d519c in bgzf_read_init /home/mfc_fuzz/htslib/bgzf.c:218
    #2 0x5d6717 in bgzf_open /home/mfc_fuzz/htslib/bgzf.c:309
    #3 0x40f0df in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:640
    #4 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #5 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #6 0x4040e7 in main test/test_realn.c:75
    #7 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f8296038602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x413cd7 in hfile_init /home/mfc_fuzz/htslib/hfile.c:98
    #2 0x4195d8 in hopen_fd /home/mfc_fuzz/htslib/hfile.c:608
    #3 0x41ffbc in hopen /home/mfc_fuzz/htslib/hfile.c:1041
    #4 0x5d66bb in bgzf_open /home/mfc_fuzz/htslib/bgzf.c:308
    #5 0x40f0df in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:640
    #6 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #7 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #8 0x4040e7 in main test/test_realn.c:75
    #9 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f8296038961 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98961)
    #1 0x407199 in kh_resize_s /home/mfc_fuzz/htslib/faidx.c:51
    #2 0x408cd6 in kh_put_s /home/mfc_fuzz/htslib/faidx.c:51
    #3 0x409df0 in fai_insert_index /home/mfc_fuzz/htslib/faidx.c:74
    #4 0x40cfd2 in fai_read /home/mfc_fuzz/htslib/faidx.c:400
    #5 0x40ef67 in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:627
    #6 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #7 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #8 0x4040e7 in main test/test_realn.c:75
    #9 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 54 byte(s) in 6 object(s) allocated from:
    #0 0x7f829600230f in strdup (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x6230f)
    #1 0x409d66 in fai_insert_index /home/mfc_fuzz/htslib/faidx.c:72
    #2 0x40cfd2 in fai_read /home/mfc_fuzz/htslib/faidx.c:400
    #3 0x40ef67 in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:627
    #4 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #5 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #6 0x4040e7 in main test/test_realn.c:75
    #7 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x7f829603879a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    #1 0x406522 in kh_init_s /home/mfc_fuzz/htslib/faidx.c:51
    #2 0x40cbc4 in fai_read /home/mfc_fuzz/htslib/faidx.c:361
    #3 0x40ef67 in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:627
    #4 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #5 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #6 0x4040e7 in main test/test_realn.c:75
    #7 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x7f829603879a in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9879a)
    #1 0x5d0c3e in kh_init_cache /home/mfc_fuzz/htslib/bgzf.c:80
    #2 0x5d5653 in bgzf_read_init /home/mfc_fuzz/htslib/bgzf.c:232
    #3 0x5d6717 in bgzf_open /home/mfc_fuzz/htslib/bgzf.c:309
    #4 0x40f0df in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:640
    #5 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #6 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #7 0x4040e7 in main test/test_realn.c:75
    #8 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f8296038602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x5d5513 in bgzf_read_init /home/mfc_fuzz/htslib/bgzf.c:228
    #2 0x5d6717 in bgzf_open /home/mfc_fuzz/htslib/bgzf.c:309
    #3 0x40f0df in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:640
    #4 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #5 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #6 0x4040e7 in main test/test_realn.c:75
    #7 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

Indirect leak of 4 byte(s) in 1 object(s) allocated from:
    #0 0x7f8296038602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
    #1 0x407009 in kh_resize_s /home/mfc_fuzz/htslib/faidx.c:51
    #2 0x408cd6 in kh_put_s /home/mfc_fuzz/htslib/faidx.c:51
    #3 0x409df0 in fai_insert_index /home/mfc_fuzz/htslib/faidx.c:74
    #4 0x40cfd2 in fai_read /home/mfc_fuzz/htslib/faidx.c:400
    #5 0x40ef67 in fai_load3_core /home/mfc_fuzz/htslib/faidx.c:627
    #6 0x40f53f in fai_load3 /home/mfc_fuzz/htslib/faidx.c:667
    #7 0x40f5a2 in fai_load /home/mfc_fuzz/htslib/faidx.c:673
    #8 0x4040e7 in main test/test_realn.c:75
    #9 0x7f829508482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: 135978 byte(s) leaked in 18 allocation(s).

@fCorleone
Copy link
Author

And there is a buffer overflow issue found as well.
The input file has been put at: https://github.com/fCorleone/fuzz_programs/blob/master/htslib/testcase1
The command line is : ./test_realn -I testcase1 -o out.sam -f faidx.fa

=================================================================
==18692==ERROR: AddressSanitizer: global-buffer-overflow on address 0x00000061f903 at pc 0x0000004845cb bp 0x7ffc4fd5bd50 sp 0x7ffc4fd5bd40
READ of size 1 at 0x00000061f903 thread T0
    #0 0x4845ca in sam_parse1 /home/mfc_fuzz/htslib/sam.c:1303
    #1 0x4878b6 in sam_read1 /home/mfc_fuzz/htslib/sam.c:1459
    #2 0x404dfe in main test/test_realn.c:111
    #3 0x7fb21be2482f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #4 0x403958 in _start (/home/mfc_fuzz/htslib/test/test_realn+0x403958)

0x00000061f903 is located 61 bytes to the left of global variable '*.LC4' defined in 'hts.c' (0x61f940) of size 22
  '*.LC4' is ascii string '1.8-40-g98efde4-dirty'
0x00000061f903 is located 27 bytes to the right of global variable '__ac_HASH_UPPER' defined in './htslib/khash.h:192:21' (0x61f8e0) of size 8
SUMMARY: AddressSanitizer: global-buffer-overflow /home/mfc_fuzz/htslib/sam.c:1303 sam_parse1
Shadow bytes around the buggy address:
  0x0000800bbed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0000800bbee0: 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 00 01 f9 f9
  0x0000800bbef0: f9 f9 f9 f9 05 f9 f9 f9 f9 f9 f9 f9 04 f9 f9 f9
  0x0000800bbf00: f9 f9 f9 f9 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9
  0x0000800bbf10: 00 00 00 00 00 00 00 00 00 00 00 00 00 f9 f9 f9
=>0x0000800bbf20:[f9]f9 f9 f9 00 00 00 00 00 00 06 f9 f9 f9 f9 f9
  0x0000800bbf30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0000800bbf40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0000800bbf50: f9 f9 f9 f9 00 00 01 f9 f9 f9 f9 f9 00 00 00 00
  0x0000800bbf60: 00 00 00 00 f9 f9 f9 f9 00 00 00 00 06 f9 f9 f9
  0x0000800bbf70: f9 f9 f9 f9 00 00 00 00 05 f9 f9 f9 f9 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==18692==ABORTING

valeriuo added a commit to valeriuo/htslib that referenced this issue Jul 11, 2018
Fix buffer overflow in `sam_parse1`.
Fixes samtools#731.
38 added a commit to 38/htslib that referenced this issue Oct 8, 2018
htslib release 1.9:

* If `./configure` fails, `make` will stop working until either
  configure is re-run successfully, or `make distclean` is used.
  This makes configuration failures more obvious.  (samtools#711, thanks to
  John Marshall)

* The default SAM version has been changed to 1.6.  This is in line
  with the latest version specification and indicates that HTSlib
  supports the CG tag used to store long CIGAR data in BAM format.

* bgzip integrity check option '--test' (samtools#682, thanks to @sd4B75bJ,
  @jrayner)

* Faidx can now index fastq files as well as fasta.  The fastq
  index adds an extra column to the `.fai` index which gives the
  offset to the quality values.  New interfaces have been added
  to `htslib/faidx.h` to read the fastq index and retrieve the
  quality values.  It is possible to open a fastq index as if
  fasta (only sequences will be returned), but not the other way
  round. (samtools#701)

* New API interfaces to add or update integer, float and array aux
  tags. (samtools#694)

* Add `level=<number>` option to `hts_set_opt()` to allow the
  compression level to be set.  Setting `level=0` enables
  uncompressed output. (samtools#715)

* Improved bgzip error reporting.

* Better error reporting when CRAM reference files can't be opened.
  (samtools#706)

* Fixes to make tests work properly on Windows/MinGW - mainly to
  handle line ending differences. (samtools#716)

* Efficiency improvements:

  - Small speed-up for CRAM indexing.

  - Reduce the number of unnecessary wake-ups in the thread pool.
    (samtools#703)

  - Avoid some memory copies when writing data, notably for
    uncompressed BGZF output. (samtools#703)

* Bug fixes:

  - Fix multi-region iterator bugs on CRAM files. (samtools#684)

  - Fixed multi-region iterator bug that caused some reads to be
    skipped incorrectly when reading BAM files. (samtools#687)

  - Fixed synced_bcf_reader() bug when reading contigs multiple
    times. (samtools#691, reported by @freeseek)

  - Fixed bug where bcf_hdr_set_samples() did not update the sample
    dictionary when removing samples. (samtools#692, reported by @freeseek)

  - Fixed bug where the VCF record ref length was calculated
    incorrectly if an INFO END tag was present. (71b00a)

  - Fixed warnings found when compiling with gcc 8.1.0. (samtools#700)

  - sam_hdr_read() and sam_hdr_write() will now return an error code
    if passed a NULL file pointer, instead of crashing.

  - Fixed possible negative array look-up in sam_parse1() that
    somehow escaped previous fuzz testing. (samtools#731, reported by
    @fCorleone)

  - Fixed bug where cram range queries could incorrectly report an
    error when using multiple threads. (samtools#734, reported by Brent
    Pedersen)

  - Fixed very rare rANS normalisation bug that could cause an
    assertion failure when writing CRAM files.  (samtools#739, reported by
    @carsonhh)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant