overflow of hevc.cpp in function updateBits

[NigelX]

Hi
I found an crash error.

System info：
Ubuntu 20.04 : clang 10.0.0 , gcc 9.3.0

tsMuxeR version git-5a9adef

---

Verification steps：
1.Get the source code of tsMuxer
2.Compile the tsMuxer

$ cd tsMuxer
$ mkdir build && cd build
$ cmake ../ -DCMAKE_C_COMPILER=clang  -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_FLAGS="-fsanitize=address" -DCMAKE_CXX_FLAGS="-fsanitize=address"
$ make -j 32

3.run poc

$ cd tsMuxer
$ ./tsmuxer poc

file:
poc.zip

info:

Program received signal SIGSEGV, Segmentation fault.
[----------------------------------registers-----------------------------------]
RAX: 0x41 ('A')
RBX: 0x1 
RCX: 0x666eb0 --> 0x0 
RDX: 0x20 (' ')
RSI: 0x5bffff --> 0x0 
RDI: 0x7 
RBP: 0xc77060 --> 0x2318a0 --> 0x2d5520 (<HevcSpsUnit::deserialize()>:	push   rbp)
RSP: 0x7fffffff7690 --> 0x0 
RIP: 0x2d36e1 (<HevcUnit::updateBits(int, int, int)+321>:	movzx  ebp,BYTE PTR [r8])
R8 : 0xd2faff 
R9 : 0xc77b00 --> 0x4d5f56006943 ('Ci')
R10: 0xf4240 
R11: 0x5c001f --> 0x0 
R12: 0x7ffff3814064 --> 0x4d5f56006943 ('Ci')
R13: 0x7fffffff83e8 --> 0x231760 --> 0x2dd1e0 (<HEVCStreamReader::writeAdditionData(unsigned char*, unsigned char*, AVPacket&, std::vector<std::pair<int, int>, std::allocator<std::pair<int, int> > >*)>:	push   r15)
R14: 0x5c001f --> 0x0 
R15: 0x7ffff3814098 --> 0xffffff0a1f010000
EFLAGS: 0x10202 (carry parity adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x2d36d5 <HevcUnit::updateBits(int, int, int)+309>:	add    bl,0x1
   0x2d36d8 <HevcUnit::updateBits(int, int, int)+312>:	adc    bl,0x0
   0x2d36db <HevcUnit::updateBits(int, int, int)+315>:	mov    BYTE PTR [rcx+0x1840],bl
=> 0x2d36e1 <HevcUnit::updateBits(int, int, int)+321>:	movzx  ebp,BYTE PTR [r8]
   0x2d36e5 <HevcUnit::updateBits(int, int, int)+325>:	mov    cl,0x8
   0x2d36e7 <HevcUnit::updateBits(int, int, int)+327>:	sub    cl,dil
   0x2d36ea <HevcUnit::updateBits(int, int, int)+330>:	shr    ebp,cl
   0x2d36ec <HevcUnit::updateBits(int, int, int)+332>:	mov    ecx,edi
[------------------------------------stack-------------------------------------]
0000| 0x7fffffff7690 --> 0x0 
0008| 0x7fffffff7698 --> 0x7ffff3814064 --> 0x4d5f56006943 ('Ci')
0016| 0x7fffffff76a0 --> 0xc77060 --> 0x2318a0 --> 0x2d5520 (<HevcSpsUnit::deserialize()>:	push   rbp)
0024| 0x7fffffff76a8 --> 0x2de818 (<HEVCStreamReader::updateStreamFps(void*, unsigned char*, unsigned char*, int)+136>:	mov    ecx,DWORD PTR [rbp+0x94])
0032| 0x7fffffff76b0 --> 0x7ffff7da3d60 --> 0x7ffff7d9cac8 --> 0x7ffff7ccf650 (<_ZN9__gnu_cxx18stdio_sync_filebufIcSt11char_traitsIcEED1Ev>:	endbr64)
0040| 0x7fffffff76b8 --> 0x7ffff78fa7b2 (<_IO_new_file_xsputn+98>:	add    r12,r13)
0048| 0x7fffffff76c0 --> 0x0 
0056| 0x7fffffff76c8 --> 0x6 
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSEGV
0x00000000002d36e1 in HevcUnit::updateBits (this=<optimized out>, bitOffset=0x5bffff, bitLen=0x20, value=0xf4240)
    at /home/hh/Downloads/tsMuxer/tsMuxer/hevc.cpp:75
75	        int prefix = *ptr >> (8 - byteOffset);


asan:
Warning: AFL++ tools will need to set AFL_MAP_SIZE to 102528 to be able to run this instrumented program!
tsMuxeR version git-f6ab2a2. github.com/justdan96/tsMuxer
This HEVC stream doesn't contain fps value. Muxing fps is absent too. Set muxing FPS to default 25.0 value.
HEVC manual defined fps doesn't equal to stream fps. Change HEVC fps from 0.000119055 to 25
AddressSanitizer:DEADLYSIGNAL
=================================================================
==246382==ERROR: AddressSanitizer: SEGV on unknown address 0x6060000bc33f (pc 0x00000051c224 bp 0x7fff7926af10 sp 0x7fff7926a980 T0)
==246382==The signal is caused by a READ memory access.
    #0 0x51c224 in HevcUnit::updateBits(int, int, int) /home/hh/Downloads/tsMuxer/tsMuxer/hevc.cpp:75:22
    #1 0x51f4b9 in HevcVpsUnit::setFPS(double) /home/hh/Downloads/tsMuxer/tsMuxer/hevc.cpp:246:5
    #2 0x52d87f in HEVCStreamReader::updateStreamFps(void*, unsigned char*, unsigned char*, int) /home/hh/Downloads/tsMuxer/tsMuxer/hevcStreamReader.cpp:364:10
    #3 0x73daa6 in MPEGStreamReader::updateFPS(void*, unsigned char*, unsigned char*, int) /home/hh/Downloads/tsMuxer/tsMuxer/mpegStreamReader.cpp:310:9
    #4 0x533175 in HEVCStreamReader::checkStream(unsigned char*, int) /home/hh/Downloads/tsMuxer/tsMuxer/hevcStreamReader.cpp:77:13
    #5 0x6dea5a in METADemuxer::detectTrackReader(unsigned char*, int, AbstractStreamReader::ContainerType, int, int) /home/hh/Downloads/tsMuxer/tsMuxer/metaDemuxer.cpp:770:21
    #6 0x6d408a in METADemuxer::DetectStreamReader(BufferedReaderManager&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) /home/hh/Downloads/tsMuxer/tsMuxer/metaDemuxer.cpp:684:35
    #7 0x5e83b1 in detectStreamReader(char const*, MPLSParser*, bool) /home/hh/Downloads/tsMuxer/tsMuxer/main.cpp:120:34
    #8 0x5f8ca4 in main /home/hh/Downloads/tsMuxer/tsMuxer/main.cpp:698:17
    #9 0x7ff4004a00b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16
    #10 0x2e9b7d in _start (/home/hh/Downloads/tsMuxer/asan_build/tsMuxer/tsmuxer+0x2e9b7d)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/hh/Downloads/tsMuxer/tsMuxer/hevc.cpp:75:22 in HevcUnit::updateBits(int, int, int)
==246382==ABORTING

Thanks