Skip to content

Commit

Permalink
avcodec/x86/mpegvideodsp: Fix signedness bug in need_emu
Browse files Browse the repository at this point in the history
Fixes: out of array read
Fixes: 3516/attachment-311488.dat

Found-by: Insu Yun, Georgia Tech.
Tested-by: wuninsu@gmail.com
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 58cf31c)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
  • Loading branch information
michaelni committed Nov 16, 2017
1 parent 8b46a95 commit 27f8d38
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions libavcodec/x86/mpegvideodsp.c
Expand Up @@ -53,8 +53,9 @@ static void gmc_mmx(uint8_t *dst, uint8_t *src,
const int dyh = (dyy - (1 << (16 + shift))) * (h - 1);
const int dxh = dxy * (h - 1);
const int dyw = dyx * (w - 1);
int need_emu = (unsigned) ix >= width - w ||
(unsigned) iy >= height - h;
int need_emu = (unsigned) ix >= width - w || width < w ||
(unsigned) iy >= height - h || height< h
;

if ( // non-constant fullpel offset (3% of blocks)
((ox ^ (ox + dxw)) | (ox ^ (ox + dxh)) | (ox ^ (ox + dxw + dxh)) |
Expand Down

0 comments on commit 27f8d38

Please sign in to comment.