Skip to content

Commit

Permalink
Fix a bug in alloc_size for high bit depths
Browse files Browse the repository at this point in the history
I introduced this bug in commit 2e32276:
https://chromium-review.googlesource.com/c/webm/libvpx/+/5446333

I changed the line

  stride_in_bytes = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s * 2 : s;

to three lines:

  s = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s * 2 : s;
  if (s > INT_MAX) goto fail;
  stride_in_bytes = (int)s;

But I didn't realize that `s` is used later in the calculation of
alloc_size.

As a quick fix, undo the effect of s * 2 for high bit depths after `s`
has been assigned to stride_in_bytes.

Bug: chromium:332382766
Change-Id: I53fbf405555645ab1d7254d31aadabe4f426be8c
(cherry picked from commit 74c70af)
  • Loading branch information
wantehchang committed May 21, 2024
1 parent 5193ce7 commit 61c4d55
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions vpx/src/vpx_image.c
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ static vpx_image_t *img_alloc_helper(vpx_image_t *img, vpx_img_fmt_t fmt,
s = (s + stride_align - 1) & ~((uint64_t)stride_align - 1);
if (s > INT_MAX) goto fail;
stride_in_bytes = (int)s;
s = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s / 2 : s;

/* Allocate the new image */
if (!img) {
Expand Down

0 comments on commit 61c4d55

Please sign in to comment.