Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix a bug in alloc_size for high bit depths
I introduced this bug in commit 2e32276: https://chromium-review.googlesource.com/c/webm/libvpx/+/5446333 I changed the line stride_in_bytes = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s * 2 : s; to three lines: s = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s * 2 : s; if (s > INT_MAX) goto fail; stride_in_bytes = (int)s; But I didn't realize that `s` is used later in the calculation of alloc_size. As a quick fix, undo the effect of s * 2 for high bit depths after `s` has been assigned to stride_in_bytes. Bug: chromium:332382766 Change-Id: I53fbf405555645ab1d7254d31aadabe4f426be8c (cherry picked from commit 74c70af)
- Loading branch information