Permalink
Browse files

mlpenc: apply_filter uses sample bit-depth

Hardcoding 24 was leading to lossless check failures when the residual
was overflowing the sample wordsize.

Signed-off-by: Jai Luthra <me@jailuthra.in>
  • Loading branch information...
jailuthra committed Jun 4, 2016
1 parent 2e2ff68 commit 8d00e456b579a323968723f3d0b908b1081752b6
Showing with 4 additions and 4 deletions.
  1. +4 −4 libavcodec/mlpenc.c
View
@@ -1869,14 +1869,14 @@ static void determine_bits(MLPEncodeContext *ctx)
*************** Functions that process the data in some way ****************
****************************************************************************/
-#define INT24_MAX ((1 << 23) - 1)
-#define INT24_MIN (~INT24_MAX)
+#define SAMPLE_MAX(bitdepth) ((1 << (bitdepth)) - 1)
+#define SAMPLE_MIN(bitdepth) (~SAMPLE_MAX(bitdepth))
#define MSB_MASK(bits) (-1u << bits)
/** Applies the filter to the current samples, and saves the residual back
* into the samples buffer. If the filter is too bad and overflows the
- * maximum amount of bits allowed (24), the samples buffer is left as is and
+ * maximum amount of bits allowed (16 or 24), the samples buffer is left as is and
* the function returns -1.
*/
static int apply_filter(MLPEncodeContext *ctx, unsigned int channel)
@@ -1919,7 +1919,7 @@ static int apply_filter(MLPEncodeContext *ctx, unsigned int channel)
accum >>= filter_shift;
residual = sample - (accum & mask);
- if (residual < INT24_MIN || residual > INT24_MAX)
+ if (residual < SAMPLE_MIN(ctx->wordlength) || residual > SAMPLE_MAX(ctx->wordlength))
return -1;
filter_state_buffer[FIR][i] = sample;

0 comments on commit 8d00e45

Please sign in to comment.