Possible undefined behavior in find_best_partition_order_()

[braheezy]

Hello!

While interacting with libFLAC using the Zig toolchain, I get an encoder crash when compiled with Zig safety rules turned on (i.e., zig build -Doptimize=ReleaseFast doesn't crash):

> zig build demo -Doptimize=Debug
...
Illegal instruction at address 0x11fd22c
flac/src/libFLAC/stream_encoder.c:4130:28: 0x11fd22c in find_best_partition_order_ (.../stream_encoder.c)
     raw_bits_per_partition+sum,
...

When this crash happens, raw_bits_per_partition is a null pointer and I'm assuming Zig considers pointer arithmetic with a null pointer an illegal instruction.

Through code inspection, raw_bits_per_partition only seems relevant when dealing with something called "escape coding" in FLAC files and it looks like escape code support was dropped in 3b5f471.

A Fix

I patched the C code with the following:

if(!
    set_partitioned_rice_(
#ifdef EXACT_RICE_BITS_CALCULATION
        residual,
#endif
        abs_residual_partition_sums+sum,
-        raw_bits_per_partition+sum,
+        do_escape_coding ? raw_bits_per_partition+sum : raw_bits_per_partition,
        residual_samples,
        predictor_order,
        rice_parameter_limit,
        rice_parameter_search_dist,
        (uint32_t)partition_order,
        do_escape_coding,
        &private_->partitioned_rice_contents_extra[!best_parameters_index],
        &residual_bits
    )
)

And now if I run my demo program, it produces a valid file:

> zig build demo -Doptimize=Debug
bad samples: 0
Decoded FLAC: Channels: 2, Sample Rate: 48000, Bits per Sample: 16, Samples: 576000
Re-encoded FLAC successfully.

I can provide more context if needed, but I think this is mostly a case of a stricter C compiler throwing an error on valid C code where other compilers (gcc) let the undefined behavior go.

[ktmf01]

Thanks. I was under the impression bugs like this would be caught by oss-fuzz's undefined behaviour fuzzing, but apparently -fsanitize=pointer-overflow isn't enabled.