stb_vorbis: fix slow loads and leaks in start_decoder.

[AliceLR]

This patch from libxmp fixes the following issues:

• Error checks in start_decoder when loading the lengths could return without freeing the lengths array, causing leaks. OGG_leaks.zip
• Missing EOF checks in start_decoder could contribute to very slow loads with corrupt/invalid files. This includes one check that originally checked for an EOP return value from get_bits—as far as I can tell, get_bits never actually returns this value. I don't remember exactly which of these missing checks contributed the most to slow loads, but IIRC I added most of them after reading values that get iterated on by loops. OGG_slow_loads.zip

[nothings]

This looks reasonable, but I wonder if we should instead do some systematic thing for the setup_temp_frees where they're all tracked and vorbis_deinit() cleans them up if they didn't get deallocated explicitly. I.e. make 'setup temp free' explicitly zero the pointer it's handed, and then they're all explicitly stored off of f-> or c->. If you're using f->alloc.alloc_buffer we do nothing with temp mem during deinit(), which means deinit() doesn't have to worry about freeing the allocations in stack order.

[AliceLR]

I pushed an update that implements (roughly) what you described. Since the buffers are never allocated for more than one codebook at a time I put them all in f->.

[nothings]

Looks good to me with github diff, I'll need to review it more carefully in the full file.

[AliceLR]

Ran a brief libFuzzer+ASan run on this patch and it looks like there are still issues to work out—will update.

edit: specifically the issue is that storing the temporary codewords allocation to both c->codewords and f->temp_codewords predictably causes issues in vorbis_deinit.

[nothings]

Do you have more tests you can run, or is that it?

[AliceLR]

I don't have any other tests that would be relevant to finding leaks/double frees, nor do I have a particularly good variety of invalid files to test against. No other issues related to this patch have showed up yet in my limited fuzzing.

[AliceLR]

My tester found some new slow load cases that might be due to another missed EOF check here, will follow up again: OGG_slow_loads2.zip

No further crashes related to this patch, though it looks like there may be an unrelated buffer overflow in codebook_decode_deinterleave_repeat.

[AliceLR]

It looks like the path the slow loads follow is valid: when ordered is true, the codebook can contain an absurd number of entries without needing to read most of them. There is an EOF check for this, it just doesn't read enough data to matter. (The slow portion of the load is calling qsort on 16m entries.)

[AliceLR]

I've updated this patch to remove (void **)&pointer usage, which flags warnings in old GCC versions (cited version was 4.2) as well as newer versions with -fstrict-aliasing -Wstrict-aliasing=2 (tested with GCC 11.2). (Found and patched by @sezero.)