libav does not decode correct sample block size

[flibitijibibo]

I finally figured out what's causing all that racket with WMA/XMA streams. This one's for @aeikum (wrote the original FFmpeg work), @0x0ade (XMA), @GloriousEggroll (Warframe), and @JohanSmet (wrote the most recent FAudio FFmpeg work).

A quick overview of how the decoding system works: When a buffer is submitted we receive not just the data itself, but the amount of data we can get (measured in samples). So when we decode, we ask for exactly the number of samples we need based on the quantum size until the end, where the number will be a little bit smaller. In either case, the decoder always provides the exact number of samples we request because we always know how many samples are available without the decoder's assistance.

FFmpeg isn't playing by the rules. In particular, we're hitting this line which should never ever happen:

https://github.com/FNA-XNA/FAudio/blob/master/src/FAudio_ffmpeg.c#L382

Here's the quick version:

• Mixer asks for, say, 512 samples
• We read in from FFmpeg, filling the convert cache as needed
• FFmpeg gives us less than the total sample block size, but also claims we're out of samples
• available is 0, breaks
• todo is less than samples, resulting in uninitialized memory at the end
• R̫̘̜̕E̼͈̭̬̥A҉͔͙̪̞̘͈L̰ ̕G̜̳̱̯̱͈̰O̼̯͓̻͓͚O̢̼̥̮͍̝D̶ ͔̰̜͚͔S̭̗͚͈͞O͇̺̻̹̲͍U̝̦͕̺̳͍̺Ṋ̼̹̼̤D̫͈͉̹S̗̳̭͔̝̭̕

Eggroll has it in Warframe, I have it in Skyrim SE. We need to figure out how to make FFmpeg cooperate and actually give us all the samples. We can't leave without the full size, because then you just get skipping, which is more pleasant-sounding but still very wrong.

Anyone have any ideas? I'm not terribly familiar with FFmpeg so I don't know if there's a trick to making decoding consistent. Sadly it's not 100% consistent, but it's very easy to reproduce with those two games.

[flibitijibibo]

Couple quick changes to zero out the uninitialized memory (for sanity) and clean up some of the sample-vs-frame math that was going on:

1afcf01...35c55e4

This somewhat unintentionally fixes one case i found where the offsets were only off by 1... for a stereo stream. Not sure how that happened, but hopefully it won't happen again now that this is in.

[flibitijibibo]

From Johan in the Discord...

I did some more debugging and I've found that the FFmpeg WMA decoder does not seem to be sample accurate. Sometimes it produces more and sometimes less samples per packet than specified in the seek table. And because we're using the seek table to calculate the expected number of samples in the buffer, this causes us to think we're missing samples at the end (or to not play the entire buffer). Looping/seeking is also going to be inaccurate but that might not be as noticable.

So that... kind of explains why it's not 100% consistent each time? In any case, this is a pretty big problem; we've fixed our end of it but if we want this to work properly it'll have to be fixed in FFmpeg.

[flibitijibibo]

Possible idea to explore: Take our existing WMA test program and add a DumpXAPO that's attached to the source; the idea is what we can run that XAPO against both XAudio2 and FAudio and compare the samples decoded (ideally not resampled) by both libraries. From there we can use XAudio2's output as a control and compare various attempts to fix this in FAudio and FFmpeg.

[flibitijibibo]

I believe this is the same issue...? https://trac.ffmpeg.org/ticket/7473

[SveSop]

@flibitijibibo
I built some of the samples from the Microsoft SDK here: https://github.com/walbourn/directx-sdk-samples/tree/master/XAudio2 and after some tweaks i got them to work with wine.

I also built libfaudio from latest git, and tried compiling with ffmpeg aswell (Ubuntu 18.04 libavcodec-dev 7:3.4.4-0ubuntu0.18.04.1), but are not able to get any sounds during the WMA/XWMA part of the XAudio2BasicSound.exe demo.

0009:trace:xaudio2:IXAudio2Impl_Release (0x114848)->(): Refcount now 1
Playing mono WAV PCM file...
Playing mono WAV ADPCM file (loops twice)...
Playing 5.1 WAV PCM file...
Playing mono xWMA file...
Playing 5.1 xWMA file...
Finished playing
0009:trace:xaudio2:DllMain (0xf7ad0000, 0, 0x1)

Is it my ffmpeg "too old", or is there a issue?

You can download the binaries i tested with here: https://github.com/SveSop/wine_patches/blob/master/xaudio.tar.xz

Build log of libfaudio w/ffmpeg: https://launchpad.net/cybermax-dexter/+archive/ubuntu/sdl2-backport/+build/16519929/+files/buildlog_ubuntu-bionic-amd64.faudio_19.03-bionic5ppa2_BUILDING.txt.gz

[flibitijibibo]

If it’s older than 4.0 it’s too old.

[SveSop]

@flibitijibibo
Updated my PPA to use ffmpeg 4.1.1, and xWMA mono sample for XAudio2BasicSound.exe demo plays fine!
However, xWMA 5.1 bugs out with:
[wmapro @ 0x7dcb6600] Reserved bit is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.

[wmapro @ 0x7dcb6600] Reserved bit is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[wmapro @ 0x7dcb6600] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[wmapro @ 0x7dcb6600] num_vec_coeffs 3800 is too large
[wmapro @ 0x7dcb6600] num_vec_coeffs 2792 is too large
[wmapro @ 0x7dcb6600] Channel transform bit is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[wmapro @ 0x7dcb6600] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)

If you want whole log, let me know.

[flibitijibibo]

I think the error message speaks for itself...

[SveSop]

@flibitijibibo
Not sure what you mean by "speaks for itself" other than "Doh.. just upgrade ffmpeg you tool".
This is the actual wav file the xaudio2 sample plays: https://github.com/walbourn/directx-sdk-samples/blob/master/Media/Wavs/musicsurround_xwma.wav

Things i have tested:
Ubuntu 18.04 w/ffmpeg-4.1.2 play the file fine with ffplay musicsurround_xwma.wav
Windows 10 w/ffmpeg-4.1.1 play the file fine with ffplay musicsurround_xwma.wav
wine-devel(GIT c7f3231) w/ffmpeg-4.1.1 play the file fine with ffplay musicsurround_xwma.wav

Compiled latest faudio w/ -DFFMPEG=ON and ffmpeg-4.1.2 (ppa here: https://launchpad.net/~cybermax-dexter/+archive/ubuntu/sdl2-backport/+packages )
XAudio2 demo mentioned earlier fails with the above error.

Ofc i could have made the packages totally wrong, but since ffmpeg (ffplay) plays the file just fine when used for playing, i would kind of believe ffmpeg library is able to handle this? Or am i totally mistaken?

I admit i am a total newb, and might just aswell have made (several) errors on my package setup. Tips are welcome :)

PS. Or should i post this as a separate issue if you deem it warrants a "issue" to work on?

[flibitijibibo]

I'll translate it:

Reserved bit is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented. If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)

Channel transform bit is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented. If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)

TL;DR: This is FFmpeg's issue, not ours.

[SveSop]

ffplay -loglevel 48 musicsurround_xwma.wav does not show any error with reserved bit, nor channel transform, so it must be some weirdness that XAudio2 does i guess.

Could it be a wine bug?

[flibitijibibo]

No. It's either an FFmpeg issue (which I just said...) or the WAVEFORMATEXTENSIBLE data needs to be looked at. You would need to look at what the samples generate for those files.

[SveSop]

[NULL @ 0x7f3f78000b80] Opening 'musicsurround_xwma.wav' for reading
[file @ 0x7f3f78001740] Setting default whitelist 'file,crypto'
Probing mp3 score:1 size:2048
Probing xwma score:100 size:2048
[xwma @ 0x7f3f78000b80] Format xwma probed with size=2048 and score=100
[xwma @ 0x7f3f78000b80] Before avformat_find_stream_info() pos: 184 bytes read:32768 seeks:0 nb_streams:1
[wmapro @ 0x7f3f78002640] extradata:
[wmapro @ 0x7f3f78002640] [10] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [e0] [0] [0] [0] 
[xwma @ 0x7f3f78000b80] stream 0: start_time: -192153584101141.156 duration: 8.759
[xwma @ 0x7f3f78000b80] format: start_time: -9223372036854.775 duration: 8.759 bitrate=134 kb/s
[xwma @ 0x7f3f78000b80] After avformat_find_stream_info() pos: 147658 bytes read:147658 seeks:0 frames:27
Input #0, xwma, from 'musicsurround_xwma.wav':
  Duration: 00:00:08.76, bitrate: 134 kb/s
    Stream #0:0, 27, 1/48000: Audio: wmapro (b[1][0][0] / 0x0162), 48000 Hz, 6 channels, fltp, 48 kb/s
[wmapro @ 0x7f3f78003840] extradata:
[wmapro @ 0x7f3f78003840] [10] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [e0] [0] [0] [0]

From the xwma.c file:

 /* xWMA shouldn't have extradata. But the WMA codecs require it,
         * so we provide our own fake extradata.
         *
         * First, check that there really was no extradata in the header. If
         * there was, then try to use it, after asking the user to provide a
         * sample of this unusual file.
         */

FFmpeg decodes and plays it, but something else might happen when "its coming from xaudio2".

[SveSop]

@flibitijibibo
The error is caused by 5.1 surround sound XWM(a) files. I have converted a couple of different WAV files to XWMa format with Microsoft xWMAEncode.exe, and it seems as when there is a 5.1 surround sound XWMa file, this bug happens.

The XWMa file plays just fine with ffplay.

2 channel or mono XWMa files also play just fine, so something strange must happen when Xaudio2 sends this to libfaudio i think.

I am not really sure how i should make a bugreport of this to FFmpeg team, since it is working fine tho. Do you want to experiment with the files? Attaching a .gz of the original .wav and the converted _xwma.wav.
Demo_wavs.zip

[flibitijibibo]

This is the spot you care about:

https://github.com/FNA-XNA/FAudio/blob/master/src/FAudio_ffmpeg.c#L116

[SveSop]

I have been toying around with this, and all i can really come up with is that when playing with ffplay, "some" of this "waveformatextensible" data is just ignored/cleared, but when FAudio copies this over for ffmpeg to play, there is some data that is "not supported" (ref. error msg).
I gather these lines copies the waveformatextensible data blob over to ffmpeg?
https://github.com/FNA-XNA/FAudio/blob/master/src/FAudio_ffmpeg.c#L138-L142

I have no clue what data this actually contains, or what bits i could/should clear out for ffmpeg to not throw this error.

Commenting out the FAudio_memcpy lines, and just copy some "fake" data with av_ctx->extradata throw a different error and do not crash.
[wmapro @ 0x7dcaed80] bits per sample is 0 is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.

I dont know what av_ctx->extradata[?] actually would contain the settings for "bits per sample" tho.

[flibitijibibo]

Yeah, that memcpy should cover everything past the base WAVEFORMATEX structure. But that 'bits per sample' message is really strange, it implies the format doesn't provide a bit depth...? I would dump out the entire WAVEFORMATEXTENSIBLE struct and see what we're being sent in XAudio2 vs what the real header of the file says. Maybe we do have to deliberately throw out some stuff, not sure.

[SveSop]

I dont have any great ideas, and i have tried to fiddle a bit with it, but i come up short :)

I wonder tho, is this some sort of conversion? Or does it not come into action when FFmpeg is used?
https://github.com/FNA-XNA/FAudio/blob/master/src/FAudio.c#L419-L422

I only ask, cos it seems as the only usable xWMA files i am able to produce using "wmapro" is multichannel audio, and if some sort of conversion would happen, perhaps there is some added bits from that causing this? (Regular pcm_s16le encoded wav's in 5.1 seems to work fine tho).

[SveSop]

I simplified the XAudio2BasicSound.exe file, so that it only loads the wmapro 5.1 wav "musicsurround_xwma.wav", so you can test it if you like?
XAudio2BasicSound.zip

If you run it in Win10 it requires DirectXRedistJune2010 package (For XAudio2), and msvc140 (c++ redist).

[flibitijibibo]

At the moment I can only test patches to FAudio directly; because of the licensing issues blocking us this is pretty low on the TODO list since there is other stuff I need to get working that doesn't have a patent blocking it.

For reference, here's our xwma test program: https://github.com/FNA-XNA/FAudio/tree/master/utils/testxwma

[SveSop]

@flibitijibibo Oh, i was not aware of the testxwma "player" :) Nice!
Compiled and used ./testxwma musicsurround_xwma.wav and that caused the same error as XAudio2BasicSound.exe does with "reserved bit" and "channel transform bit".

Atleast another testproggy to fiddle with in my spare time. Thx :)

[flibitijibibo]

With the latest revision the decoding responsibility has been moved to GStreamer. We still have an issue when moving from one WMA buffer to the next, but this was present in the previous decoder as well.

[Hello71]

At the moment I can only test patches to FAudio directly; because of the licensing issues blocking us this is pretty low on the TODO list since there is other stuff I need to get working that doesn't have a patent blocking it.

I don't understand, shouldn't WMA1/WMA2 US patents be expired by now if the formats were released in 1999?

[flibitijibibo]

The patent portfolio that covers the various formats and their revisions over the years is several dozen pages long with countless companies with various expiration dates. If there was a straightforward date, it probably would have gotten its own website like S3TC did.

[flibitijibibo]

Doing a bit of spring cleaning...

Closing as NOTOURBUG - WMA issues with libav should be filed with that project; at this point my only real data is going to come from protonmediaconverter so this is effectively abandoned.