XACT crashes with a SharpDX Exception when ADPCM compression is enabled

[SirDragonClaw]

When using XACT with monogame I expected to not be able to use xWMA (sadly) and I can live with the massive files of ADPCM for the most part with disk streaming. But when I change from PCM to ADPCM my game crashes when trying to play a Cue. With the following exception.

{SharpDX.SharpDXException: HRESULT: [0x88960001], Module: [SharpDX.XAudio2], ApiCode: [XAUDIO2_E_INVALID_CALL/InvalidCall], Message: Unknown
   at SharpDX.XAudio2.XAudio2.CreateSourceVoice_(SourceVoice sourceVoiceOut, IntPtr sourceFormatRef, VoiceFlags flags, Single maxFrequencyRatio, IntPtr callbackRef, Nullable`1 sendListRef, Nullable`1 effectChainRef)
   at SharpDX.XAudio2.SourceVoice.CreateSourceVoice(XAudio2 device, WaveFormat sourceFormat, VoiceFlags flags, Single maxFrequencyRatio, IntPtr callback, EffectDescriptor[] effectDescriptors)
   at SharpDX.XAudio2.SourceVoice..ctor(XAudio2 device, WaveFormat sourceFormat, VoiceFlags flags, Single maxFrequencyRatio)
   at Microsoft.Xna.Framework.Audio.SoundEffect.PlatformSetupInstance(SoundEffectInstance inst)}

My platform is UWP, running on the latest windows 10, fast ring.

I would expect this to be supported, and unlike xWMA it doesn't throw a not supported exception. So I assume it is either a bug or I am doing something wrong somehow.

[tomspilman]

Yeah this should work... in particular on Windows.

[SeriousMartin]

I ran into the same problem, updating to build 1041 from 362.

Might have something to do with the state of the device.

In addition I wonder whether this has something to do with the SamplesPerBlock value that is not between 32 and 512 as stated here: https://msdn.microsoft.com/en-us/library/microsoft.directx_sdk.xaudio2.adpcmwaveformat(v=vs.85).aspx

Here is a an example of a values that go into the SourceVoice constructor when the crash happens

Any suggestions on this one?

[nkast]

@SeriousMartin

In addition I wonder whether this has something to do with the SamplesPerBlock value that is not
between 32 and 512

Indeed, the ADPCM generated by ffmpeg are unuseable by XAudio.
I posted an enhancement request about that here https://trac.ffmpeg.org/ticket/6585
It will probably take years until someone look at it. Meanwhile we need an alternative.

[SeriousMartin]

@nkast Do you know how this worked before? The used ffmpeg version hasn't changed for years. Did microsoft change the xaudio2 adpcm requirement?

Looks like AdpcmEncode is a suitable commandline tool to do the job: https://msdn.microsoft.com/de-de/library/windows/desktop/ee415711(v=vs.85).aspx.

This means for the purpose of Adpcm encoding it would be necessary to let the processor decode to wave first and then use AdpcmEncode.

[KonajuGames]

FFMPEG is unfortunately hard-coded to that sample block size. I wrote MS-ADPCM and IMA/ADPCM decompression for platforms that don't support one or either of those formats. It may probably be more reliable to write our own compressors for those formats as well. The AdpcmEncode tool is only in the DirectX SDK on Windows.

[tomspilman]

I am confused here.

We have been using FFMPEG to convert PCM to MSADPCM since 2014. Any games built in that time using compressed audio have used MSADPCM audio compressed by FFMPEG.

So how is it that suddenly FFMPEG doesn't work for MSADPCM used by XAudio2?

What has changed?

[KonajuGames]

Previously, our XACT support always decompressed ADPCM at load. A quick fix to this for now would be to always make it decompress ADPCM on DirectX platforms if the SampleBlockSize is greater than 512. Then we do our own MS-ADPCM compression and re-enable it.

When I did that last major update to audio, I extensively tested every format on both DirectX and OpenAL. Running it now, it fails with this SharpDX exception straight up. FFMPEG's hard-coded sample block size (during that audio update I even checked FFMPEG's source code to see if the sample block size could be specified) is outside the range specified in the MS documentation for XAudio2.

I also realised that while we import, process and load a range of audio formats in our tests, we don't actually play any of them. This would have shown the error sooner. I can add that too with the changes mentioned above.

[tomspilman]

Previously, our XACT support always decompressed ADPCM at load.

What is the time period on this "previously"? We have been using compressed MSADPCM with XAct for years now.

But actually... the original issue has nothing to do with FFMEPG now that i think of it. @Danthekilla was using XACT and MonoGame does not process XACT audio in any way. We use XACT content built from the official Microsoft XACT tool which does not use FFMPEG and hasn't changed for over 6 years now.

@SeriousMartin - Was your issue with a sound from XACT? Or was it from a sound assets generated by the MG content pipeline?

make it decompress ADPCM on DirectX platforms if the SampleBlockSize is greater than 512. Then we do our own MS-ADPCM compression and re-enable it.

That could add minutes of startup time to some projects... that really isn't a good runtime solution for us.

When I did that last major update to audio,

That was merged only 2 months ago #5750 .... so why does it fail now and not 2 months ago?

[KonajuGames]

Come to think of it, I agree that it can't be our content pipeline because we don't produce XACT wavebanks and the SoundEffect data for XACT are embedded in the wavebank. So these errors can only come from trying to load a SoundEffect that has been compressed by our content pipeline via ffmpeg.

I'm not sure where my own testing missed this. I think maybe because I was using XNA-produced XNB files at first for ADPCM testing, then swapped in files produced by our content pipeline later in the process. I think it wasn't noticed at first because no-one was using ADPCM with our content pipeline back then due to problems with generating/using the compressed audio.

I'm not too keen on decompressing at load either, so I'm happy to not go down that path, even as an interim measure. I have the reference material to write our own MS-ADPCM and IMA/ADPCM compressors, and I'll add SoundEffect playback tests to our test suite. This allows us control over it (the samples per block and other parameters) and removes a native dependency (that is difficult/cumbersome to modify and rebuild) for this part.

[tomspilman]

I have the reference material to write our own MS-ADPCM and IMA/ADPCM compressors,

I guess we have little choice there. Sounds like the best solution for now.

I'll add SoundEffect playback tests to our test suite

We already play a few sounds now.. I hear them daily in the server room. Just need to add compressed ones too.

[KonajuGames]

We already play a few sounds now.

They are the tests for DynamicSoundEffectInstance. We don't currently play any of the wide variety of SoundEffects we currently test by importing, processing and loading.

[SeriousMartin]

Was your issue with a sound from XACT? Or was it from a sound assets generated by the MG content pipeline?

It's a sound effect build by the MG pipeline. I'm using quality "medium" which automatically converts it to ADPCM (as far as i know). This used to work without any problems in MG 3.7.0.362. I still don't get what has changed since then. Neither ffmpeg nor xaudio...

[nkast]

@KonajuGames

I think we can use the data from FFMPEG and inject our own starting Block-preamble every 32 samples.
Not as accurate as a real encoder but will do the trick.
We can use the MSADPCMToPCM to decode the stream and at the right time inject a new starting block with the current values of predictor , delta , sample_1 ,sample_2.

MonoGame/MonoGame.Framework/Audio/MSADPCMToPCM.cs

Lines 133 to 137 in cdd28ff

	// Read block preamble
	byte predictor = Source.ReadByte();
	short delta = Source.ReadInt16();
	short sample_1 = Source.ReadInt16();
	short sample_2 = Source.ReadInt16();

Of course if you are close with the encoder there no need for this hack.

EDIT
Well, this wont work because the preamble is part of the block.
A 1024 block contain 2036 samples but two 512 blocks contain 2024 samples.

[tomspilman]

@KonajuGames @dellis1972 @nkast

I see three paths here for a long term fix:

1. Fix and rebuild FFMPEG to our needs.
2. Find another cross platform MSADPCM encoder.
3. Write our own cross platform MSADPCM encoder.

As for a short term fix... we could use AdpcmEncode.exe which is included in all the Windows SDKs and is the official MSADPCM encoder. Of course the issue is it is Windows only... but Windows developers are easily over 90% of MG users.

[SeriousMartin]

I'm currently looking into this.

1. In addition to the trac ticket @nkast created (https://trac.ffmpeg.org/ticket/6585) I'm talking to the ffmpeg community hoping they will be able to provide a solution: https://lists.ffmpeg.org/pipermail/ffmpeg-user/2018-January/038411.html. There is also an basic xaudio2 test to check whether generated files are working.
2. I have had a try with SoX, but it's behaving almost like FFMPEG

I have also create a short term fix for DefaultAudioProfile.cs using AdpcmEncode.exe. I would be willing to provide a pull request if someone is interessted and willing to merge it.

One thing I stumbled upon was that AdpcmEncode is not able to encode wav files generated by ffmpeg using pcm_f32le as codec. AdpcmEncode says it's compatible with 32bit floating point. Any idea on what could cause this?

[SeriousMartin]

Took me a while to compile ffmpeg on my windows system. MSYS2 did the trick, but it some dependencies wont be included in the compiled exe. Still ok for testing.

I changed the BLKSIZE constant in ffmpeg's adpcm.h to 512 and it still didn't work with XAudio.

So I took a closer look at the code. Turns out that the BLKSIZE constant is actually used to set the block_align value. Microsoft measues the size of a block in samples (https://msdn.microsoft.com/de-de/library/windows/desktop/ee415711(v=vs.85).aspx). This seems to be different from the block_align used in adpcmenc.c. My impression is that frame_size is instead what has to be between 32 and 512.

I tested this by setting BLKSIZE to 70. With my mono wave file this resulted in a frame_size of 128 = (BLKSIZE - 7 * avctx->channels) * 2 / avctx->channels + 2.

The resulting compressed file is working with XAudio.

I am very unsure if this is the right thing to do.

[KonajuGames]

I've got a pure managed code MS-ADPCM encoding routine that creates files usable by XAudio2 (using 128 sample blocks = block alignment of 70 bytes for mono and 140 for stereo). Just got a slight bug in the encoding to work out that is causing a static-like noise in the output.

[SeriousMartin]

Until now there is basically no reaction from the ffmpeg community. I will have another try, but ain't very confident there will be a ffmpeg solution.

@KonajuGames Have you worked out the problems in your code? I would be happy to test it in my project.

@Others Any thoughts on this? Is a managed code solution ok?

[Halofreak1990]

Reading through this issue, it's not clear why this issue refers to, and is labeled as, XACT, whilst the actual problem is with SoundEffects using ADPCM compression.

[SeriousMartin]

@KonajuGames Any news on the bug you mentioned? As I mentioned, I would be happy to port and test your implementation to monogame. Any idea how this might affect content pipleline performance?

[tomspilman]

I think i may have figured this out.

Anyone have a moment to test this code change to SoundEffect.XAudio.cs:

private void PlatformInitializeXact(MiniFormatTag codec, byte[] buffer, int channels, int sampleRate, int blockAlignment, int loopStart, int loopLength, out TimeSpan duration)
{
  if (codec == MiniFormatTag.Adpcm)
  {
    duration = TimeSpan.FromSeconds((float)loopLength / sampleRate);

    CreateBuffers(  new WaveFormatAdpcm(sampleRate, channels, (blockAlignment + 22) * channels), // CHANGED!
      ToDataStream(0, buffer, buffer.Length),
      loopStart,
      loopLength);

    return;
  }

  throw new NotSupportedException("Unsupported sound format!");
}

Then try to play some XACT sounds that have been generated with ADPCM compression.

I suspect this fixes things.

@SeriousMartin @Danthekilla ?

[Sebanisu]

I added an alternative build of our opengl game to be able to render in directx. The audio worked in opengl. But in directx I get this problem. All of our raw audio is converted to pcm via ffmpeg. Has someone found a workaround for this?

Though maybe my error is just the same but is unrelated. I was using ffmpeg to decompress adpcm and ogg audio. And feed to dynamicsoundeffectinterface and soundeffect.

[Sebanisu]

In my project I added a Try Catch for this exception and I wrote some code to fall back to the nAudio library and use that to output to DirectSound. It is working till this issue is solved.

[tomcashman]

This issue is still occurring on latest MonoGame. I tested with the fix @tomspilman suggested but it didn't resolve the issue 😐

[mrhelmut]

I've run into it too while trying to debug the XACT implementation. I'll try to check what's going on. Knowing that it is linked to ADPCM is a start.

[squarebananas]

I've been using the above fix for a few months now without any issue. If you upload a sample sound effect and xact file I'll give it a try with my project.

[mrhelmut]

Out of curiosity, which version of XACT are you all using?
I'm assuming that the most common version is the one that shipped with XNA 4.0 Refresh, but I've seen unexpected versions popping which have a different file format (and notably produced unexpected ADPCM format, like broken sample rates).

You can check that by opening your .xap files with a text editor and check both the "format version" and "version" numbers.

[tomcashman]

I didn't realise XACT was separate tool (thought it was bundled part of MGCB). I was getting this exact crash when trying to use Medium compression on an .ogg sound effect file with MGCB when targeting UWP. Should I open a separate issue?

[squarebananas]

Yeah that seems to be a separate issue unrelated to XACT. If you add a breakpoint on the fix line for PlatformInitializeXact you should see it is never called for your sound effect. I can replicate your issue though with non XACT ogg sound effects with medium compression on WindowsDX and WindowsUniversal platforms.

Edit: It seems this github issue contains 2 different issues. So while your bug isn't related to the original XACT issue, it is related to the FFMPEG block size issue. Also it effects wav as well as ogg.

According to the FFMPEG bug tracker (https://trac.ffmpeg.org/ticket/6585) the issue is fixed. I have tried updating FFMPEG but I'm not sure what the parameter is to change the ADPCM block size. If anyone can work that out I'll test it out.

[squarebananas]

Unfortunately it seems the FFMPEG bug fix has been implemented incorrectly and valid samples per block can't be generated still.

Most likely the fix would have been to update FFMPEG and change the content pipeline DefaultAudioProfile.cs line from
ffmpegCodecName = "adpcm_ms";
to
ffmpegCodecName = "adpcm_ms -block_size 70";

However FFMPEG now forces the block size to be a power of 2 which appears to generate invalid samples per block. (A block size of 70 would be needed to generate a valid 128 samples per block). I guess this will need another ticket raising with them, not sure how that is done.

[tomcashman]

I've logged the bug with FFMPEG using the info from this thread.

https://trac.ffmpeg.org/ticket/9397#ticket

[mrhelmut]

I'm closing this one since #7564 got around it for the time being.

[vs49688]

Re-posting my reply (https://trac.ffmpeg.org/ticket/9397#comment:1) here:

I suspect the fix is a simple matter of relaxing the check.

Could you please comment out this block at libavcodec/adpcmenc.c:92 and see if the generated file works?

    if (avctx->codec->id != AV_CODEC_ID_ADPCM_IMA_AMV &&
        (s->block_size & (s->block_size - 1))) {
        av_log(avctx, AV_LOG_ERROR, "block size must be power of 2\n");
        return AVERROR(EINVAL);
    }

[squarebananas]

Could you please comment out this block at libavcodec/adpcmenc.c:92 and see if the generated file works?

I can confirm this change works.

I've just commented out the block, built FFmpeg with MSYS2, copied this for the content builder to use, built a sound effect with a block size of 70 and then successfully played this in a MonoGame project.

Just to make it clear while the XACT ADPCM issue is closed, this is an on-going issue for non-XACT ADPCM sounds. The on-going issue is actually #6701 but confusingly most discussion related to that non-XACT issue is in this closed XACT issue instead.