assertion failure in GetInvInt() for 96kHz input + AACENC_BITRATEMODE: 5

[nu774]

As the title says, when trying to encode 96kHz input with AACENC_BITRATEMODE: 5 (VBR), fdk-aac crashes with an assertion failure.
Stack trace is as follows:

GetInvInt(int intValue=54)  Line 459 + 0x27 bytes
FDKaacEnc_prepareIntensityDecision(const long * sfbEnergyLeft=0x00dcfa1c, const long * sfbEnergyRight=0x00dd0750, const long * sfbEnergyLdDataLeft=0x00dbee58, const long * sfbEnergyLdDataRight=0x00dc13a8, const long * mdctSpectrumLeft=0x00dbcdb8, const long * mdctSpectrumRight=0x00dbf308, const INTENSITY_PARAMETERS * isParams=0x0038ddb8, long * hrrErr=0x0038dfcc, int * isMask=0x0038dcc0, long * realScale=0x0038dddc, long * normSfbLoudness=0x0038ded4, const int sfbCnt=48, const int sfbPerGroup=12, const int maxSfbPerGroup=12, const int * sfbOffset=0x00dd0470)  Line 331 + 0x21 bytes
FDKaacEnc_IntensityStereoProcessing(long * sfbEnergyLeft=0x00dcfa1c, long * sfbEnergyRight=0x00dd0750, long * mdctSpectrumLeft=0x00dbcdb8, long * mdctSpectrumRight=0x00dbf308, long * sfbThresholdLeft=0x00dcf83c, long * sfbThresholdRight=0x00dd0570, long * sfbThresholdLdDataRight=0x00dc11c8, long * sfbSpreadEnLeft=0x00dd028c, long * sfbSpreadEnRight=0x00dd0fc0, long * sfbEnergyLdDataLeft=0x00dbee58, long * sfbEnergyLdDataRight=0x00dc13a8, int * msDigest=0x00de82dc, int * msMask=0x00de82e0, const int sfbCnt=48, const int sfbPerGroup=12, const int maxSfbPerGroup=12, const int * sfbOffset=0x00dd0470, const int allowIS=1, int * isBook=0x00de4d68, int * isScale=0x00de4e58, PNS_DATA * * pnsData=0x0038ec74)  Line 629 + 0x50 bytes
FDKaacEnc_psyMain(int channels=2, PSY_ELEMENT * psyElement=0x0017ffa0, PSY_DYNAMIC * psyDynamic=0x00dcf838, PSY_CONFIGURATION * psyConf=0x00dd1be8, PSY_OUT_ELEMENT * psyOutElement=0x00de82d0, short * pInput=0x00d80068, int * chIdx=0x00dbcc8c, int totalChannels=2)  Line 1249 + 0x111 bytes
FDKaacEnc_EncodeFrame(AAC_ENC * hAacEnc=0x00dbcc68, TRANSPORTENC * hTpEnc=0x00df7b08, short * inputBuffer=0x00d80068, int * nOutBytes=0x0038f180, AACENC_EXT_PAYLOAD * extPayload=0x001761bc)  Line 744 + 0x5a bytes
aacEncEncode(AACENCODER * const hAacEncoder=0x001760b0, const AACENC_BufDesc * inBufDesc=0x0038f36c, const AACENC_BufDesc * outBufDesc=0x0038f350, const AACENC_InArgs * inargs=0x0038f340, AACENC_OutArgs * outargs=0x0038f32c)  Line 1632 + 0x30 bytes

[mstorsjo]

Try the fix-high-samplerate-short-blocks branch - this fixes the crash for me. The fix involves redistributing some coefficients, and doing that probably does affect the psychoacoustic quality/bitrate tradeoff so I'm a little weary about which tradeoff is the best. (OTOH, anything that doesn't crash probably is better than what we have right now.)

[mstorsjo]

A probably even better solution is available in the "extend-getinvint" branch - which just makes GetInvInt support larger values.

[nu774]

A probably even better solution is available in the "extend-getinvint" branch - which just makes GetInvInt support larger values.

Yeah, I thought about that as an obvious workaround, but I wondered if this setting was out of assumption of psy-logic in the first place.
Thanks anyway.

[mstorsjo]

Yeah, I pointed out the issue to Fraunhofer, and they told me that these coefficients are set in the spec and thus can't be adjusted. As far as I can see, GetInvInt will need to be able to handle any coefficient *3/2, and only these ones that are 36 exceed the current GetInvInt table, and their initial suggestion also was to just extend the table. For their final fix one can only wait for a future code drop into a future android release.