Handle memory allocation failures, and reduce overflow

[jyavenard]

with comments addressed.

[jyavenard]

r? @tmatth

[tmatth]

Since this is a static function I don't think this check is necessary, you could do speex_assert(result) at the beginning of this function if you want to be safe.

[jyavenard]

To be honest, I feel like you're now stretching my good will. You are free to use this pull request as it is, or modify it as you want. But be aware that that it has been classified as sec-critical.
Seeing that it's been over 3 weeks since I first submitted handling out of memory errors and it has yet to be included, I believe my job is done now.

[tmatth]

I understand that slow review and resubmitting patches over details is tedious. Please keep in mind that this is just a github mirror of speexdsp. Normally, development takes place on the speex-dev mailing list (hence this PR flew under my radar). As is too often the case, review/dev/maintenance is done on a best effort basis by volunteers.
In any case, thanks for the contribution, will merge today. Please feel free to ping me or jmspeex on #speex if you'd like to talk further.

[tmatth]

Applied in 8c2981f and 335a9a1

[jyavenard]

I couldn't find on the speex website a description about how to best handle a change request.
Nor could I find details on what the coding style is, so I took a shot. So I have to admit that I got frustrated as the target to reach appeared to be a moving one.
The main reason I didn't want to assume that result wasn't null, was that it allowed to use the _muldiv to only check for overflow, without actually performing the operation. There's a few other places in the code doing a = a*b/c, all of those appeared fine to me, but you can't never be sure about it.

Maybe, just to be safe and avoid all the issues alltogether, would be to have a quick check at the beginning of the resampling operation to check that the number of frames and the number of channels are reasonable (and in particular that frames * channels * sizeof(float) will not overflow) and abort if not. I understand that speex had the assumptions that the incoming values were reasonable. Unfortunately, the way firefox used it broke those assumptions. FWIW, speex can't be called with those crazy user input anymore.

[tmatth]

Maybe, just to be safe and avoid all the issues alltogether, would be to have a quick check at the
beginning of the resampling operation to check that the number of frames and the number of
channels are reasonable (and in particular that frames * channels * sizeof(float) will not overflow)
and abort if not.

Sounds reasonable to me.