Fix for unsafe alloca usage (CWE-770)

[emabrey]

Removed two looping usages of alloca that could lead to smashed stacks. This may come at a very small cost to latency in some cases, but it helps with program stability, as any errors or issues with alloca usage causes UB/stack overflows. When used in a loop it is incredibly easy to smash the stack, so it was identified as a high security issue. Additionally, usage of alloca thwarts inlining, as it is never safe for the compiler to inline alloca containing functions. So, in some cases, rather than an increase in latency, the compiler may now be able to provide a decrease due to increased ability to inline functions.

Signed-off-by: Emily Mabrey emabrey@tenacityaudio.org

Checklist

• I have signed off my commits using -s or Signed-off-by* (See: Contributing § DCO)
• I made sure the code compiles on my machine
• I made sure there are no unnecessary changes in the code*
• I made sure the title of the PR reflects the core meaning of the issue you are solving*
• I made sure the commit message(s) contain a description and answer the question "Why do those changes fix that particular issue?" or "Why are those changes really necessary as improvements?"*

* indicates required

[Paul-Licameli]

Learn RAII please.

[Paul-Licameli]

Friendly words of advice: no naked new! And (almost) never write delete (unless inside a destructor of your own smart pointer, or some cases where delete this is needed).

I worked hard to banish them from Audacity. Don't let them creep back into your fork. Use proper RAII.

auto chans = std::make_unique<WaveTrack*[]>(numPlaybackChannels);
auto tempBufs = std::make_unique<std::unique_ptr<float[]>[]>(numPlaybackChannels);

[Paul-Licameli]

Memory leak. Tsk tsk.

To compile with my other suggested change:

tempBufs[c] = std::make_unique<float[]>(framesPerBuffer);

And then the missing delete for this new would not matter.

But then other things are needed to fix the compilation too. Exercise for you.

[emabrey]

See line 4006: delete[] tempBufs; I don't believe it's missing. Additionally, the reason they are all the way down there at the bottom is to match the way that alloca deallocates from the stack at the end of the function.

[emabrey]

I was incorrect, I missed a level on the essentially 3D array.

[emabrey]

This was a good catch. I'm used to Java style arrays where float[x][y][z] is just syntactic sugar for float[x*y*z], which led to me not thinking about how delete[] on the float[x] array won't automatically iterate through the z level like it does the y level. For whatever reason I didn't notice it in the original leak testing, probably because on the recording setup I have it works out to be about 4.96 kBps leaked (1 * 2 * 640 = 1240 floats/sec at 4 bytes each), which given that I only recorded a 10 second check worked out to about 50 kB. That kind of leak is hard to notice on a program that starts up at ~30000 kB. Assuming that is all approximately correct, that works out to 0.01% of the initial program memory leaked per second of recording/playing, which was hard to spot given I wasn't using an automated tool, and as I already said my mental model wasn't perfect due to my having a lot more written in similar but not identical languages.

On a side note, I appreciate the constructive criticism.

[Paul-Licameli]

Constructive criticism is the highest compliment. It means work is taken seriously. We must all learn to give it and take it with grace.

[Be-ing]

The first rule of audio programming is no time unbounded behavior in the audio thread. That means no heap allocation. Not sometimes, zero -- never. The worst case scenario will happen sometimes and the result is audible glitching.

I was unfamiliar with alloca before so I didn't raise concern about this earlier. You may very well be right that this usage of alloca is dangerous. But switching from the stack to the heap is not an acceptable solution in realtime code. You are correct that the right solution is to move memory allocation outside the realtime thread. That is a bigger task than this PR though. In the meantime, I am inclined to heed @Paul-Licameli's advice and revert this change without evidence of real world crashes caused by the old code.

[Paul-Licameli]

Stack "allocation" happens all the time and isn't so great a concern.

alloca is a nonstandard function implemented as a compiler intrinsic on all of the few compilers that matter, for a stack allocation of a size not known statically, and it just changes a few registers. (And may specially affect compilation of a function containing it.)

C99 diverged from C++ and made variable-length arrays a feature, making alloca unnecessary -- but then C11 made that language feature optional. https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Optional_features