BeatDetect problems

[labkey-matthewb]

Hi all, I copied out a few files (PCM, BeatDetect, and fftsg) to put together a little sound reactive project. In reading through the code, I came to the conclusion that the code in BeatDetect is really broken. I hope I'm wrong, but here's what I'm seeing.

1. Whenever audio data is added to the PCM buffer, we compute an FFT on the last 512 samples and put the result in PCM->vdataL and PCM->vdataR.

getPCM(vdataL,512,0,1,0,0);
getPCM(vdataR,512,1,1,0,0);

2. BeatDetect::detectFromSamples() ignores the FFT data in vdataL/R and passes pcmdataL/R into BeatDetect::getBeatVals(). This is the first disconnect, as I believe getBeatVals() thinks it is operating on FFT data not PCM data.

getBeatVals(pcm->pcmdataL,pcm->pcmdataR);

3. Even if getBeatVals() were handed FFT data, I think it is not quite right. Looking at the documentation for rdft() in https://github.com/projectM-visualizer/projectm/blob/master/src/libprojectM/fftsg.cpp, it looks like the real components of the fft are stored in the even indexes of the output array. I think we should be skipping the imaginary coefficients in the odd slots and also the value at [0] (that's just the mean of the input data).

4. I'm not sure the ranges for bass,mid,treb make sense. For instance, the code sums up the first 16 coefficients (indexes 0-15) to compute beat_instant[0], but if I understand this correctly, I don't think coefficient 15 would be considered bass. Assuming sample rate of 44100/sec I think it would represent something like 15*44100/512 = 1291hz?

But this does seem to work today? I think what's going on is that bass/mid/treb are all basically just proxies for volume (with some additional randomness), and this probably works fine for most presets. It's definitely possible I've got this all wrong, but I couldn't make sense of the BeatDetect output using generated sine wave data input, until I really reworked all this.

[revmischa]

Well it sounds like it probably couldn't be made worse by the fixes you're proposing.
For bass/mid/treb, should the range cutoffs be dynamic based on sample rate?

[labkey-matthewb]

Might not matter much if sample rates vary say by a factor of 2x, but it would certainly be more correct. I can propose a change, but it would be great to have someone with more than a "read the example code" experience with sound analysis to chat with.

[auralarch]

So, I'm not an expert on sound analysis; but I do have some knowledge which comes from working with sound signals as a DJ—often in high amperage settings (such as the large sound system of any club, festival or other system with many large speakers), and I also have knowledge sound wave frequencies from my traditional/classical music training.

So, it seems to me that some of the assumptions implied by the text above are fundamentally flawed. Let me explain... So, frequencies remain constant in music playback and are not affected by sample rate.

To make an analogy comparing a segment of digitized recorded music with a digital image. The sound frequencies (or specifically the sound wave lengths) would correspond to the colors in the image (the light wave frequencies)—changing these would fundamentally and radically alter and deform our experience of the media into something very difference (although probably not entirely unrecognizable).

The sample rate of music on the other hand corresponds to the resolution of the image. Just as we perceive shapes in a digital image when we are actually just seeing a bunch of dots/pixels. What we percieve as sustained tones in digital music are actually a series of consecutive samples—perhaps a better analogy would be with video and frame rate: digital music is made up of frames just like video and we perceive the series of images as moving objects when they are really a series of still images. That is exactly how recorded sound works. The sample rate is the number of frames of sound per second (which is much higher than the number of frames of images we need per second in order to perceive the end result as fluid movement/music. But the sample rate and the frame rate do not affect or alter the wavelengths of the media represented—if it did the colors of the images would change when the frame rate changed and the key/pitch of the music would shift wildly (think about playing a vinyl record and speeding up or slowing down the platter: speeding it up results in the music sounding like chipmunks or the smurfs and slowing it down makes a woman sound like a man and a man sound like the devil.

For a real world example of this you can get this effect using the VLC media player. All you have to do is change one of the settings. Open the preference panel in VLC, click the "Show All" button to activate the advanced settings panel, navigate to the main Audio page and deselect the checkbox that activates "time stretching". Save the change and relaunch VLC for the change to take affect. Now play a video and then change the playback speed (the default keyboard shortcuts are [Cmd]/[Ctrl] + the left & right bracket keys (Cmd if you're on a Mac).

So, the notes or pitch (highness/lowness) of sounds is determined by the wavelength (which is measured as the distance between two peaks of a sound wave), just as the color is determined by the wavelength of light (the wavelengths of light are just many factors smaller/shorter than those of sound).

Volume/amplitude/loudness in music, on the other hand, is determined by the height of the waves (the distance between the trough and the peak of a wave)—so this is a vertical measurement and the one above (for pitch/color) is a horizontal measurement. And both of these are separate & independent from the sample/frame rate.
Now, where compression comes in with recorded sound is not with the frequencies of the wave length (the pitch) but in the differences between the loudest and softest sounds. So, it's not just shrinking the height of all the sound waves; it's much more complex than that. The softer sounds are made louder and the louder ones softer so that everything is much closer to the same volume—it's the dynamic range that is compressed in order to use less data.

In other words, assuming that the developers who first created/ported this project didn't screw up the way that the beat detection works, the ranges for bass,mid,treble will be constant and unaffected by sample or compression rate.
(So, in direct response to your question (@revmischa ): No, the range cutoffs should not be dynamic (in the sense of "shifting" up and down) based on sample rate. But if you mean dynamic where "dynamic range"=volume/loudness (which I don't think that's what you're asking); I'm not completely sure although I still feel confident in saying: No. Although, there typically is some loss of dynamic range associated with lower sample rates.

Of course, there's a whole lot more interaction going on than just beat detection as far as the music affecting the visualizations (at least in Milkdrop) wavelength frequencies are driving all sorts of other aspects of the visualizations.

I really hope that this information is helpful in some way. I would be happy to explain things further (to the extent of my knowledge and understanding) if you have any questions. Although, I can't guarantee a prompt response time as my schedule and access to my computer is cyclic. But I will monitor this as best as I can.

I don't have any idea how similar the code used by projectM is to the code used by Milkdrop for Beat Detection, etc. But I know that Milkdrop was/is very effective in the implementation of this feature, so as much as you can copy, port or glean/learn from the original code the better this feature will be!

[labkey-matthewb]

What would be most useful is confirmation that I'm using the fft data returned by rdft() correctly in pull request #312. The bulk of my testing was a) printf and b) viewing test presets, this one in particular https://github.com/projectM-visualizer/projectm/blob/master/presets/tests/300-beatdetect-bassmidtreb.milk

[kblaschke]

Milkdrop itself only uses 1/4 of the spectrum in relation to the samplerate, so if the input is at 44100 Hz, the spectrum will cover 0-11025 Hz. Then it's split into three bands for further processing.

Same as projectM currently, Milkdrop actually never cared much about the samplerate. The algorithms are based on a 44.1 kHz unsigned 8-bit PCM data stream, so if the actual audio has a different samplerate, beat detection and spectrum rendering will suffer.

[kblaschke]

The beat detection code has seen some work recently, and should generally work better than at the time this issue was created.

If anyone still has specific issues with beat detection, please provide an example/use case and open a new issue!