Played notes are exceedingly out of tune due to rounding errors #317
Description
Overview
A rounding problem with Note Block Studio's pitch calculation during playback was found to exist since the legacy versions of NBS. The farther a note is from its original key (F#4 for every default instrument), the worse the error; notes just two octaves away from their original pitch can be a jarring 25 cents (a quarter of a semitone) off from their intended tuning.
To further clarify, this does not have to do with the note editing system; the reported pitch and key values for each note work correctly, but the notes they produce when played back in NBS are not.
Investigation
Initially, the MP3 export, provided by audio.dll, was thought to render notes a few cents off from their intended tuning, supposedly due to rounding errors in the pitch shifting calculation -- increasing the pitch by a non-integer factor might result in a non-integer sampling rate, which is much harder to deal with and isn't accepted in ordinary resampling functions. So it must be rounded to an integer prior to converting the sound, which may make the pitch slightly lower or higher than intended:
This was always thought, in the NBS community, to be the reason why MP3 export was "out of tune": precisely because it didn't match the pitch you'd hear in NBS for a given note.
Due to other flaws with the built-in MP3 export, nbswave was created in an attempt to circumvent those issues, intended to replace audio.dll as the default audio export in NBS. However, it suffered from the exact same issue: notes were out of tune in comparison to their NBS counterpart, supposedly (again) because of integer rounding:
As an experiment, nbswave was changed to use a non-integer signal resampling algorithm using sinc interpolation provided by the resampy library. This, however, prompted no success: notes were equally out-of-tune as before.
After some thorough testing, I came to the conclusion that nothing else in the code could be causing the sounds to get out of tune. Which led me to think the problem must've been somewhere else: fatally, in NBS itself.
The bug
Here's the line that controls the pitch of played back notes in Note Block Studio:
This formula arises from the following calculation, which is explained beautifully in this article:
Our experience of pitch is logarithmic, so that the multiplication of a frequency produces a particular musical interval. Double a frequency it goes up an octave, halve it it goes down an octave, multiply by ≈1.059463 and the pitch goes up one (equal-tempered) semitone. This catchy number is actually the 12th root of 2, which makes sense if you think about it. Multiply it by itself 12 times and it reaches 2, because there are 12 ‘equal’ jumps between in each octave in the 12-tone equal-tempered system.
However, there are two jarring issues in the NBS implementation:
- The twelfth root of 2, 1.059463, was manually rounded to 1.06. This exists since legacy NBS.
- The constant 0.495 is supposed to be 0.5, which it used to be in legacy NBS -- commit a5fc47a changed this factor, likely in an attempt to make the pitch more accurate. While this change did help counteract the rounding error, it added another layer of inaccuracy and didn't actually fix the problem.
Result
The farther away a note is from its original key (F#4 for all default sounds), the more out-of-tune it will be in relationship to its intended pitch in equal-tempered tuning. The pitch error is estimated to be ±10 cents for every octave up or down from the default pitch. For instance, C2 is off by 33 cents, an awful third of a semitone:
key = 17 (C2)
(key + (ins.key + (pit/100) - 78))) = (17 + (45 + 0 - 78)) = 17 + (-33) = -18
0.495 * (1.06 ^ (-18)) = 0.1734
0.5 * (1.059463 ^ (-18)) = 0.1768
F#4 = 369.99 Hz
C2 = 65.41 Hz
0.1734 * 369.99 = 64.1637 Hz = C2 -33
0.1768 * 369.99 = 65.4057 Hz = C2 +0
Incidentally, this means that there's nothing wrong with the MP3 export: the error caused by rounding off a non-integer sampling rate is negligible compared to the rounding error added up for every semitone off from the default pitch.
In short, the MP3 export, nbswave and every other thing using the same resampling method were thought to be wrong just because they didn't match what we heard in Note Block Studio. Well, turns out it was the bad guy after all!!! 😱😱😱
Demonstration
In the pictures below, the note C2 is being spammed in NBS. A tuning website is capturing the speaker's output using my phone's microphone, and displaying the estimated note. Left is the result in NBS 3.9.3, right is the patched pitch calculation algorithm:
Following this experiment, this exact song was exported using the legacy MP3 export, the MP3 export after the pitch calculation patch in ceb65b0, and nbswave. Here's the comparison:
While the MP3 export is still not as accurate, the patched version makes it perform worse on lower notes, since it always subtracts 23 cents regardless if the note is lower or higher than the intended pitch.
Rather, we notice here that the MP3 export suffers from the same problem as note playback (the dreadful 1.06!), which, again, was patched in a way that doesn't actually solve the problem! However, in comparison to the playback in NBS, the unpatched version sounds slightly closer to the intended pitch. Why is that? Because it lacks the change from 0.5 to 0.495, done in the playback code.
So we have two different components of NBS doing pitch calculations in wildly different ways, which causes pitch across the program not only to be wrong, but inconsistently wrong!
While nbswave was used a "correct" reference above, the built-in MP3 export can be easily fixed to produce the same results, unlike what we thought before -- some problem with rounding that could only be fixed by recompiling the DLL.
(Try the tuning website, it's really fun :) )
Attached here is a song comparing notes exported by nbswave (using the non-integer resampling method) with the notes in NBS: outoftune-test.zip
Fix
While fixing the constants above can fix rounding issues to a reasonable extent, as shown by the pictures above, replacing hardcoded values with their originating calculation can virtually remove any rounding errors, since no manually calculated values will be embedded in the formula. This is already being done in data pack export; ideally, both the playback and MP3 export pitch calculations should be changed to match it.