Playback of 24bit FLAC is not bit-perfect despite 16bit FLAC being bit-perfect

[NintendoManiac64]

I know I know, this is some super-duper OCD audiophile crap that nobody in their right mind should ever care about, but I'm concerned about the possibility of a bug in how Ren'Py handles greater-than-16bit audio (which is actually pretty common with lossy formats since the likes of MP3, Vorbis, and Opus store their audio data as floating point).

Basically, while doing some audio tests to see how Ren'Py handles different audio formats, I discovered that it supports both 16bit and 24bit FLAC

...yet, when actually doing an audio recording to compare how Ren'Py renders the audio compared to the source files, only the recording for the 16bit FLAC ended up being 100% identical to the according FLAC file while the recording for 24bit FLAC ended up being a teeny bit different - almost like it was being reduced in bit depth to something between 24bit and 16bit, like 20bit or something.

Again, I realize this is super-duper unimportant, but the inconsistency in audio rendering makes me wonder if there's a bug or something. I mean, if Ren'Py doesn't handle 24bit audio output then I wouldn't expect it to even support 24bit FLAC files, and yet it does. This could arguably make sense if it just always truncated or down-sampled everything to 16bit audio output, but doesn't explain why the resulting audio render seemed to still be so different from the 16bit equivalent.

　

For reference I took a 192kHz 24bit copy of Simon & Garfunkel's "The Boxer" and downsampled it using foobar2000 and the SoX plug to both 48kHz 24bit and 48kHz 16bit. For recording I just used Audacity (though admittedly an older version from before the UI change - v2.1.2) set to record "speakers" on my Xonar DS sound card which was similarly set to 48kHz 24bit. All volume sliders in Windows, Audacity, and Ren'Py were set to 100%.

To compare the recording to the according FLAC I lined up the waveforms to the exact identical sample, inverted one of the waveforms, mixed the two together to a new waveform, and then looked at the value listed in "amplify" to see the difference - the larger the value, the smaller the difference.

Comparing the 16bit FLAC to its according recording returned an amplify value of "Infinity" (identical) while comparing the 24bit FLAC to its according recording resulted in an amplify value of 50dB with "New peak" listed as -40dB (so basically 90dB). Comparing the 16bit FLAC to the recording of the 24bit FLAC returned an amplify value of 50dB with "New peak" listed as -13dB (so basically 63dB).

[renpytom]

Ren'Py only support 16 bit audio playback, and should downsample.

[NintendoManiac64]

Ren'Py only support 16 bit audio playback, and should downsample.

Do you know if the internal volume slider also only operates at 16bit?

I ask because the audio mixer/renderer, including the volume slider, in Windows since Vista has been 32bit float and only dithers down to 24bit or 16bit for the final output.

So if the internal volume slider in Ren'Py is only 16bit as well, then I'll keep in mind to rely much more on the OS volume control rather than Ren'Py's since my IEMs don't take much volume to operate (sometimes I have the OS volume set to upper single digit percentages).

[uyjulian]

Ren'Py is hardcoded to 16 bits output and decode.

renpy/module/ffmedia.c

Line 730 in 2fbb7be

converted_frame->format = AV_SAMPLE_FMT_S16;

renpy/module/renpysound_core.c

Line 1162 in 49e4a11

audio_spec.format = AUDIO_S16SYS;

[NintendoManiac64]

Ren'Py is hardcoded to 16 bits output and decode.

Bloody nora, I only just realized it's freakin you uyjulian! I knew you had knowledge in Ren'Py, but I didn't know you were straight-up a code contributor for it.

Well uh, thanks for the clarification I guess? I'm also guessing that simply changing those two specific "16" numerical values in the Ren'Py code to something like "24" would cause horrible breakage elsewhere in the codebase?

[uyjulian]

I'm also guessing that simply changing those two specific "16" numerical values in the Ren'Py code to something like "24" would cause horrible breakage elsewhere in the codebase?

You would probably need to change buffer sizes too...

renpy/module/ffmedia.c

Line 26 in 2fbb7be

static int audio_target_samples = 44100 * 2;

renpy/module/ffmedia.c

Line 29 in 2fbb7be

const int BPC = 2; // Bytes per channel.

renpy/module/ffmedia.c

Line 30 in 2fbb7be

const int BPS = 4; // Bytes per sample.

However, this may not be all the locations that need to be changed. I haven't actually tested this myself.

[Andykl]

Don't forget, you need not only change that but also recompile the RenPy as it is C code. I wouldn't go into it without a deep understanding of what you're doing.

[NintendoManiac64]

While 16 vs 24bit at 100% is pretty placebo level, I'm mainly just thinking about at lower volumes since, as was mentioned, Ren'Py apparently is using 16 bit output even for the mixer which is something that desktop operating systems moved away from 15 years ago while even mobile OSes have moved away from a few years back.

I mean technically the best solution would be 32bit floating point which is what those OSes and stuff use nowadays, but the idea was maybe that simply bumping the integer values from 16 to 24 would be simpler. Though that does kind of make me wonder about the likes of simply bumping the integer values even higher, like to 32bit integer or going the opposite route of sticking with 16bit but making it 16bit floating point.

[renpytom]

Is there some documentation about mobile OSes supporting 32-bit? I'm not going to to move Ren'Py to a floating point audio format, but s16 integer seems reasonable. Also, note that this is meant for a game engine, so I'm not going to judge audio playback by the standards of audiophile applications.

[NintendoManiac64]

So I was slightly off - iOS uses 24bit integer and 16bit integer, but possibly for different things? My major lack of experience with Apple stuff is making me uncertain as to what they're actually referring to:
https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/CoreAudioOverview/CoreAudioEssentials/CoreAudioEssentials.html#//apple_ref/doc/uid/TP40003577-CH10-SW16

But Android by comparison is straight-up 32float nowadays:
https://source.android.com/devices/audio/highres-effects

So I would imagine that whether iOS would allow 24bit or if it requires 16bit from apps is the bottleneck here.

And yes 16bit integer is "reasonable" but it's kind like having your monitor set to a 60Hz refresh rate back in the days of CRTs rather than 75Hz despite AFAIK basically all CRT monitors also supporting 75Hz (which of course on a CRT also reduced the flicker) - outside of actually needing to actually put in a bit of time to change it, there's not really much (if any?) downside except maybe ever so slightly increased processing time? (at least for floating point audio; I'm not even sure if 24bit integer audio would use more CPU than 16bit integer audio when all of these CPUs are at least 32bit architectures anyway unless the compiler was specifically able to fit two 16bit integer operations within a single 32bit integer operation).

It's also kind of like the difference between decoding an h.264 video at 720p 30fps vs 720p 60fps - for specifically the decoding (because obviously you'd ideally want a higher bitrate and therefore larger file size for 60fps), yeah it used to be quite the difference in terms of hardware demands, but nowadays the difference for decoding is a drop in the bucket in terms of demands due to hardware decoders, but even older mobile CPUs are fast enough to software decode it nowadays (you have to get to Pentium 4/Athlon XP/1st gen Atom levels of bad IPC at the usual 1+GHz clockrates before it becomes a problem).

A more apt analogy is probably how PC gamers used to select 16bit color for better performance back in the late 90s, but nowadays you might as well just render everything at 24bit color/32bit color since the performance difference is practically non-existent on even the weakest of Intel integrated GPUs.

Now admittedly those three analogies have more known noticeable differences in the end result than 16bit vs 24bit audio, but again my main focus here is more on the actual mixing since lower volume levels are inherently more detrimental and thereby more noticeable the lower your bitdepth is.

[NintendoManiac64]

So I was re-visiting this thread...

You would probably need to change buffer sizes too...

renpy/module/ffmedia.c

Line 26 in 2fbb7be

static int audio_target_samples = 44100 * 2;

I've admittedly not really much (if any) idea of the code that handles audio in Ren'Py, but I don't see why sampling rate would have any relation to bitdepth.

...though my lack of knowledge of how Ren'Py is actually decoding the audio is becoming apparent as, in my mind, it saying 44100 samples flies directly in the face of my audio recording tests only getting a perfect 1:1 lossless result vs the source file with 48000Hz audio implying that Ren'Py's native sampling rate is 48000 and not 44100 (even if the OS, sound card, and audio recorder are all set to 44100Hz, Ren'Py still definitely resamples everything to 48000Hz).

renpy/module/ffmedia.c

Line 29 in 2fbb7be

const int BPC = 2; // Bytes per channel.

renpy/module/ffmedia.c

Line 30 in 2fbb7be

const int BPS = 4; // Bytes per sample.

However, this may not be all the locations that need to be changed. I haven't actually tested this myself.

I would imagine for 24bit the buffers would only need to be increased by 1.5x (so 3 and 6 in this case) since 16bit LPCM equals 2 bits per sample while 24bit LPCM equals 3 bits per sample (which can be confirmed via that "Header Investigator" program I previously mentioned to you, or via foobar2000 or even just the Windows Explorer's "bitrate" value if looking at plain uncompressed LPCM where 48kHz 16bit = 1536kbps while 48kHz 24bit = 2304kbps).

(and if it wasn't obvious, 32bit would simply have a straight 2x the bitrate of 16bit whether integer or floating point 32bit, though that ignores the intricacies that an integer decoding path can't decode floating point audio as-is nor does it take into account that the 32bit floating point, such as what all modern OSes save for iOS uses, only has 24bits of actual precision which makes outputting 32bit integer audio a bit of a question mark, so... 24bit audio just seems the the best place to focus for the time being)

Regardless, I don't suppose it's possible to take a theoretical compiled-for-24bit Ren'Py and simply overwrite the .EXE and/or .SH of a given previously-compiled Ren'Py VN without the need to change anything else in a sort of drop-in-replacement manner? Or would the entire VN itself need to be recompiled?

I only ask because there's one relatively lengthy Ren'Py VN that I've been reading that, if I were able to use with a theoretical 24bit-enabled version, could work as a real-world test - especially since it just uses a bunch of freely-available Kevin MacLeod music (which I applied wavegain to targeting 83dB encoded as FLAC). The VN is FOSS though so recompiling the whole thing could still be done if necessary, but would be more of a hassle obviously.