New Rubberband R3 breaks mono compatibility, phase issues

[atskler]

Bug Description

I just observed that engaging the Rubberband R3 engine it creates serious phase issues and breaks mono compatibility. This can be really serious problem for broadcasting and for partially or totally mono sound systems. For me the issue is more apparent with speakers than with headphones.

Here is a sound sample:

https://on.soundcloud.com/YwQmG

keylock on
0:0.01 default sound
0:8.00 mono mode on
0:13.00 adjusting speed +0.13, 0, -0.13
0:36.00 switching back to stereo
0:40.00 adjusting speed +0.13, 0, -0.13

Version

2.4-alpha-1419-g0243efdaf8 (main)

OS

OpenSUSE Tumbleweed

[atskler]

I tried the command line version with:
-R -t 1.013 -3
and it sounds like the implementation in Mixxx

The following additional option can somewhat improve the situation:
--centre-focus Preserve focus of centre material in stereo (at a cost in width and individual channel quality)

Either way it sounds terrible especially with poor mono compatibility and it is really really far from any kind of near-hi-fi quality (as far as a bad 96kbps mp3 is), so describing it this way in preferences is quite misleading. Near-total-cpu-consumption-for-pointless-sonic-destruction could be a better description of it. Seriously.

Anyone who would play through a mono system will sound really bad.

This library maybe could be good for individual solo instruments, but not for complex final mixes.

[daschuer]

What could be a solution? Down mix the track to mini before processing with Rubberband?
Thus could be also a CPU saver.

[atskler]

I experimented with it a little. For R3 probably the Mid/Side (M/S) processing could be the best compromise (and stamping it "experimental" in preferences). Although I tried it with one track. (In M/S processing the usage of --centre-focus is forbidden.)

Downmix to mono would just create another problems, you would need 6dB headroom for it and probably it will result in rightful user complaints, and...

The last question is how it will sound in Mixxx when it engages just above or below 0% of tempo change, as at 0% it does no processing. Probably 0% should be not allowed, it should be skipped, the 0% should be +0.001% or -0.001% to circumvent the sound difference at the point of engage.

Again this is not an ideal solution. It will disturb the original M/S ratio or balance (it translates as modified stereo wideness), but if Rubberband R3 is needed this is probably the best compromise. Until someone suggest a better one.

[atskler]

(It seems that M/S processing produces somewhat more pleasing results with R2 also. Still R2 M/S and R3 M/S is far from near hi-fi sound.)

[atskler]

Most importantly the M/S processing solves the mono compatibility issue.

[robbert-vdh]

None of the Rubberband algorithms perfectly preserve phases between channels. There's sadly not really a way around it. Here's a comparison between Rubberband R2 and R3, Elastique, and Elastique Pro on some transient heavy material with distinct textures. The Rubberband algorithms are tested as is, with the 'channels together' option (which is what the option you're looking for here, rather than center focus), with the shortwin option, and with both options. This comparison uses pretty extreme time settings but the effect is already noticeable when doing very slight time stretching.

time-stretch-comparison-opus.tar.gz

[robbert-vdh]

From the samples in that archive R2+channels together sounds the closest to the stereo image from the Elastique algorithms.

[atskler]

@robbert-vdh your samples are too short, no original provided and to be honest they are just squashed industrial noise pitch shifted.

And the problem is that even with slightest speed change drastic changes are in the sound.

And I found on the following link no 'channels together' option:
https://breakfastquay.com/rubberband/usage.txt

And M/S processing preserves the phase between channels, or at least keep the mono compatible. Try it for yourself.

What does Mid/Side and Sync mean in pitch shift?
https://forums.cockos.com/showthread.php?t=92385

[daschuer]

Is there an option in Rubberband to use M/S Processing? If not, I can imagine to do the the whole engine processing in M/S.

[atskler]

The command line version don't have M/S option.

The API documentation mentions these:

• OptionChannelsApart - Channels are handled for maximum individual fidelity, at the expense of synchronisation. In the R3 engine, this means frequency-bin synchronisation is maintained more closely for lower-frequency content than higher. In R2, it means the stereo channels are processed individually and only synchronised at transients. In both engines this gives the highest quality for the individual channels but a more diffuse stereo image and an unnatural increase in "width". This option is the default.

• OptionChannelsTogether - Channels are handled for higher synchronisation at the expense of individual fidelity. In the R3 engine, this means stereo synchronisation is maintained more closely for the full frequency range. In R2, it means the first two channels are considered to be a stereo pair and are processed in mid-side format, with mid and side processed as if they were separate channels before being recombined. This usually leads to better focus in the centre but relatively less stereo space and width and lower fidelity for individual channel content.

Probably the best to do in Mixxx before and after Rubberband, maybe with providing an option in preferences for M/S processing.

[daschuer]

I agree. Do you have interest to contribute that?

[atskler]

Unfortunately I still don't have any programming knowledge.

[atskler]

@daschuer I created a bug report for this issue on the following link, because it can help others with other projects also:
breakfastquay/rubberband#81

[uklotzde]

Thank you for reporting the issue upstream! Very concise and well-written summary of the observed behavior.

[atskler]

You are welcome. Most probably this issue will be solved quickly upstream.

[cannam]

Rubber Band 3.2.0 fixes this - OptionChannelsTogether now preserves mono compatibility.

[daschuer]

🎉 Thank you very much!

[Swiftb0y]

Great, now we just need to actually implement mixxx to use that option... Should we put it behind a config option?

[daschuer]

For my understanding the mono compatible sound is what we want. Do you think we need a preferences option for the old behavior? I don't see a use case for it.

[Swiftb0y]

My understanding is that it results in slightly worse quality. When you don't need mono compatibility (for example when you're just livestreaming your DJ set), you might not want that. I may do some tests to hear whether it makes much of a difference.

[cannam]

Please do do some tests - and if you find any horrible quality problems, report them in the Rubber Band tracker!

This option is expected to sound a little worse than the default in the R2 engine (except in mono situations) but I am pretty optimistic about the R3 engine in 3.2.0. Initial tests suggest that it might indeed be good enough to be used by default, and in fact may sometimes be preferable to the default options even when not mixed to mono. I would certainly at least test it.

[Swiftb0y]

So I just spend 30 mins testing and while it sometimes feels like there is a difference, I can't rank the results any difference in terms of quality.

Can you clarify when you recommend to use this option? What I could gather from breakfastquay/rubberband/issues/81 is that you just recommend it for R3 with version >= 3.2.0. Is that correct?

[atskler]

When you don't need mono compatibility (for example when you're just livestreaming your DJ set), you might not want that.

@Swiftb0y your listener/watcher can still have mono sound system.
And without the mono compatibility option the stereo result will be less true to the original.

[Swiftb0y]

uncommon, but fair. As I see no real disadvantage to using the mono-compatible processing, whether to use it does not have to be discussed anymore. The question is now, under what circumstances can we turn it on unconditionally without impacting the quality otherwise.

[cannam]

With R2, the default options "win" (according to a modest amount of listening testing) when using stereo equipment, but OptionChannelsTogether gives mono compatibility, so there is a genuine tradeoff there. Since R2 is somewhat "legacy" it perhaps makes sense just to stick with whatever your current configuration is. With R3 prior to v3.2.0, the default options also "win" in stereo, but by a finer margin - most people won't notice the difference. However, while OptionChannelsTogether improves mono compatibility a little bit, it is still not really there in those versions (hence this bug report). So there is not much of a tradeoff - you could pick either and probably nobody would be much the wiser. With R3 in v3.2.0 we have far less testing to support the idea that either option sounds better than the other in stereo. First impressions are that they are different in character but fairly similar in quality. But OptionChannelsTogether now gives mono compatibility again. So unless problems with the new OptionChannelsTogether implementation arise, there is a reasonable, if provisional, case for using it as default. Altogether this seems to suggest one could probably switch it on whenever the engine is R3. But caveats apply... and I would always suggest more testing...

…

On Wed, 29 Mar 2023, at 20:45, Swiftb0y wrote: uncommon, but fair. As I see no real disadvantage to using the mono-compatible processing, whether to use it does not have to be discussed anymore. The question is now, under what circumstances can we turn it on unconditionally without impacting the quality otherwise. — Reply to this email directly, view it on GitHub <#11361 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAZTZNRT63S6KXBM7RWNFLW6SGNXANCNFSM6AAAAAAVYMYNFU>. You are receiving this because you commented.Message ID: ***@***.***>

[cannam]

(To be clear, I'm just commenting to describe the different behaviours of the various Rubber Band versions with this option. I don't presume to know what the users of DJ software want or need in terms of compatibility or characteristics.)

[Swiftb0y]

Thank you very much for your insight. @daschuer whats your opinion? Should we just turn on OptionChannelsTogether for R3 unconditionally?

[daschuer]

Yes that sounds reasonable.

[Swiftb0y]

fixed by #11418