Allways remove infs/nans

[zonkmachine]

Brief

Testing to switch the inf/nan tests that are performed on export to be 'on' allways.
This seem to work well. For some 2.5% increase in cycles most of the NaN oriented bugs can be transformed from a project cutting out all sound at a specific point or from start, to just a moderate glitch.
I currently lack real test cases to assess this.

---

Verbose

I split the original PR into two separate commits and added a new one.

2fc1802 Turn on sanitizers in the FX-Mixer. This effectively prevents the whole projects to go silent because of one channel with a bad signal.

a85b02c Turn on sanitizers in the effect chains. This prevents bad data from shutting down the channel. Basically all NaN ( Warning! Will get loud without 47ac277 )

47ac277 As of the previious commit some dangerously large signals can come through the mix.

... has some amazing new noise going on with this PR

Fixed. Clamp values to, for the time being, +/-1.0f . I understand introducing hard limiting like this could trigger some but I see no reason to allow overly large signals. Don't other projects perform some form of clipping?
At this level a glitch will propagate through a reverb with long decay more like a distinct tick. A slight performance hit. +~0.2%.

---

Profiling

Test Project: Skiessi - Onion
Result: ~2.5% is spent in MixHelpers::sanitize() and MixHelpers::addSanitizedMultiplied()routines.

$ operf ./lmms

zonkmachine@zonkmachine:~/builds/lmms/lmms/build$ opreport --callgraph
CPU: AMD64 family15h, speed 3.4e+06 MHz (estimated)
Counted CPU_CLK_UNHALTED events (CPU Clocks not Halted) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        image name               symbol name
-------------------------------------------------------------------------------
433771   14.1108  caps.so                  void Clip::one_cycle<&(store_func(float*, int, float, float))>(int)
  433771   100.000  caps.so                  void Clip::one_cycle<&(store_func(float*, int, float, float))>(int) [self]
-------------------------------------------------------------------------------
390387   12.6995  libQtGui.so.4.8.6        /usr/lib/x86_64-linux-gnu/libQtGui.so.4.8.6
  390387   100.000  libQtGui.so.4.8.6        /usr/lib/x86_64-linux-gnu/libQtGui.so.4.8.6 [self]

/////

61433     1.9870  lmms                     MixHelpers::sanitize(float (*) [2], int)
  61433    100.000  lmms                     MixHelpers::sanitize(float (*) [2], int) [self]
8048      0.2603  lmms                     MixHelpers::addSanitizedMultiplied(float (*) [2], float const (*) [2], float, int)
  8048     100.000  lmms                     MixHelpers::addSanitizedMultiplied(float (*) [2], float const (*) [2], float, int) [self]

fixes #1048 #3685

[zonkmachine]

The demo project in #3408, envelopeNaN.mmp.zip has some amazing new noise going on with this PR.

What happens is probably something like this: Some faults give you infinite values and these propagated through the sound chain will metamorphose into NaN. The NaN will make lmms go silent.
When we add the sanitizer the NaN/inf values are sorted out but some crap will get through in the form of short glitches around where the infs. would have been, because you'll have values leading up to/from the infinites. These will now pass straight through and if you have a reverb in there... Magic!!! ( Edit: turn volume down... )

[zonkmachine]

If you only turn on the Sanitizers in the FX-Mixer the sound will cut out on just the troubled channel and not affect the whole mix. This could be a better option.

[PhysSong]

I see no reason to allow overly large signals.

Some projects use them as a intermediate signals. If we clip them, some projects may give different sound output.

And I have a question: what if an instrument generates NaN and its output goes to master FX channel directly? If there is something like master reverb, the result will be worse.

[zonkmachine]

Some projects use them as a intermediate signals. If we clip them, some projects may give different sound output.

Yes. This is a destructive PR in that sense. You could have a switch to allow higher signals. I don't know how many projects would be affected though. It would probably be more user dependent, the ones that has the habit of driving a large signal through the FX channel.

what if an instrument generates NaN and its output goes to master FX channel directly? If there is something like master reverb, the result will be worse.

Not worse but equally shitty I think. I've kept an eye open on the master channel but so far I haven't seen it acting up because of this in any way. And I've hit it with NaN intentionally. I'll try and come up with a more angry test... :)

[tresf]

If this fixes the notorious bug where the mixer from goes silent, this comes with a lot of value. @zonkmachine @PhysSong Can we make a decision on this and merge?

[zonkmachine]

If this fixes the notorious bug where the mixer from goes silent

It does. I think it should be merged and I think perhaps 1.2 is a better target than master.

[PhysSong]

A sample file may contain inf/NaN stuffs. In that case, this PR won't help to fix such issues.
I guess PlayHandle is also a candidate for sanitization, but I don't know how it will affect performance.

Anyway, Clipping signal at +/- 1.0f doesn't sound good for me. I think it should be some appropriate larger level(ex. 4.0f~10.0f) for intermediate signals and 1.0f(or slightly larger value) for master FX channel.

[zonkmachine]

I think it should be some appropriate larger level(ex. 4.0f~10.0f) for intermediate signal

Yes, I started out with 4.0f in the first version I pushed and in testing this 10.0f was about where the glitches would start to be too loud.

[zonkmachine]

A sample file may contain inf/NaN stuffs.

Interesting, I didn't think of this possibility. I went looking for some audio test suites containing various defective sound files for further testing but I couldn't find any. There are functions macros to generate problematic/borked floats like http://en.cppreference.com/w/cpp/numeric/math/nan so I may try to make some. Qt5 has functions to generate non-finite doubles but I haven't made the switch yet.

[PhysSong]

@zonkmachine I have a .wav file with NaN value. I can provide it if you want.

[zonkmachine]

Yes please!

[zonkmachine]

Changing base for the time being as my master branch isn't anywhere near sane.

[zonkmachine]

Anyway, Clipping signal at +/- 1.0f doesn't sound good for me. I think it should be some appropriate larger level(ex. 4.0f~10.0f) for intermediate signals

Changed to +/-4.0f

[PhysSong]

nan2.wav.zip

@zonkmachine Here is a .wav file full with NaN. I made it using hex editor.

[zonkmachine]

@zonkmachine Here is a .wav file full with NaN. I made it using hex editor.

Super! Can you fix one with infinites too? :)

As you suspected an audio file to Master will sneak past the checks. As will any sound that doesn't have an FX unit in it's path.

[zonkmachine]

As will any sound that doesn't have an FX unit in it's path.

Actually, sound via a mixer channel is catching NaN even without an FX unit. It's only direct to Master that's an issue.

[unfa]

I guess no one would want to use Infs/NaNs in a creative way. One might use a very loud intermediate signal to do something crazy and break effects in creative ways with that, but invalid float samples could probably be sanitized safely, without hurting creativity.

[zonkmachine]

One might use a very loud intermediate signal to do something crazy and break effects in creative ways

But how much louder? What levels do other studios, like Ardour and Bitwig, use for clipping?
With my hardware a 10.0 clipping level is a bit too audible for my taste when the glitches come. I think the VST specification use +/-1.0 for audio signals but I don't know if that also apply to hosts and if developers really stay within those levels.

[zonkmachine]

One might use a very loud intermediate signal to do something crazy and break effects in creative ways with that, but invalid float samples could probably be sanitized safely, without hurting creativity.

@unfa Here's the thing. The NaN's will cut the sound for a channel or even a whole mix. If you remove nan's/infinites, there will instead be an audible glitch in it's place where the sound is on it's way to/from infinitely high. This needs to be limited or it will give you a wall of noise when the glitch is followed by a reverb unit. Right now the suggested level for limiting is +/-4.0 . I've tried to trigger some of the glitches we've seen with ladspa pluggins in audacity and if they have a limit it seem to be much higher than that and I think we should err on being less generous in lmms.

Have you seen any similar glitches in other daw's? Ardour?

[zonkmachine]

Anyway, Clipping signal at +/- 1.0f doesn't sound good for me. I think it should be some appropriate larger level(ex. 4.0f~10.0f) for intermediate signals and 1.0f(or slightly larger value) for master FX channel.

Right now the suggested level for limiting is +/-4.0

As I've understood it, exporting will clip at +/-1.0 anyway so I think this should be the clipping level. I believe this is the signal level limit for VST's too.

A sample file may contain inf/NaN stuffs. In that case, this PR won't help to fix such issues.

I think this is a separate issue and should probably be fixed in SampleBuffer.cpp

[zonkmachine]

@zonkmachine Here is a .wav file full with NaN. I made it using hex editor.

OK. I made a project with nan2.wav

1. As a Sample Track playing at the very beginning.
2. A note playing at the 5th measure.

nanwav.mmp.zip

Without this fix the sound will cut out from when the nan ridden file starts sounding and you need to reload the project or bypass the reverb to get the sound back. With this fix the sound will cut out with the bad sound file playing but it will come back again as soon as the sample is done playing. So, a bit of an improvement but it still needs fixing on the sample side. SampleBuffer.cpp...

[zonkmachine]

Export issue: https://lmms.io/forum/viewtopic.php?f=7&t=27199
I think this should be merged. The last commit would take care of the issue above.

[tresf]

I don't have much to offer this because I don't know the DSP code well enough to understand what this will break.

My initial concerns were that stuff like C*CLIP would have problems, but it doesn't appear to. I'm sure this is simply a lack of understanding where sanitize is being called. Anyway, I did two tracks with BEFORE and AFTER... In both cases, the tracks play fine. Both tracks make heavy use of effects plugins and to the naked ear, I can't hear a difference.

There are slight visual differences, but I believe these are directly attributed to nondeterministic plugins and effects, which will create the differences regardless of this PR.

@zonkmachine I think we can merge this, but it would be nice to have a real-world track in the wild to unit-test against (rather than the one that as made intentionally to test this scenario).

(I know this is not what unfa-Spoken normally looks like, the track has some playback issues on Mac that are unrelated to this PR)

[zonkmachine]

I don't have much to offer this because I don't know the DSP code well enough to understand what this will break.

We already do this sanitizing but only on export. The only difference you'd see should be:

• No more NaN issues when not exporting.
• No more noises from feeding delay/reverb with extreme signals.
• Possible side effect from the forced clipping at +/-4.0 .

I think we can merge this

I'll go about it tomorrow.

[tresf]

We already do this sanitizing but only on export.

That makes sense. I thought the code was already there for export. Ok, yes, in that case we definitely need to merge this.

[PhysSong]

Possible side effect from the forced clipping at +/-4.0 .

It seems to cause clipping for one of our demo projects, Jousboxx - BuzzerBeater. I guess some channels in the project reach the level 5.0~6.0 and are affected.

[zonkmachine]

Jousboxx - BuzzerBeater

I don't know what to look for with this one as it's clipping with or without this PR. I exported as tracks and 11 out of 56 tracks was clipping but the rendered tracks are also clipped in the render process to +/-1.0 so it needs some more probing into which ones go above +/-4.0.
Do you think we should do something with this prior to rc6. Bump up the clip level to +/-10 ?

[Sawuare]

Since MixHelpers::sanitize() is called in EffectChain::processAudioBuffer() when an effect chain is enabled, I think effect chains should always be disabled when they are empty, to avoid unneeded calls to MixHelpers::sanitize().

Edit: A better approach would be checking for m_effects.isEmpty() in:

lmms/src/core/EffectChain.cpp

Line 201 in fce9326

if( m_enabledModel.value() == false )

[tresf]

The team is in talks about reverting this INF/NAN sanitization. I tested the example project in this bug report and was unable to reproduce the bug on nightly, however I'm still able to reproduce the bug with the project files from #1048.

Some arguments for removal:

• Performance. Quoting @LostRobotMusic

  These hundreds of thousands of checks done every single second inside of LMMS are being done to protect the program from NaN-generating plugins. I would first like to know if one of these plugins even exists in the first place. If it does, I would then like to test it in other programs to see how they handle the situation. It would be embarrassing if this whole time LMMS's infamously poor performance has been due to protections from plugins that don't even exist.

• Not how other DAWs behave. Quoting @LostRobotMusic

  I heavily doubt other DAWs are insane enough to sanitize anything

  Evidence to support this claim is provided by this PR from Audacity: which disables a plugin for NaNs/INFs rather than using sanitization Remove vocal reduction effect audacity/audacity#6147.

• Not-Our-Bug. Quoting @LostRobotMusic

  es. We should fix the issue by fixing the source of the infinitely loud nothing, rather than trying to cover it up (and ruining performance as a result).

So far, no measured evidence of performance loss has been provided. This should be quantified. Furthermore, despite both of these bug reports containing steps to reproduce, there was no attempt by @LostRobotMusic to quantify "plugins that don't even exist". A simple re-testing of #1048 shows that this problem is still very real.

I did do a quick look for the problematic plugin. My initial investigation shows zita-reverb as the likely cause. We don't ship this plugin with LMMS, so we don't (easily) have the ability to patch it. If someone can link a repository where this plugin is maintained, it may add merit -- or at least scope -- to the "plugins that don't even exist" claim (since patching it would support this argument by one less plugin).

Related: #1048 (comment)

[tresf]

nan2.wav.zip

@zonkmachine Here is a .wav file full with NaN. I made it using hex editor.

Audacity playhead just freezes with this file loaded, supporting @LostRobotMusic's point about other DAWs not sanitizing.

... however, Audacity also can be recovered by removing the problematic track, whereas LMMS requires a restart.

[LostRobotMusic]

the "plugins that don't even exist" claim

there was no attempt by @LostRobotMusic to quantify "plugins that don't even exist".

That is because I never once made the claim that they don't exist, and implying that I did is blatantly dishonest and deeply unprofessional. It's impossible to prove such a claim without showing every plugin that has ever been made, which obviously cannot be done.

Here's what I said in that same message:

I would first like to know if one of these plugins even exists in the first place. If it does, I would then like to test it in other programs to see how they handle the situation.

I said I would like an example of one of these plugins so it can be used in testing. I never once made the claim that they do not exist, though I did emphasize their overwhelming rarity and the fact that invalid output is solely the fault of the plugin. I also said that if the plugin doesn't work properly in LMMS due to NaN output, it won't work properly in other DAWs either, which is demonstrably true.

... however, Audacity also can be recovered by removing the problematic track, whereas LMMS requires a restart.

This is false, LMMS doesn't require a restart. The issue goes away once all of the offending plugins are removed or disabled, exactly like Audacity. The issue LMMS has to deal with is the fact that these NaNs can "infect" some future plugins in the signal chain, causing them to output NaNs just like the original offending plugin. It's for this reason that reloading the project file is oftentimes necessary. Restarting the entirety of LMMS is not, in any of the situations I've experienced.

[tresf]

It's impossible to prove such a claim without showing every plugin that has ever been made, which obviously cannot be done.

The absence of this bug was used as supporting argument for its removal, quoting (Discord)

For the record, I've had sanitization disabled in my personal-use build for over two years now and I haven't hit a single issue.

... when combined with targeted, speculative statements of performance, these statements often become persuasive in conversation, quoting: (Discord)

I think I was the one who originally said we need to remove the sanitation entirely.

Responses in support of this statement, or to #7323, of which this statement takes credit for the idea of:

• NaN could be annoying, but at the same time, it's not something we should be catching
• I agree with that removing sanitation but it should be thoroughly tested, so that we don't get any regressions.
• Reducing complexity will definitely help us in the long run.
• Remove the setting; the major benefit of this PR was optimization

...

...

This is false, LMMS doesn't require a restart. [...] It's for this reason that reloading the project file is oftentimes necessary

Thanks, this drastically speeds up testing.

The requested example has been provided. I did try to test this plugin in Audacity but I couldn't recreate the problem, likely due to incorrect parameters. Testing was taking a long time due to constants restarts of LMMS -- I understand now there was a quicker way to test -- while trying to isolate the parameters that cause the NaNs.

I won't be dedicating any more time to this myself, but I hope that narrowing down a problematic plugin can help the project decide if it's worthwhile.

Removal of this patch would be a detriment to projects which use these buggy plugins.

With regards to professionalism (either in Disord or on the bug tracker), I believe this is a conversation for another medium, perhaps Discussions.

[zonkmachine]

narrowing down a problematic plugin can help the project decide if it's worthwhile.

Lv2, swh, Dyson Compressor. Turn release to 0.