PulseAudio: Cork audio input when it isn't needed #171

Open
wants to merge 5 commits into
from

Conversation

Projects
None yet
9 participants
@cbs228

cbs228 commented Aug 12, 2013

As noted in https://sourceforge.net/p/mumble/bugs/868/ , mumble consumes a great deal of CPU when it is idle. The vast majority of this usage can be traced to the audio input DSP. When mumble is self-muted, these operations are largely unnecessary and unhelpful. To save CPU, the audio input stream can be "corked" (muted) when it is not needed. This blocks the audio input DSP when it isn't needed.

This update automatically corks the PulseAudio input when the user is self-muted. In order to do this, MainWindow emits a new signal, corkStream(bool), whenever the audio input is or is not required. This signal is slotted by the PulseAudio source, which passes it to the underlying PulseAudio server. This implementation will enable the fix to be ported to other sound servers and systems (i.e., Windows).

A downside to this fix is that the AEC will stop updating while Mumble is muted. If the user is truly idling or AFK, however, there probably won't be much sound with which to update it. While this patch appears to work, I would welcome more widespread testing.

@Natenom

This comment has been minimized.

Show comment Hide comment
@Natenom

Natenom Sep 4, 2013

Contributor

Thank you for this patch; I use it for two weeks now without problems.

Is it possible to also disable the PulseAudio source when using push to talk?

Contributor

Natenom commented Sep 4, 2013

Thank you for this patch; I use it for two weeks now without problems.

Is it possible to also disable the PulseAudio source when using push to talk?

@cbs228

This comment has been minimized.

Show comment Hide comment
@cbs228

cbs228 Sep 8, 2013

@Natenom: Muting the audio stops the echo canceler from updating. Such a patch, if implemented, would probably effectively break echo cancellation with PTT. Perhaps we can disable audio input for PTT if echo cancellation is also disabled.

I'm also not sure how long it takes PA to uncork a stream, and that might make the PTT "keyup" take a bit longer. If PA is slow to uncork, we might risk chopping off the first syllable of outgoing speech.

Thanks for testing this!

cbs228 commented Sep 8, 2013

@Natenom: Muting the audio stops the echo canceler from updating. Such a patch, if implemented, would probably effectively break echo cancellation with PTT. Perhaps we can disable audio input for PTT if echo cancellation is also disabled.

I'm also not sure how long it takes PA to uncork a stream, and that might make the PTT "keyup" take a bit longer. If PA is slow to uncork, we might risk chopping off the first syllable of outgoing speech.

Thanks for testing this!

@ngollan

This comment has been minimized.

Show comment Hide comment
@ngollan

ngollan Sep 11, 2013

Contributor

Echo cancellation should be mostly fine (AFAIK it's just acting on a delayed version of output, so it should only really "fail" for a few milliseconds). I'm more worried about noise filtering, which depends on constant updates to keep the noise power spectrum in order. So if noise changes significantly while the stream is inactive, e.g. from a printer starting to run, a window being opened/closed, or whatever else, there's a chance that things sound quite ugly after input resumes, possibly throwing off voice activation too. Having the client's audio processing always-on is (partly) to avoid that issue and always provide a cleaned input.

Contributor

ngollan commented Sep 11, 2013

Echo cancellation should be mostly fine (AFAIK it's just acting on a delayed version of output, so it should only really "fail" for a few milliseconds). I'm more worried about noise filtering, which depends on constant updates to keep the noise power spectrum in order. So if noise changes significantly while the stream is inactive, e.g. from a printer starting to run, a window being opened/closed, or whatever else, there's a chance that things sound quite ugly after input resumes, possibly throwing off voice activation too. Having the client's audio processing always-on is (partly) to avoid that issue and always provide a cleaned input.

@cbs228

This comment has been minimized.

Show comment Hide comment
@cbs228

cbs228 Sep 13, 2013

Interesting. The patch was not written with rapid toggling in mind: it was intended more for people who go AFK and idle, and it only kicks in when you self-mute. Some additional enhancements could include:

  • A timer, which will only turn off the audio when the user has been Self-Muted for so long
  • A preferences setting to turn this behavior on (or off)
  • A small delay when un-muting to allow the AGC to re-converge.

I've not observed any obnoxious behavior with the AGC while using this patch, but I tend to use PulseAudio's own echo canceler (AEC) and AGC, instead of Mumble's (see http://sourceforge.net/p/mumble/discussion/492607/thread/81d0b264/). The PulseAudio code has an input/output synchronizer and runs with real-time priority, and I suspect one or both of these things makes it work. I've been considering trying to patch Mumble to use PA's canceler directly, via filter.want=echo-cancel. That would, however, be a separate patch.

cbs228 commented Sep 13, 2013

Interesting. The patch was not written with rapid toggling in mind: it was intended more for people who go AFK and idle, and it only kicks in when you self-mute. Some additional enhancements could include:

  • A timer, which will only turn off the audio when the user has been Self-Muted for so long
  • A preferences setting to turn this behavior on (or off)
  • A small delay when un-muting to allow the AGC to re-converge.

I've not observed any obnoxious behavior with the AGC while using this patch, but I tend to use PulseAudio's own echo canceler (AEC) and AGC, instead of Mumble's (see http://sourceforge.net/p/mumble/discussion/492607/thread/81d0b264/). The PulseAudio code has an input/output synchronizer and runs with real-time priority, and I suspect one or both of these things makes it work. I've been considering trying to patch Mumble to use PA's canceler directly, via filter.want=echo-cancel. That would, however, be a separate patch.

@@ -55,6 +55,7 @@ bool CompletablePage::isComplete() const {
bInit = true;
bLastActive = false;
g.bInAudioWizard = true;
+ g.mw->onChangeMute();

This comment has been minimized.

Show comment Hide comment
@ei-grad

ei-grad Nov 3, 2013

Why there are 4 spaces instead of 2 as in previous line?

@ei-grad

ei-grad Nov 3, 2013

Why there are 4 spaces instead of 2 as in previous line?

@cbs228

This comment has been minimized.

Show comment Hide comment
@cbs228

cbs228 Nov 6, 2013

@ei-grad It's probably just what my text editor was set to. If the patch is something that the mumble team wants to include, I can re-format it.

cbs228 commented Nov 6, 2013

@ei-grad It's probably just what my text editor was set to. If the patch is something that the mumble team wants to include, I can re-format it.

@Kissaki

This comment has been minimized.

Show comment Hide comment
@Kissaki

Kissaki Dec 7, 2013

Member

We discussed this PR. We think that this should definitely be implemented - but for all audio systems (as mentioned in the newly created #1089 ) if noise suppression effectiveness is not noticeable.

This PR will be a good starting- and testing-base.

Thank you so far for your suggestion and implementation. We prioritized other PRs for 1.2.5 though, since this needs some more work and testing, so none of us will concretely work on this at this point in time.

Member

Kissaki commented Dec 7, 2013

We discussed this PR. We think that this should definitely be implemented - but for all audio systems (as mentioned in the newly created #1089 ) if noise suppression effectiveness is not noticeable.

This PR will be a good starting- and testing-base.

Thank you so far for your suggestion and implementation. We prioritized other PRs for 1.2.5 though, since this needs some more work and testing, so none of us will concretely work on this at this point in time.

@mkrautz

This comment has been minimized.

Show comment Hide comment
@mkrautz

mkrautz Dec 8, 2013

Member

Clarification: the mechanism which activates the "corking" should be compatible with all audio systems. That is, it should be implementable by subclassing AudioInput. Not all of them need to implement it, necessarily.

Member

mkrautz commented Dec 8, 2013

Clarification: the mechanism which activates the "corking" should be compatible with all audio systems. That is, it should be implementable by subclassing AudioInput. Not all of them need to implement it, necessarily.

@cbs228

This comment has been minimized.

Show comment Hide comment
@cbs228

cbs228 Dec 28, 2013

I definitely agree on the need to support all audio systems. I really only develop for linux, but please let me know if I can be of further assistance.

cbs228 commented Dec 28, 2013

I definitely agree on the need to support all audio systems. I really only develop for linux, but please let me know if I can be of further assistance.

@Vaskozl

This comment has been minimized.

Show comment Hide comment
@Vaskozl

Vaskozl May 23, 2014

I've been running this patched version for over a month now and I can confirm that it significantly decreases CPU usage while idling in a channel muted. This functionality should definitely be part of core mumble. I have not experienced any issues and it works marvellous.

Vaskozl commented May 23, 2014

I've been running this patched version for over a month now and I can confirm that it significantly decreases CPU usage while idling in a channel muted. This functionality should definitely be part of core mumble. I have not experienced any issues and it works marvellous.

@lkraav

This comment has been minimized.

Show comment Hide comment
@lkraav

lkraav Aug 2, 2015

Is the patch still usable for 1.2.10+? mumble is a no go here thus far, since it also hogs CPU in idle PTT mode. I see multiple issues trying to pinpoint a solution #1092 #1089 #1681 etc, but haven't been able to find out whether there's a solid solution to avoiding idle CPU usage.

lkraav commented Aug 2, 2015

Is the patch still usable for 1.2.10+? mumble is a no go here thus far, since it also hogs CPU in idle PTT mode. I see multiple issues trying to pinpoint a solution #1092 #1089 #1681 etc, but haven't been able to find out whether there's a solid solution to avoiding idle CPU usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment