libobs/UI: Master mixer & input monitoring

[pkviet]

Authors: Andersama, pkv, Shaolin
(warning: the screen captures may not be up to date due to changes in icons, check the latest comments)

covers:
https://ideas.obsproject.com/ideas/170/output-volume-visualization
https://ideas.obsproject.com/posts/472/vu-meter-and-fader-for-master-audio-outs
https://ideas.obsproject.com/posts/398/if-you-have-monitoring-enabled-there-should-be-a-separate-mute-button-for-it

This PR has 7 commits.
The first two add a master mixer dock (displaying the levels of the 6 output tracks). Hidden on default.

The next two add a monitor button to input mixer (the current mixer).
The monitoring was previously triggered from the Advanced Audio panel; the button
is much more convenient.
The monitoring API couples monitoring and output through three states:

• Monitoring and Output,
• Monitoring Only (no output),
• Monitoring OFF.

In terms of UI this creates a challenge because monitoring UX in most software is achieved
with a two state button: monitoring ON or OFF.
To keep a two state button we rely on the tracks to which is assigned an audio source. If the source is routed to no track, monitoring is switched to state 'Monitoring Only' while if the source is routed to any track monitoring is set to 'Monitoring and Output'.

The reasons we followed different options to implement monitoring than in PR 1253 are
explained in detail below (Heading (2) at the end of this text).

The next commit adds tracks buttons to input meters for convenience; this allows
selection of the tracks for each source without having to open the Advanced Audio Properties.
(The track buttons are hidden on default to avoid clutter for new users.)

The last two commits allow a SOLO mode for monitoring tracks (following a suggestion by cg2121).

NB: the PR initially had hotkey support for monitoring; it 's been removed per Jim's request.

Why a master mixer?

First and foremost to control the audio levels which are going into the streams and recordings.
For instance, in the next screenshot, one can see that the sources are not clipping although the total audio level of the tracks is in the red area (source mixer is on the left; the master mixer is on the right):

One might be clipping without knowing it.

Another example of use is with sources whose audio is being buffered to the max: such
sources still show audio on the regular mixer. But there is no audio from such a source
on the stream nor on the recording (btw this is what happened to this user)
Incidentally the master-mixer helped us detect a bug with 'Monitoring Only' state.
(see Mantis issue here )

)

Some screenshots:

The master mixer is hidden on default and is toggled through the View Menu > Docks .
This is to keep a simpler UI for new users.
(1) master mixer shows track 1 when output is set to Simple in Settings. The headphone
button is for monitoring the track.

The track can be renamed, hidden unhidden, and be filtered as well as monitored.
The mon button supports hotkey (feature removed). The monitoring is POST-FADER, POST-MUTE and POST-FX.

(2) filters can be used on tracks from the master mixer. Here a negative gain is applied.

(3) master mixer shows all tracks when output mode is Advanced; two new buttons appear :
a green stream button (antenna icon) and a REC button (red disk).

• The stream button (antenna) is a toggle to select the track used for streaming output,
• while the red round button selects tracks used for recording in Advanced mode.
  Any track can be hidden so that the end-user can display only what is useful to him/her.

The buttons are synced dynamically to the tracks selected in Settings > Output.

(4) input mixer has 2 sets of new buttons; a monitoring button + 6 tracks buttons.

• The tracks buttons have the same functionality as the Track checkboxes in Advanced
  Audio Properties. They are HIDDEN on default and enabled through context menu.
• audio is sent to output tracks only if the corresponding track buttons are switched on (either on the mixer or in the advanced Audio Panel)

See this suggestion : https://ideas.obsproject.com/posts/398/if-you-have-monitoring-enabled-there-should-be-a-separate-mute-button-for-it
It is asked for a way to have monitoring and toggle on/off the routing to tracks.

The PR was tested:

• on all themes currently supported,

• on all supported OS

Here are details on what was implemented.

(1) master mixer

UI/libobs: Add Master Mixer

This adds a new dock with vue-meters and faders for all tracks, which
will allow to monitor clippings for outputs as well as change their
volume.
A particular attention has been given to UX :

• capability to apply filters to each track;
• capability to monitor each track;
• hotkeys to mute/unmute, monitor/unmonitor; (removed at this stage to simplify code)
• additional buttons to enable/disable tracks for streaming/record;
  (as a shortcut to Settings > Output)
  (while keeping the other buttons of source mixer - mute, advanced audio
  properties, hide, unhide)
• display of master mixer is synced to the output mode (simple/advanced)
  (in simple mode, only track 1 is shown)
• to unclutter front-end for beginners, the master mixer is hidden on
  default settings.
• master mixer is vertical on default to be closer to customary display
  in most DAWs.

See : https://ideas.obsproject.com/ideas/170/output-volume-visualization
https://obsproject.com/forum/threads/master-audio-level-meters.78608/#post-332745
Closes https://obsproject.com/mantis/view.php?id=405

Thanks to SuslikV, cg2121 and WizardCM for feedback.

(2) monitor for input mixer

libobs/UI: Add source monitoring in mixer

• This commit requires the master mixer commit to work.
  The latter added monitoring to output Tracks.
  The changes allow easy extension to support monitoring from the mixer
  UI.

• The implementation differs from that proposed in PR 1253. The main
  difficulty is the coupling by the API of monitoring to output. The
  monitoring of an input audio source is set by source->monitoring_type
  which can take three values (no monitoring, monitoring and output,
  monitoring but no output).
  Instead of having the user set directly one of the monitoring states,
  we rely on tracks settings to set them.
  The ned-user only sees a monitoring button.

• If the mon button is switched off, monitoring is off.

• If the mon button is switched on, and if any track is selected, the source is
  in the OBS_MONITORING_TYPE_MONITOR_AND_OUTPUT state.

• if the mon button is switched on, and if no track is selected, the source is
  in the OBS_MONITORING_TYPE_MONITOR_ONLY state.

• PR 1253 uses the mute button to get a second bool for such mapping.

• Our approach differs in making the monitoring tap be fully post-fader
  and post-mute while PR 1253 is post-fader but not post-mute.
  In terms of UX we believe it can be confusing for new users to handle a
  'post-fader, not post-mute' monitoring for which muting has no effect on the
  levels shown on the meter when monitoring is enabled.
  Without any ability to check meters for output tracks a new user might
  press mute, see no effect on the input source meter and be baffled when
  checking his/her recording.

The commit has been tested on windows 10 (pro 1810), linux, macOS
(10.13.6).

(3) tracks buttons for input mixer

Adds tracks buttons to input mixer as a convenience.
Avoids having to open Advanced Audio Properties while having a direct display
of the routing.
(Suggestion made by SuslikV)

(4) Solo monitoring for tracks
When one track is selected for monitoring, the monitoring of sources and other tracks is
disabled.
Conversely if a source monitoring is switched on, the monitoring of tracks is toggled off.
(Suggestion by cg2121)

Note :
During the development of monitoring for inputs we uncovered a bug at then git head.
(see Mantis ).
PR 1522 fixes completely that issue though. This shows however how master mixer can prove useful.

[SuslikV]

It is not visible (from UI) which source goes to which track. Only visible how many tracks/sources is active right now.

Edit: For example, I cannot understand is the VLC Video Source goes only to Track 1 or it goes to all 6 tracks simultaneously.

[pkviet]

right , we thought of adding checkboxes to source mixer to indicate to which tracks the sources are routed. But we've been told that the UI is too cramped as it is ... but we agree this would be useful.

[pkviet]

Update: 10/06/18

• changed the icons;
• changes due to ffmpeg output supporting multi-track.

[pkviet]

Update: 10/21:18
Following SuslikV suggestion, we've added tracks buttons. They are more convenient in our opinion than in the Advanced Audio Properties. This simplifies the routing setup.

[ghost]

Only thing that might need improvement is the streamON/OFF.png the icon is horizontally squashed/vertically stretched. Also not sure if its ok to use a modified version of the cast-icon

Also the naming seems to be inconsistent, one thing is called sendsON/OFF, the other streamON/OFF
I'd suggest sticking with send/stream or sends/streams

There's also a little inconsistency with the UI States, if you compare it to the sourcevisibility or to the source locking, the active state is 100% white and the other state is a different icon with (i suppose) 50% transparency. This is important for theme compatibility since if someone had a green theme the active state of the buttons would not be visible.
So i would suggest making the active state white, and the inactive state maybe the same icon but with a diagonal line through it and transparency (not sure how the other features have implemented this but i would try and keep it the same everywhere).

I hope this gets merged ASAP, since this is really an essential feature. If i can be of any help with the icons etc. i'd be glad to help.

[pkviet]

sends could be send but i've put plural because it's sending a source audio to several tracks (to several sends, , same term as in a DAW) . Anyway no big deal.
As to the icons, I must say that design is not my thing. Feel free to create new set of icons ; you can contact me on discord to drop them.
Be sure though to check against the four themes; there might be different sets of icons for each of the current themes.
As to merge time frame, I doubt this will be merged any time soon. There are higher priority features. You can always upvote the feature on fider:
HERE

[ghost]

How can i contact you on discord?

[pkviet]

go to obs server, #development , ping me there (@pkv )

[SuslikV]

Making icons similar to Source's list custom rendering is bad idea - there was used painting based on Palette and Roles while whole application uses Style Sheets. It always was pain in the _ to make any new themes because of this.

[pkviet]

@SuslikV i'm not knowledgeable enough with qt styling to really understand your suggestions, not your fault, just my ignorance. Feel free to make a PR to my branch so that I understand better what you mean, I'm really open to any improvements

[SuslikV]

@pkviet it was just a comment to the last suggestion from @ivan2340 to make icons similar to Source's list rendering.
This custom painting, that were mentioned above - it was nice trick that shows that programmer can use some technique, but it worsens all other stuff. For example, next change to the sources/tree list, that adds colored background to the sources, unintentionally removes Styles for all underlying widgets (including QEdit, that is placed over, only when user renames the source) and this was bad. Now, its color only depends on Palette setting that is set as base option in OBSTheme section of the .qss file. And this (OBSTheme) is another artifact that was developed for another reason.

[WizardCM]

I've been testing this PR myself in streams over the past few weeks, and here's what I've seen:

• functionally, this is flawless. all the buttons do what they're supposed to, no complaints on that front
• I feel that the sends/monitoring buttons are a really good idea (and should be merged), but I'd probably remove the tracks buttons only because visually it becomes very noisy. I can see the benefit, and maybe this'd be better suited to a future "advanced" UI of some sorts (or a toggle), as I feel having to open the Edit Advanced Audio Properties window for specific things isn't fun, but I feel this capability shouldn't be merged together with the master mixer as it'll be a big jump
• the master mixer dock is fantastic for seeing if there is audio buffering - i've had situations where my audio got delayed by a media source by 500ms and it's immediately visible side-by-side. means i can immediately check the log to confirm
• The stream button in the master mixer seems to be stretched in vertical mode, where it looks fine in horizontal mode (or the other way around - not sure).
• I haven't run into any audio issues or any stability issues with this PR.

Overall, love this.

[DDRBoxman]

[pkviet]

update 02/03/19

• hide the tracks buttons (of input mixer) on default.
• the tracks buttons are displayed through context menu of the mixer.
  (see @WizardCM comment)

[Himura2la]

It looks great and feels just like common mixer interface of major audio software and hardware. The current Tracks implementation in OBSS is very non-intuitive unlike this clear division on inputs and outputs with the individual routing of every output. The one thing that I'm not sure about is a separate button for recording. I guess in 99.9% cases the streaming state should equal to the recording state... This division may lead to human mistakes.

[pkviet]

Answering Dodgepong for the sake of clarity: When Jim and I discussed closing this PR, Jim didn't fault us with not having a spec phase, nor with not having incremental code, whatever this may mean. Either the master mixer is coded or not. The feature itself is separated in the first two commits anyway.
He did not put the onus on us. As to me, my motivation was to relieve Jim by closing the PR. He's the single full time dev and his time is very constrained.
@axd1967 filtering indeed works on master mixer. I use it with a limiter to protect the final mix from clipping. Note that you can use my fork as an alternative.

[matiaspl]

If OBS is to compete with commercial products then sooner or later audio workflow will have to be fixed. The audio subsystem seems to me as the enfant terrible of OBS. Video visibly gets much more love.

Is there anything we could do to relieve @jp9000 and put master mixer back in business?

[dodgepong]

I agree that the audio subsystem could use improvement, but for the sake of argument here, what would be the motivation for the developers of OBS to compete with commercial products beyond meeting user demand?

[ghost]

I feel like Audio Mixing is by far one of the most lacking features of obs. I've been using my own build with this PR merged for ages, and it's by far one of the best changes we could get. Especially now that there are so many people creating live music on twitch & Co.

[ryantheleach]

@pkviet What would be the minimum amount of changes needed to OBS to implement most of this as a plugin? Would that be more acceptable to OBS dev team?

[matiaspl]

I used the term "commercial products" as a figure of speech. "Something done right" (even though we all know it doesn't necessarily work that way...). For some "doing something right" is good enough motivation.

A good video mixer in OBS studio mode lacks a good audio mixer. Introducing studio mode created a lot of workflow inconsistencies

• It's not possible to replace the current audio mixer with external solutions because there's no way of routing audio sources originating from OBS to the outside world.
• In studio mode you can preview all sorts of video sources that aren't on program, but you can't do it for audio.
• You can do group and scene filtering but only for video. Based on this you can create auxiliary buses (e.g. NDI filter allows this) for specific input monitoring. If you wanted to do mix-minus audio for communication with remote sides you can't do that.

Reworking audio as proposed by @pkviet would be a first step in the right direction to create a full program/preview (or A/B) mixer for both audio and video, where WYGIWIG (and WYHIWYG ;) ) works for both video and audio.

[axd1967]

a minimum for monitoring the input level is to use the system that is coded in the Rachni scheme: there, one can at least see when there is clipping. in the other schemes this is not possible.

the two feeble dots at the extreme left indicate input clipping, because this was a clever choice in that scheme: it uses a discontinuous color change.

this is the sole reason I use the Rachni scheme; the fonts are too small, but at least i see if there is a problem with the source.

[vannomad]

Sadly this is not a workaround since two sources that do not clip seperately might still be clipping on the output.

[axd1967]

you are right, but one should then keep an eye on the individual levels (an additional workaround), reduce them all sufficiently; now it is difficult to even see if there is input clipping.

[pkviet]

@axd1967 it should be a simple matter to modify the Dark theme to have the same as in Rachni. They are css files basically: check this commit to know what to change in other Themes pkviet@5576823#diff-9a0c4b0f14e5a443507219515f7e3323
The themes are the .qss files in here: C:\Program Files\obs-studio\data\obs-studio\themes

[axd1967]

sorry, but it is the average end user that needs this feature, not only the few that are able to access source code. if I find time I can do this but I don't have the time. if you know what I mean, this should be a very easy fix, so can you do this? all that is required is to ensure that the clip colors can be distiguished from the high volume colors (in all themes).
(thx)

[ryantheleach]

I found the input clipping colors were different on OBS already.?

If they are green, they blend a bit, in which case the input audio is great right?

Furthermore, I fail to see how it is relevant to this (closed) PR

[axd1967]

i tend to forget the issue (that is unrelated to this issue...): in the default theme it is hard to distinguish the yellow (intermediate level) color from white (clipping indication). whoever reduced that indicator to a few pixels has no understanding of human vision and/or the value of that indication. in Rachni there is at least no yellow but an orangeish color, which makes the white clipping color stand out unambiguously. that is the problem: there is too much similarity between yellow and white, but how can I make myself more clear than that?

[bbbradsmith]

When an input clips in OBS it also turns completely red temporarily, does it not?

The much bigger problem is there's no meter for master output, so there's no way to see the volume of the resulting mix, clipping or no.

[axd1967]

When an input clips in OBS it also turns completely red temporarily, does it not?

The much bigger problem is there's no meter for master output, so there's no way to see the volume of the resulting mix, clipping or no.

AFAIK not! the red bar is the result of output going over the top, but you can have input clipping without that too.

and yes, there is (or was, after this PR?) no overall output metering. I'm curious to see the result.

[ryantheleach]

I'm worried about off-topic chatter getting this conversation locked indefinitely, making this PR effectively dead in the water more then it already is. I'd suggest you take those complaints either into suggestions, issues, or create a PR fixing those specifically.

[matiaspl]

Just pinging @jp9000 and @WizardCM about the future plans on the audio subsystem in OBS. Is there any room for audio improvements in the roadplan?

[WizardCM]

As far as I know, there are currently no plans to revisit this. It might be up to @pkviet to potentially put together an RFC and discuss with Jim about where improvements could be made, in stages.

[ryantheleach]

I know this is closed, but if further work happens here, I'd like to show Elgato's Wave Mixer as a comparison to the design work done above.

Notably, they use

1. Ears to represent the Monitor toggle.
2. Antenna to represent the Stream output device,
3. Headphones to represent the Monitor Device.
4. Speaker icons to represent mute, as is tradition.

Additionally, this software was released to public on the 18 June 2020 I can't help but wonder if they read our discussion, or came to this independently based on similar trains of thought.

[pkviet]

Interesting. Independent levels for monitoring and output. I have some code for prefilter and post filter but haven't thought of monitoring volume.

…

On Thu, Nov 26, 2020, 21:11 Ryan Leach ***@***.***> wrote: I know this is closed, but if further work happens here, I'd like to show Elgato's Wave Mixer as a comparison to the design work done above. [image: image] <https://user-images.githubusercontent.com/141554/100388712-4d3ebd00-307b-11eb-9a4b-ec9faf64ac3b.png> Notably, they use 1. Ears to represent the Monitor toggle. 2. Antenna to represent the Stream output device, 3. Headphones to represent the Monitor Device. 4. Speaker icons to represent mute, as is tradition. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1469 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACG2OTYEWOCJVUC2R7AN5RDSR2Y75ANCNFSM4FTCK63Q> .

[WizardCM]

It's worth noting that the main difference is that the Wave Link software is purely audio-focused, whereas OBS isn't. I have seen requests for being able to control monitoring volume, but I think for OBS it'd only make sense as a single global slider.

[Himura2la]

Wave Link software is purely audio-focused, whereas OBS isn't

Totally disagree! Any serious streaming setup requires complex audio management, and its UX must be perfect. Audio is a huge part of Video, good sound is even more important than good video.

As for separate sliders for monitoring or a global monitoring level, I think we don't need any of them, because

1. Monitoring should reflect what people hear on the stream, but separate sliders can cause mistakes when streamer think that the sound is fine, but it's not.
2. Global monitoring level can be controlled using the system level of monitoring device (or even a physical knob on usb sudiointerfaces)

[ryantheleach]

As for separate sliders for monitoring or a global monitoring level, I think we don't need any of them, because

I think you are misunderstanding Elgato's design intent here.

The headphone output may be the monitor device, but it's also possible to swap to monitoring what the STREAM hears.

The wave mixer is a complete intermediary mixer for the system AND the stream, instead of a sub-mixer for the stream.

So for THOSE reasons, I don't think it's paramount that OBS have separate global monitor volumes.

The main usecase that people have of (permanently) monitoring OBS sources, is that they are using media sources / videos inside OBS that they can't hear, that they need to cue off of.

THIS is why we potentially need separate 'monitor' sliders. But in reality, it's not really 'monitor' in the purest sense, but creating a sub-mix for the stream operator. In addition to this, it needs to be possible to monitor the master mixes output to each of OBS's outputs.

Whether we call these source sub-mix sliders monitor or return or some other term I'm not sure on, but they are currently called 'monitor' in the OBS UI.

[pkviet]

@ryantheleach what you say can be achieved by dedicating a track to monitoring; i'll think about implementing it. Will be quite handy

[ryantheleach]

That would probably be the cleanest method of muting all monitored mixes when switching to monitoring master etc as well. I still think we are possibly using incorrect terminology, and shouldn't be using 'monitor' to describe this track though.

[eazuka]

@pkvie, you are great. this PR is amazingly awesome.
It is such a shame that this PR isn't being merged and closed.

I am really hoping there will be some willingness by the core team and @pkviet to resolve the blockers and get this feature/PR merged.

[RiyonLive]

I've been testing this PR myself in streams over the past few weeks, and here's what I've seen:

• functionally, this is flawless. all the buttons do what they're supposed to, no complaints on that front
• I feel that the sends/monitoring buttons are a really good idea (and should be merged), but I'd probably remove the tracks buttons only because visually it becomes very noisy. I can see the benefit, and maybe this'd be better suited to a future "advanced" UI of some sorts (or a toggle), as I feel having to open the Edit Advanced Audio Properties window for specific things isn't fun, but I feel this capability shouldn't be merged together with the master mixer as it'll be a big jump
• the master mixer dock is fantastic for seeing if there is audio buffering - i've had situations where my audio got delayed by a media source by 500ms and it's immediately visible side-by-side. means i can immediately check the log to confirm
• The stream button in the master mixer seems to be stretched in vertical mode, where it looks fine in horizontal mode (or the other way around - not sure).
• I haven't run into any audio issues or any stability issues with this PR.

Overall, love this.

how do you install this?

[RiyonLive]

Can anyone let me know how to install this on my obs. i don't know much about github and projects. was looking for a installation method but couldn't find one. badly need this.

[Fenrirthviti]

You can't just install it, this is core changes to the program which require it being recompiled to work. There's a good chance these changes are no longer compatible, or would need updating to work with the current version.

I'd also like to reiterate for those still following this or anyone who comes across this in the future, this was not closed because the changes were rejected or because we don't want to make audio controls more flexible. This was closed because the changes were too large for us to realistically review at the time, and we wanted them broken up in to smaller pieces which would make discussion and implementation/design feedback a lot easier. The original author just has not had the time. See: #1469 (comment)