Upcoming plug-in system and fairkom's translation feature

[mklampfer]

In our meeting with @ffdixon, @antobinary and @GuiLeme on March the 26th, we discussed the integration of fairkom's translation feature fairblue using the upcoming plug-in system.

It was suggested that we create an issue on github, outlining where fairblue uses and extends the BBB code. This would help determining how the plug-in system can be used to move fairblue out of BBB's core and maintain it as a separate plug-in.

This document provides a rough overview of fairblue and explains which code parts were added or changed.

Fairblue can be roughly divided into two parts:

1. One part deals with all the UI related stuff that provides interpreters the means for creating translation channels,
   and lets listeners choose which channel they want to listen to.

2. The other part provides the underlying logic for these UI elements and also is responsible for maintaining the
   additional freeswitch connections that are created on top of the regular ones.

1. UI elements

Fairblue's interface consists of

1. The translation section in the sidebar
2. Buttons in the actions bar

The translation section in the sidebar is meant for moderators/interpreters to create/delete translation channels before
the translation is started. The buttons in the actions bar allow to quickly change channels.

We have adapted the following BBB components to include our stuff:

• ActionsBar
• AudioControls
• SidebarContent
• UserList
• UserContent

We have also created some new components. You can see them at the detailed table at the end of this document.

2. Business logic and connection handling

HTML audio elements

Fairblue adds two html audio elements to main.html:

<audio id="translation-media" autoPlay="autoplay">
    <track kind="captions"/>
</audio>
<audio id="translator-media" autoPlay="autoplay">
    <track kind="captions"/>
</audio>

One is used for translators the other one for listeners.

TranslationManager

In the earlier versions of fairblue we had just added our code to BBB's AudioManager, but later decided to create a
separate file, TranslationManger. It deals with all translation-related logic and handles freeswitch connections.

Just like AudioManager, is uses create a new SIPBridge ('/imports/api/audio/client/bridge/sip') for each translation
channel. Multiple channels are kept open in the background, and whenever a speaker/listener chooses a certain channel,
it is hooked up to the corresponding html audio element that was created before (see above).

All changes, compared to vanilla BBB

We have made a few tables that show which parts of BBB were changed, where we included extra code and which parts of BBB
we import.

A quick explanation of the columns:

• added or modified: Shows whether we added a separate file or modified an existing file.
• Name: The name of the function/method/service/component etc.
• Description: a short explanation of that the code does
• Files: paths to the files that were edited
• Used imports: show which parts of BBB we hooked into and used in our code

Components

added or modified	Name	Description	Files	Used Imports
added	Language	displays languages in the sidebar	/imports/ui/components/translation/LanguageField	Icon (/imports/ui/components/common/icon/component)
added	NewLanguage	for adding new languages in the sidebar	/imports/ui/components/translation/NewLanguage
added	Translation	Translation section in the sidebar	/imports/ui/components/translation	* Meeting (/imports/ui/services/meeting) Button (/imports/ui/components/common/button/component) * layoutSelect, layoutSelectInput, layoutDispatch (/imports/ui/components/layout/context)
added	TranslationListItem	Outer sidebar section for toggling the translation sidebar child	/imports/ui/components/user-list/user-list-content/translation-list-item	* ACTIONS, PANELS (/imports/ui/components/layout/enums)
added	TranslationMixer	Translation volume control in the sidebar	/imports/ui/components/user-list/user-list-content/translation-mixer
added	TranslationDropdown	Dropdown Menu in ActionBar	/imports/ui/components/actions-bar/translation-dropdown/	* withModalMounter (/imports/ui/components/common/modal/service * withShortcutHelper (/imports/ui/components/shortcut-help/service * BBBMenu (/imports/ui/components/common/menu/component * Fab (@material-ui/core * Meeting (/imports/ui/services/meeting * Icon (/imports/ui/components/common/icon/component
added	LanguageOverlay		/imports/ui/components/LanguageOverlay/	* MeetingService (/imports/ui/services/meeting)
modified	ActionsBar	Adds Buttons to the actions bar. One chooses the speaker channel, the other one the listener channel. Also wraps some TranslationManager and AudioManager methods in the component service.	/imports/ui/components/actions-bar/component.jsx	* layoutSelectOutput, layoutDispatch (/imports/ui/components/layout/context) *BreakoutsHistory (/imports/api/breakouts-history) *isPollingEnabled (/imports/ui/services/features)
modified	AudioControls	Changes mute button depending on whether the user is translator or listener.	/imports/ui/components/audio/audio-controls	* Settings (/imports/ui/services/settings) * Storage (/imports/ui/services/storage/session) * setUserSelectedMicrophone, setUserSelectedListenOnly (/imports/ui/components/audio/audio-modal/service) * getFromUserSettings (/imports/ui/services/users-settings) * layoutSelect (/imports/ui/components/layout/context) * AppService (/imports/ui/components/app/service) * withModalMounter (/imports/ui/components/common/modal/service)
modified	SidebarContent	Adds sidebar panel for translations	/imports/ui/components/sidebar-content/
modified	UserList	Add check if user is moderator	/imports/ui/components/user-list/
modified	UserContent	Add translation items to the sidebar, Add checks for, * if a meeting has languages, * if translation is enabled, * if speech detection is enabled	/imports/ui/components/user-list/user-list-content

Methods

added or modified	Name	Description	Files	Used Imports
added	setLanguages	displays languages in the sidebar	/imports/api/meetings/server/methods/setLanguages.js	Meetings (/imports/api/meetings)
added	translatorSpeakStateChange	for adding new languages in the sidebar	/imports/api/meetings/server/methods/translatorSpeakStateChange.js	Meetings (/imports/api/meetings)
added	getParentMeeting		/imports/api/breakouts/server/methods/getParentMeeting.js	Meetings (/imports/api/meetings)
modified	breakout api methods		/imports/api/breakouts/server/methods.js	getParentMeeting
modified	meetings server methods		/imports/api/meetings/server/methods.js	setLanguages translatorSpeakStateChange
modified	addMeeting	add 'languages' property when creating new meetings in Meeting API	/imports/api/meetings/server/modifiers/addMeeting.js	setLanguages, translatorSpeakStateChange

Services

added or modified	Name	Description	Files	Used Imports
added	TranslationManager	provides all methods for controlling translation channels	/imports/ui/services/translation-manager/	*Meeting (/imports/ui/services/meeting) *SIPBridge(/imports/api/audio/client/bridge/sip) *Service (/imports/ui/components/actions-bar/service) *AudioManager (/imports/ui/services/audio-manager) *Auth (/imports/ui/services/auth)
added	MeetingService	provides methods for language related Meeting API calls	/imports/ui/services/meeting/	*Breakouts (/imports/api/breakouts) *Auth (/imports/ui/services/auth) *makeCall (/imports/ui/services/api) *Meetings (/imports/api/meetings/)
modified	AudioManager	Adds some connection handling logic	/imports/ui/services/audio-manager/	*Breakouts (/imports/api/breakouts) *Auth (/imports/ui/services/auth) *makeCall (/imports/ui/services/api) *Meetings (/imports/api/meetings/)
modified	SIPSession	add media tags for translation	/imports/api/audio/client/bridge/sip.js	* getCurrentAudioSessionNumber, getAudioSessionNumber, getAudioConstraints, filterSupportedConstraints (/imports/api/audio/client/bridge/service)
modified	Audio Components Service	Adds some methods to the audio components service, initializing TranslationManager etc.	/imports/ui/components/audio/service.js	* getCurrentAudioSessionNumber, getAudioSessionNumber, getAudioConstraints, filterSupportedConstraints (/imports/api/audio/client/bridge/service)

Assets

added or modified	Name	Description	Files	Used Imports
added	de.json	german UI strings	/private/locales/de.json
added	en.json	english UI strings	/private/locales/de.json
added	bbb_languages_icon.svg	icon	/public/svgs/bbb_languages_icon.svg
added	bbb_translations_icon.svg	icon	/public/svgs/bbb_translations_icon.svg
added	translations.svg	icon	/public/svgs/translations.svg
added	translations.svg	icon	/public/svgs/translations.svg
modified	added translation icon	css	/client/stylesheets/bbb-icons.css

3rd party libs

added or modified	Name	Description	Files	Used Imports
added	harbor.js	Used for positioning UI items	/public/compatibility/harbor.js
added	hark.bundle.js	Used for checking if someone is speaking	/public/compatibility/hark.bundle.js
added	rxjs	Reactive extensions, used for meeting status updates in the UI

Config

added or modified	Name	Description	Files	Used Imports
modified	settings.yml	added translation related settings	/private/config/settings.yml
modified		Adds panel designations	/imports/ui/components/layout/enums.js

Other

added or modified	Name	Description	Files	Used Imports
modified	main.html	added imports for harbor & hark, added two html audio elements which are used for translation, one for listening and one for speaking	/client/main.html
modified	main.jsx	added key event listeners for muting	/client/main.jsx

[GuiLeme]

Hello guys, thanks for proposing this plugin!

As spoken earlier this week, here are some details for the development:

• So for the ActionsBar as you mentioned, we have a way to add a button and a separator in it. Refer back to the sample plugin for this, and also, you'll want to look for the extensible area actions bar item;
• For the AudioControls we have a way of adding a option to the audio settings dropdown. See the sample for it, and the extensible area you are looking for is this;
• For the SidebarContent we still don't really have it, but it is on the roadmap for the next days, this applies too for the UserList, because as we spoke, those items on the userList area are just some tabs for the sidekick panel or the sidebar content as mentioned before;
• For UserContent, we already have two things: One that can add an information to the userlist item itself as demonstrated in here and that extensible area would be UserListItemAdditionalInformation, and the other one which has this sample;

That's for changing the UI. I also think you will make good use of the useDataChannel to exchange information from each plugin (to make everyone add the sidekick tab for each translation channel, for instance...), see the code for that here and the sample here;

Just for the recors, all links here are in the SDK repository, and of course, refer to the main readme for more info.

If you guys have any doubts, please ping me and we can figure it out together! Thanks again!

[GuiLeme]

Hello, guys, there have been some time since the last comment about this thread, so I will update you with some things we've been doing in this direction: there are some PRs open, this one right here: #20272 which lays the ground for the following two, that are:

• Typed captions support bigbluebutton-html-plugin-sdk#83
• feat(plugin): added a couple of features to support typed-captions migration #20292

These last 2 PRs, among other things, build the sidekick panel that will be necessary for this feature.

Note: they are already in the way for being reviewed and merged. Sorry for the late response, and if anything comes up, I will let you know through this channel! Ping me, if any doubts.

[GuiLeme]

Hello, guys, thanks for being patient. This is the PR in which the sidekick panel entered for the SDK: bigbluebutton/bigbluebutton-html-plugin-sdk#88. Now with that I think this issue can be done normally, so if you need any help from me or bump into some unexpected problem, please let me know.

[antobinary]

@mklampfer please confirm we can close this issue.