2022 Spring Endsemester Concert MCT Portal and Science Library
At the Science Library stage, we had 3 performers with a stereo setup each, a speaker mic, and a talkback mic for communicating with the Portal only, in addition to the network audio. The audio routing for the LoLa rack used in the Science Library was based on the user guide found at this Wiki-page. We had some special demands for our setup which made us change the routing in some ways, as explained below.
Input list
1. Joseph Drum Pad (L)
2. Joseph Drum Pad (R)
3. Kristian Rig (L)
4. Kristian Rig (R)
5. Joachim Rig (L)
6. Joachim Rig (R)
7. Speaker Mic
8. Talkback Mic
11. Network Audio (L)
12. Network Audio (R)
Routing of the audio from the Portal
Since both Joachim and Kristian needed to individually get the audio from the Portal into their respective setups, the group outputs were used for this. Group output 1-2 where used to send network audio to Kristian, and group output 3-4 where used to send network audio to Joachim. Kristians rig played along with the network audio in the first part, while Joachims rig processed the network audio with the network audio channels muted in the second part. The soundcard in Joachims rig, however, had a knob for crossfading between the “dry” (network audio) and “wet” (processed audio) signals.
Monitoring
We used the PFL (pre-fader listen) buttons for monitoring, and connected the monitor wedge to the [Monitor Out] jack. This gave us no control of individual signal levels, but made us quickly able to punch in and out channels from the monitor instead of using the AUX 3-4 knobs. Since the first part of our concert needed network audio sent to the monitor and stereo out, but the second part shouldn't have any original network audio in the monitor or stereo out (just the processed version from Joachims rig), this was a nice alternative for our setting.
Gain control of network audio
To have more hands-on control of the incoming network signal, we changed from the suggested channels 15-16 to channels 11-12. We did this because stereo channels from 9-12 have gain knobs, which the channels from 13-16 don’t have. The signal from the network could be a bit too hot, so it was necessary to be able to reduce the gain. (This gain reduction could’ve been done on the soundcard, or even in the Portal end, but it was a good feature to have more hands-on control on the mixer.)
Audio
Audio routing was a relatively simple process. There were only three inputs, the guitar for performance 1, the output of the sonification computer for performance 2, and the talkback microphone for communicating with the Science Library. These were routed as inputs to the LOLA rack. The stereo output of the LOLA rack, including the received audio from the Science Library, were routed to the MIDAS mixer in order to enable playback over the room speaker array.
Video
Video requirements for performance 1 was simple. The received LOLA video was simply to be displayed on the performer's monitor and on the NEC TVs. However, performance 2 had slightly more complicated requirements. The output of the performance display computer was required to be routed to the NEC TVs for the performers physically present with the mocap system. As a result, the video feed received over LOLA was to be displayed on the Sony TV. In order to meet the different needs of each performance, all video, both local and received, were routed over the HDMI matrix. This allowed a quick change in routing between the performances. For more information on the method through which two video feeds were streamed between locations, see the section Video routing for audience.
Routing Diagram
The setup for our first part was based around the core concept of using gestural control to manipulate audio in a telematic performance, as explained in this blogpost. This setup was based around Kristians extended keyboard instrument using a Myo armband to process and synthesize audio through a modular synthesizer environment, in which the inner patchwork is explained in this blogpost.
The only change in routing with regard to Kristians original extended keyboard design is that the network audio was sent, through the mixers group output 1-2, to his keyboard input jacks. This made it possible to manipulate both the network and keyboard audio in a straight-forward way.
The set up for the second part of the concert mainly focus on generating and processing audio data in real-time while interacting with two types of program in two different locations.
Motion Controlled Interactive System
The motion controlled interactive system allows the users to configure shapes and sounds through the placement and motion of rigid bodies with the Portal's OptiTrak Motive system. Read more about it here.
Gestural Interactions for Multi-parameter Audio Control
To recognize hand gestures, we use the Python library Mediapipe, a simple way to build world-class ML solutions and applications. MediaPipe Hands is a high-fidelity hand and finger tracking solution. It employs machine learning (ML) to infer 21 3D landmarks of a hand from just a single frame. That way, the user can use the index finger to trigger audio effects. Moreover, using Open Sound Control server and client implementations in Python, this makes it possible to send UDP packets from Python to Pure Data, to finally output audio from the soundcard of the computer. The hand gestures are thus used as controllers for the multi-parameter audio effects (pitch, reverb, delay and modulation) in Pure Data.
Code and instructions for setting up the program can be found here
The Motion Controlled Interactive System had a graphical interface to display what was going on. This system was located in the portal at Zeb, but the audience was located at Vilhelm Bjerknes hus at the library. We wanted to display the graphical output of the interactive system at ZEB for the audience at the library. This meant that we had to show both the live camera-feed and a screen-share at the same time. The problem with this is that the NMP-kits which displays the camera-feed cant run any other program than LOLA, meaning we had to display two video signals at one screen at the same time.
Using the screen-share function of Zoom, the interactive music system shared their screen over the internet. This could be mixed together with the HDMI-signal from the NMP-kit going to the projector, using the Blackmagic ATEM mini’s picture-in-picture function. This was the best solution that we were able to do for this concert. This made the camera-feed from ZEB still able to have a low latency for the audience, but the screen-share from the Zoom-feed did not offer low latency and high framerate, resulting in a sporadic feed for the audience. This worked well enough for what we wanted to do, but a better solution should be made which is more stable.
To document the process, start by defining reasons for documenting. This can be for entertainment value, educational value, marketing, or any other reason you could think of. In our case, it was mostly to provide an insightful walkthrough of the systems used for the concert, and expose the thought process of those involved to fuel your ideas as to what can be done for your next telematic music concert!
Next, consider the documentation format, where it will be accessible, and why. A downloadable, instructional pdf? An instagram image series? We ended up making a video, as we were not planning to use this behind-the-scenes material for promotional purposes, and breaking down these systems through dialogue would be quicker than designing a collage. Still, expect at least 2-3 hours of editing. We used a basic camcorder, simply because it was what we had available without having to fix up any neglected equipment. If you have people available, get help recording audio separately. Tuning audio levels with a camcorder is difficult, and we had to throw away some good footage due to severely clipping audio. We also recommend adding some background music, maybe even from the concert if you record it. It will make silent shots that much less awkward.
Finally, plan out some prompts for participants that will get them talking. Good questions start with how and why, and will often provide the most insightful responses in turn. If you’re thinking of doing a vlog-style video, consider narrating the video a bit more than we did as well, so your audience knows what’s happening and why.
Video link: https://drive.google.com/file/d/18Sa-wyyH4_hXrqppESTm-6xii_SlCxhE/view?usp=sharing
If you ever need to get a hold of the people in charge of the IMV’s or the Science Library’s social media accounts, here’s their info:
- Linn Katrine Hirsti (Science Library, Facebook & Instagram): l.k.hirsti@ub.uio.no
- Julie Øderud Danielsen (IMV, Facebook): j.o.danielsen@imv.uio.no
In order to post a promotional Instagram story, it’s important to take the following factors into account:
- Whether you post a photo or a video, it has to be clear what the content is depicting. Make sure that the visual content is not blurry and that people can understand what is going on without a lengthy explanation.
- Add relevant information without crowding the screen with text, so as to avoid covering the other visual content. A good plan is to stick to time, location and name of the event (ideally, the name would be self-explanatory).
- Use different colors for the text to emphasize certain words and to make sure that the text is visible in contrast with the other visual content. Also choose a font that fits the kind of event you are promoting.
- Be selective with what you post and how often:
- Posting several stories about the same thing might get boring and cause the viewers to skip after a couple of them, even if the last ones are about different content.
- Posting an abundance of stories in a short span of time might make viewers feel overwhelmed and also cause them to skip the stories after the first few ones.