The performance of music involves the physical expression of musical material in a complex and multimodal process. Furthermore, musical performance involves a sense of 'flow', or immersion in the creative act, that can be better understood through a careful and holistic examination of data captured from this complex physical activity. Flow is especially relevant in improvisatory performance contexts where musicians must make real-time decisions about content and its expression. BLUE HAZE is an open source tool that allows collecting simultaneous streams of data from improvising human musicians that are performing from a common score. The application records and synchronises audio recording with body-, facial- and physiological response tracking with a ground-truth annotation through the reported flow of a performer. This association yields a robust dataset that serves to capture the complex and multi-model process of making music 'in the flow'. This dataset can be a useful tool for a range of applications, such as creative AI practices, music generation in game engines, music information retrieval and humanisation of static systems—e.g. MIDI file playback and sound processing parameters.
Blue Haze was fully supports data capture in an improvisation context. It means the musician is, or should be, embodied interacting with the music, adhering to the concept of musicking in the flow while the software captures the data in role-interaction performances.
To achieve all these criteria, Blue Haze is as less-intrusive as possible, collecting and integrating in real-time all information coming from the five inputs: USB video camera, Intel Realsense camera, USB microphone, Brainbit EEG headband and Bitalino. The project sample rate is 10Hz. The captured data is stored, also in real-time, in a MongoDB database, which is a document-oriented database platform, using JSON-like documents. The reasons to choose MongoDB are manifold:
-
It is a NoSQL database, meaning that it is easy to change the structure of the data without having to deal with complex relational tables and fixed structures. This particular characteristic proved to be central to the rapid development of Blue Haze. As time was a limiting factor, the ability to test the system and change it in seconds to reflect the data we needed was pivotal.
-
MongoDB works seamlessly with Python. In Blue Haze, we are using the asynchronous library “Motorv” to connect to MongoDB. It works well with the overall design of the app, which is based on threads to guarantee one operation does not block the whole app.
-
It is easy to export data from MongoDB to formats like CSV, and then use it to visualise the data or to train machine learn models.
We built the app interface using Qt, a modern, free and open-source widget toolkit for creating graphical interfaces. Qt was also used to record the USB microphone to a WAV file, taking advantage of its integrated multimedia capabilities. One of the setbacks of using Qt on Windows is that it is not able to record video files. It is a well-known limitation on Windows, which required to find creative solutions. In the end, FFmpeg, also a free and open-source project to handle video and audio, was used to capture the video and audio from the USB camera. From the technical point of view, this solution requires instantiating FFmpeg as an external process, managed by Windows.
We considered macOS and Linux versions of Blue Haze and due to Python’s cross-platform nature, Blue Haze indeed started as a cross-platform project. However, because of limitations in the libraries used to connect to the Brainbit and to acquire the skeleton data from the Intel Realsense camera, Linux and macOS support was abandoned throughout the app development. Windows was the only OS to support all the sensors fully.
The data generated by Blue Haze consists of an AVI video file, a WAV audio file and MongoDB data objects. Each MongoDB object is comprised of timestamps used to sync the data, unique session identifiers, a delta time – the time difference between each data capture in the same session –, the location of the video and audio files, cleaned up and organised captured data from Bitalino, Brainbit and Intel Realsense, the backing track file used to guide the musician during the recording session, and a flow feedback provided by the musician after each recording session.
Feedback is obtained through the use of a physical device, a PicoBoard slider. Having as a visual guide a spectrogram of the recording and listening to the audio, the musician moves the slider, assigning a value from 0 to 100 to each particular moment a change in the flow level was felt. The flow values are updated in the database in real-time.