Proposal: Audio Attribute Extractor, measuring audio files in a workflow

[Matthew-DWM]

Summary

A new operator that reads audio files from a binary column, as File Scan produces them, and measures one property of the sound over time: its waveform, loudness, noisiness, pitch, onsets or beats. Each file becomes rows of x (time in seconds), y (the value) and series (the property's name), so the existing chart, aggregate and join operators work on it unchanged.

The idea comes from issue #3762. A prototype of the feature exists at PR #9146 with a demo video and additional details.

Motivation

Texera can already load audio files as binary data, and the Hugging Face operator can send audio to a hosted model for speech recognition or classification. What's missing is a way to look at the sound itself. Today that takes a Python UDF that brings its own decoder and signal-processing code.

Some uses:

• Compare a folder of recordings by tempo, loudness or how busy each one is, joined with their metadata.
• Follow the pitch of a voice or an instrument over time.
• Find quiet, loud or noisy stretches, for example to check recordings before sending them to a speech model.
• Count sound events, such as notes or taps, per file.

Approach

For the user. One operator in Utilities. Pick the audio column (WAV, MP3 or FLAC) and one attribute:

Attribute	y
waveform	the signal
rms	loudness of each frame
zcr	zero-crossing rate: high for noise, low for tones
pitch_hz	pitch, 60 to 1000 Hz
onsets	1.0 at each new sound
beats	1.0 at each beat

The other settings are the sample rate, the frame and hop length, a cap on rows and seconds per file, keeping the input's other columns (such as the filename), and skipping unreadable files instead of stopping.

How it runs. Like the other Python-based operators: the Scala descriptor checks the settings when the workflow compiles and generates the Python a worker runs. The worker decodes each file, mixes it to mono (single audio stream), resamples it, measures it and emits a bounded number of rows per file.

Dependencies. One new Python package, miniaudio (MIT), which brings cffi (MIT-0) and pycparser (BSD-3). NumPy and SciPy are already there. Librosa's beat tracker (ISC) is ported to about 170 lines of NumPy, with attribution in LICENSE, rather than adding librosa as a dependency.

Prototype

#9146 has the operator (about 1,350 lines, most of it the Python the worker runs), 68 tests, a docs page and a recording of an example workflow. On my machine, a four-minute WAV takes at most about 1.2 s per attribute.

Open questions

1. Would you find an audio operator useful?
2. Is a new Python dependency acceptable (miniaudio, plus cffi and pycparser)?
3. Should the scope of the current PR be reduced and split into multiple PRs?
4. Is use of the librosa-derived code acceptable?

[zyratlo]

Thanks for writing this up. I don't personally work with audio files, but I can see the use cases that would need an operator like this.

The Python dependencies should be fine. I checked the licenses and they all look good (miniaudio is MIT, cffi is MIT-0, pycparser is BSD-3), which are all on the ASF Category A list. Same with the librosa-derived code, since ISC is also Category A. The PR reviewer should verify this though.

On PR size, we generally want PRs to be as small and modular as possible so they're easier to review. Yours is about 3.5k lines, so even if most of that is tests and other non-production code, it's still quite large. It would be good to split it up so that each PR still makes sense on its own. For example, the first PR could add the operator with decoding and the simpler attributes (waveform, rms, zcr), and then pitch, onsets and beats could follow. Beats is the only part that uses the librosa port, so that would also keep the licensing changes out of the first PR.

[chenlica]

A general question is: do we have users who want this feature? If not, I would like to give it a low priority.

The system has many issues. It will be good to fix existing issues first.