Sound class hierarchy from the paper "HiSSNet: Hierarchical Sound Event Detection and Speaker Identification Prototypical Network." hierarchy.yaml contains the hierarchical relationships between general and specific sound classes. taxonomy.json contains the mappings between specific sound class labels and audio dataset classes.
Modern noise-cancelling headphones have significantly improved users’ auditory experiences by removing unwanted background noise, but they can also block out sounds that matter to users. Machine learning (ML) models for sound event detection (SED) and speaker identification (SID) can enable headphones to pass through important sounds selectively; however, implementing these models for a user-centric experience presents several unique challenges. First, most people spend limited time customizing their headphones, so the sound detection should work reasonably well out of the box. Second, the models should be able to learn over time the specific sounds that matter to users based on their implicit and explicit interactions. Finally, such models should have a small memory footprint to be run on low-power headphones with limited on-chip memory. In this paper, we propose addressing these challenges using HiSSNet, an SEID (SED and SID) model that uses hierarchical prototypical networks to detect both general and specific sounds of interest and characterize both alarm-like and speech sounds. We show that HiSSNet outperforms a joint model trained using non-hierarchical prototypical networks by 6.9 – 8.6%. When compared to state-of-the-art (SOTA) models trained specifically for SED or SID alone, HiSSNet achieves similar or better performance while reducing the memory footprint required to support multiple capabilities on-device.