Skip to content
Go to file

Latest commit

added citation requirements for the repository

Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Task 4 Large-scale weakly supervised sound event detection for smart cars

Update Log

  • Dec 13: Added Groundtruth release folder - Contains groundtruth information for evaluation, testing set.
  • Nov 20: We've provided direct-download links for the audio of the development and evaluation sets in Section 1.
  • August 16: Addition of script for SubTaskB computation for participant's convenience
  • July 19: Update to params for Task 4 to maintain consistency with Task 3 parameters
  • July 8: Corner case SubTaskA metrics - bug fix for compute metrics function. Additional files added to the prediction folder located in evaluation for sanity checks of metrics computation script.
  • July 4: Added development as well as challenge support after incorporating latest paramount changes from DCASE Framework repository made by Toni on July 3rd. Integration testing for challenge and development setup done.
  • June 26: Added script to compute duration of audio files. Added Training and testing set duration of audio files as a reference for participants to cross check with. Alternate Method to download audio for Development Set [Training and Testing Set] updated.
  • June 24: Modification to groundtruth_strong_label_testing_set.csv - Fixed annotations for redundant class
  • June 21: Carriage return removal from groundtruth weak and strong labels csv files.
  • June 17: Evaluation Script Update for ClassWise Metrics Computation for Task4 SubTaskAMetrics
  • May 1: Added baseline code (based on Task 3's system), performance and Subtask A metric code.
  • April 2: Added strong labels.
  • April 1: Added evaluation folder.


Benjamin Elizalde, Emmanuel Vincent, Bhiksha Raj

Data Preparation, Annotations

Ankit Shah (, Benjamin Elizalde (

Annotations, Baseline and Subtask A Metric

Rohan Badlani (, Benjamin Elizalde (, Ankit Shah (


  1. Direct-download for the audio of the development and evaluation sets

  2. Script to download the development data for Task 4

  3. Scripts to evaluate Task 4 - Subtask A (Audio tagging) and Subtask B

  4. Strong Label's annotations for Testing

1. Direct-download for the audio of the development and evaluation sets

The annotations for the evaluation set have not been released yet. The password can be shared via sending a request email to Ankit Shah ( or Benjamin Elizalde (

Development: Training

Development: Testing


Quick download of large files via Google Drive.

Thanks to Justin Salamon for sharing the trick -

2. Script to download the development data for Task 4

Prerequisite installations

[Current version of our scripts require python 2.7 version installation - If user has other python revision installed, we encourage them to setup a virtualenvironment for the DCASE Challenge until the scripts support python 3 as well]

  1. youtube-dl - [sudo] pip install --upgrade youtube_dl
  2. pafy - [sudo] pip install pafy
  3. tqdm (progress bar) - [sudo] pip install tqdm
  4. multiprocessing - [sudo] pip install multiprocessing
  5. sox tool - sudo apt-get install sox
  6. ffmpeg - sudo apt-get install ffmpeg

Cloning this repository

Since this is a repository that references DCASE2017-baseline-system as a submodule, you should use the following command to clone this repository completely:

git clone --recurse git submodule foreach git pull origin master

Running Instructions for Task4 Baseline

Please follow the following steps to get the baseline results:

1. git submodule foreach git pull origin master
2. Go to DCASE2017-baseline-system (cd DCASE2017-baseline-system)
3. Run pip install -r requirements.txt
4. Go back to root (cd ..)
5. Run python

The above will be using task.defaults.yaml for the baseline run.

If you would like to run your own modifications on top of DCASE baseline system, pls refer to the following links: 1. Basic Usage: 2. Extending the framework:


  1. Downloads the audio from the videos for the testing set first and then for the training set. - Multiprocessing - ensures three files are downloaded simultaneously to reduce the heavy download time to 40 percent as compared with single threaded performance.
  2. Formats the audio with consistent parameters - currently set as 1 channel, 16 bit precision, 44.1kHz sampling rate.
  3. Extracts the 10-sec clips from the formatted audio according to the start and end times.
  4. The script output includes the audio for 1,2 and 3, unless testing script is modified to remove audio from 2 and/or 3, that is the original audio and the formatted audio.
  5. To denote a unique identifier for every run/launch of downloading files - script stores the timestamp and assigns to each of the output files and folder names.
  6. Please, contact Ankit/Benjamin in case one or more videos are not properly downloaded or available, or with any other issue. Participants can create their own scripts to download the audio. Please ensure that you have all the 10-sec clip whose duration matches organizer's list.


Download audio: testing_set.csv, training_set.csv

Groundtruth weak labels: groundtruth_weak_label_testing_set.csv groundtruth_weak_label_training_set.csv

Groundtruth strong labels: groundtruth_strong_label_testing_set.csv

Training Set Audio Duration: duration_of_files_in_training_set.csv

Testing Set Audio Duration: duration_of_files_in_testing_set.csv


$python <CSV filename - relative path is also fine> Sample Usage - python training_set.csv

Alternate Audio Download Method.

User Modifiable Parameters and Options

  1. Audio formatting can be modified in the "format_audio" method defined in the script
  2. Removal of original audio and/or formatted audio paths can be done by uncommenting and modifying <os.system(cmdstring2)> in "download_audio_method" function defined in


  1. First folder contains original best audio from youtube: <csv_name><testing/training>_audio_downloaded
  2. Second folder contains the corresponding formatted audio: <csv_name><testing/training>_audio_formatted_downloaded
  3. Third folder contains the extracted 10-sec clips: <csv_name><testing/training>_audio_formatted_downloaded_and_ssegmented_downloads

Note:- To each downloaded audio string "Y" is added as tools like sox and ffmpeg causes problem when filename starts with "--" or "-".

Number of Audio id count files

  1. testing_set_num_files_per_class.csv - For each class - specifies number of audio segments present in the testing set

Total testing set downloaded files - 488

  1. training_set_num_files_per_class.csv - For each class - specifies number of audio segments present in the training set

Total training set downloaded files - 51172

Script to compute duration of audio files.

$bash <training/testing/evaluation> <Audio Path> <Output Path where duration list will be written>

3. Script to evaluate Task 4 - Subtask A (Audio tagging)

Usage Subtask A - refer to folder called "evaluation" and use the following command.

$python groundtruth/groundtruth_weak_label_testing_set.csv prediction/perfect_prediction.csv output/perfect_prediction_output.csv

Usage Subtask B - refer to folder called "TaskB_evaluation" for instructions.

4. Strong Label's annotations for Testing

  1. Only one person was involved in the annotation of each 10-sec clip and all the clips were randomly divided and assigned to 5 people.
  2. The sound event annotations were based on the audio and not the video.
  3. The strong labels correspond to the file: groundtruth_strong_label_testing_set.csv
  4. The format of strong labels is the same as the DCASE format (Task 3 and Task 4: Audio tagging).
  5. Less than 2% of the 10-sec clips had the presence of a sound according to AudioSet, but didn't seem to contain the sound event. Thus, start and end time were assigned 0.

Results for Task 4 DCASE 2017 challenge

DCASE 2017 Results -


  title={DCASE 2017 challenge setup: Tasks, datasets and baseline system},
  author={Mesaros, Annamaria and Heittola, Toni and Diment, Aleksandr and Elizalde, Benjamin and Shah, Ankit and Vincent, Emmanuel and Raj, Bhiksha and Virtanen, Tuomas},
  booktitle={DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes and Events},


    Author = "Badlani, Rohan and Shah, Ankit and Elizalde, Benjamin",
    title = "{DCASE} 2017 Challenge Setup: Tasks, Datasets and Baseline System",
    institution = "DCASE2017 Challenge",
    year = "2017",
    month = "September",
    abstract = "DCASE 2017 Challenge consists of four tasks: acoustic scene classification, detection of rare sound events, sound event detection in real-life audio, and large-scale weakly supervised sound event detection for smart cars. This paper presents the setup of these tasks: task definition, dataset, experimental setup, and baseline system results on the development dataset. The baseline systems for all tasks rely on the same implementation using multilayer perceptron and log mel-energies, but differ in the structure of the output layer and the decision making process, as well as the evaluation of system output using task specific metrics."
You can’t perform that action at this time.